Package org.apache.wayang.basic.operators

Class ParquetSource

java.lang.Object

org.apache.wayang.core.plan.wayangplan.OperatorBase

org.apache.wayang.core.plan.wayangplan.UnarySource<Record>

org.apache.wayang.basic.operators.ParquetSource

All Implemented Interfaces:

Serializable, ActualOperator, ElementaryOperator, Operator

Direct Known Subclasses:

JavaParquetSource, SparkParquetSource

public class ParquetSource
extends UnarySource<Record>

This source reads a parquet file and outputs the lines as Record units.

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected class	ParquetSource.CardinalityEstimator	Custom CardinalityEstimator for FlatMapOperators.

Nested classes/interfaces inherited from class org.apache.wayang.core.plan.wayangplan.OperatorBase

OperatorBase.GsonSerializer

Field Summary

Fields inherited from class org.apache.wayang.core.plan.wayangplan.OperatorBase

inputSlots, outputSlots, STANDARD_OPERATOR_ARGS

Fields inherited from interface org.apache.wayang.core.plan.wayangplan.Operator

FIRST_EPOCH

Constructor Summary

Constructors

Constructor	Description
ParquetSource(String inputUrl, String[] projection, String... fieldNames)
ParquetSource(String inputUrl, String[] projection, DataSetType<Record> type)
ParquetSource(ParquetSource that)	Copies an instance (exclusive of broadcasts).

Method Summary

Modifier and Type	Method	Description
static ParquetSource	create(String inputUrl, String[] projection)	Creates a new instance.
Optional<CardinalityEstimator>	createCardinalityEstimator(int outputIndex, Configuration configuration)	Provide a CardinalityEstimator for the OutputSlot at outputIndex.
String	getInputUrl()
org.apache.parquet.hadoop.metadata.ParquetMetadata	getMetadata()
String[]	getProjection()
org.apache.parquet.schema.MessageType	getSchema()

Methods inherited from class org.apache.wayang.core.plan.wayangplan.UnarySource

getOutput, getType

Methods inherited from class org.apache.wayang.core.plan.wayangplan.OperatorBase

accept, addBroadcastInput, addTargetPlatform, at, collectMappedInputSlots, collectMappedOutputSlots, copy, createCopy, getAllInputs, getAllOutputs, getCardinalityEstimator, getContainer, getEpoch, getName, getOriginal, getSimpleClassName, getTargetPlatforms, isAuxiliary, isSupportingBroadcastInputs, propagateInputCardinality, propagateOutputCardinality, setAuxiliary, setCardinalityEstimator, setContainer, setEpoch, setName, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.wayang.core.plan.wayangplan.ActualOperator

accept

Methods inherited from interface org.apache.wayang.core.plan.wayangplan.ElementaryOperator

getCardinalityEstimator, isAuxiliary, setAuxiliary, setCardinalityEstimator

Methods inherited from interface org.apache.wayang.core.plan.wayangplan.Operator

addBroadcastInput, addTargetPlatform, broadcastTo, broadcastTo, collectMappedInputSlots, collectMappedOutputSlots, connectTo, connectTo, getAllInputs, getAllOutputs, getCardinalityPusher, getContainer, getEffectiveOccupant, getEffectiveOccupant, getEpoch, getEstimationContextProperties, getForwards, getInnermostLoop, getInput, getInput, getLoopStack, getName, getNumBroadcastInputs, getNumInputs, getNumOutputs, getNumRegularInputs, getOuterInputSlot, getOutermostInputSlot, getOutermostOutputSlots, getOutput, getOutput, getParent, getTargetPlatforms, isAlternative, isConversion, isElementary, isExecutionOperator, isFeedbackInput, isFeedforwardOutput, isLoopHead, isLoopSubplan, isOwnerOf, isReading, isSink, isSource, isSubplan, isSupportingBroadcastInputs, isUnconnected, propagateInputCardinality, propagateOutputCardinality, propagateOutputCardinality, replaceWith, setContainer, setEpoch, setInput, setName, setOutput

Constructor Details

ParquetSource

public ParquetSource(String inputUrl,
 String[] projection,
 String... fieldNames)

ParquetSource

public ParquetSource(String inputUrl,
 String[] projection,
 DataSetType<Record> type)

ParquetSource

public ParquetSource(ParquetSource that)

Copies an instance (exclusive of broadcasts).

Parameters:

that - that should be copied

Method Details

create

public static ParquetSource create(String inputUrl,
 String[] projection)

Creates a new instance.

Parameters:

inputUrl - name of the file to be read

projection - names of the columns to filter; can be omitted but allows for an early projection

getInputUrl

public String getInputUrl()

getProjection

public String[] getProjection()

getMetadata

public org.apache.parquet.hadoop.metadata.ParquetMetadata getMetadata()

getSchema

public org.apache.parquet.schema.MessageType getSchema()

createCardinalityEstimator

public Optional<CardinalityEstimator> createCardinalityEstimator(int outputIndex,
 Configuration configuration)

Description copied from interface: ElementaryOperator

Provide a CardinalityEstimator for the OutputSlot at outputIndex.

Parameters:

outputIndex - index of the OutputSlot for that the CardinalityEstimator is requested

configuration - if the CardinalityEstimator depends on further ones, use this to obtain the latter

Returns:

an Optional that might provide the requested instance