Package org.apache.wayang.basic.operators

Class MapPartitionsOperator<InputType,OutputType>

java.lang.Object

org.apache.wayang.core.plan.wayangplan.OperatorBase

org.apache.wayang.core.plan.wayangplan.UnaryToUnaryOperator<InputType,OutputType>

org.apache.wayang.basic.operators.MapPartitionsOperator<InputType,OutputType>

All Implemented Interfaces:

Serializable, ActualOperator, ElementaryOperator, Operator

Direct Known Subclasses:

FlinkMapPartitionsOperator, JavaMapPartitionsOperator, SparkMapPartitionsOperator

public class MapPartitionsOperator<InputType,OutputType>
extends UnaryToUnaryOperator<InputType,OutputType>

This operator takes as input potentially multiple input data quanta and outputs multiple input data quanta.

Since Wayang is not a physical execution engine, its notion of partitions is rather loose. Implementors of this operator should guarantee that the partitions are distinct in their data quanta and that all partitions together are complete w.r.t. the data quanta.

However, no further assumptions on partitions shall be made, such as: whether partitions can be iterated multiple times; whether partitions can be empty; whether there is a partition on each machine on distributed platforms; or whether partitions have a certain sorting order.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.wayang.core.plan.wayangplan.OperatorBase

OperatorBase.GsonSerializer

Field Summary

Fields

Modifier and Type	Field	Description
protected final MapPartitionsDescriptor<InputType,OutputType>	functionDescriptor	Function that this operator applies to the input elements.

Fields inherited from class org.apache.wayang.core.plan.wayangplan.OperatorBase

inputSlots, outputSlots, STANDARD_OPERATOR_ARGS

Fields inherited from interface org.apache.wayang.core.plan.wayangplan.Operator

FIRST_EPOCH

Constructor Summary

Constructors

Constructor	Description
MapPartitionsOperator(MapPartitionsOperator<InputType,OutputType> that)	Copies an instance (exclusive of broadcasts).
MapPartitionsOperator(FunctionDescriptor.SerializableFunction<Iterable<InputType>,Iterable<OutputType>> function, Class<InputType> inputTypeClass, Class<OutputType> outputTypeClass)	Creates a new instance.
MapPartitionsOperator(MapPartitionsDescriptor<InputType,OutputType> functionDescriptor)	Creates a new instance.
MapPartitionsOperator(MapPartitionsDescriptor<InputType,OutputType> functionDescriptor, DataSetType<InputType> inputType, DataSetType<OutputType> outputType)	Creates a new instance.

Method Summary

Modifier and Type	Method	Description
Optional<CardinalityEstimator>	createCardinalityEstimator(int outputIndex, Configuration configuration)	Provide a CardinalityEstimator for the OutputSlot at outputIndex.
MapPartitionsDescriptor<InputType,OutputType>	getFunctionDescriptor()

Methods inherited from class org.apache.wayang.core.plan.wayangplan.UnaryToUnaryOperator

getInput, getInputType, getOutput, getOutputType

Methods inherited from class org.apache.wayang.core.plan.wayangplan.OperatorBase

accept, addBroadcastInput, addTargetPlatform, at, collectMappedInputSlots, collectMappedOutputSlots, copy, createCopy, getAllInputs, getAllOutputs, getCardinalityEstimator, getContainer, getEpoch, getName, getOriginal, getSimpleClassName, getTargetPlatforms, isAuxiliary, isSupportingBroadcastInputs, propagateInputCardinality, propagateOutputCardinality, setAuxiliary, setCardinalityEstimator, setContainer, setEpoch, setName, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.wayang.core.plan.wayangplan.ActualOperator

accept

Methods inherited from interface org.apache.wayang.core.plan.wayangplan.ElementaryOperator

getCardinalityEstimator, isAuxiliary, setAuxiliary, setCardinalityEstimator

Methods inherited from interface org.apache.wayang.core.plan.wayangplan.Operator

addBroadcastInput, addTargetPlatform, broadcastTo, broadcastTo, collectMappedInputSlots, collectMappedOutputSlots, connectTo, connectTo, getAllInputs, getAllOutputs, getCardinalityPusher, getContainer, getEffectiveOccupant, getEffectiveOccupant, getEpoch, getEstimationContextProperties, getForwards, getInnermostLoop, getInput, getInput, getLoopStack, getName, getNumBroadcastInputs, getNumInputs, getNumOutputs, getNumRegularInputs, getOuterInputSlot, getOutermostInputSlot, getOutermostOutputSlots, getOutput, getOutput, getParent, getTargetPlatforms, isAlternative, isConversion, isElementary, isExecutionOperator, isFeedbackInput, isFeedforwardOutput, isLoopHead, isLoopSubplan, isOwnerOf, isReading, isSink, isSource, isSubplan, isSupportingBroadcastInputs, isUnconnected, propagateInputCardinality, propagateOutputCardinality, propagateOutputCardinality, replaceWith, setContainer, setEpoch, setInput, setName, setOutput

Field Details

functionDescriptor

protected final MapPartitionsDescriptor<InputType,OutputType> functionDescriptor

Function that this operator applies to the input elements.

Constructor Details

MapPartitionsOperator

public MapPartitionsOperator(FunctionDescriptor.SerializableFunction<Iterable<InputType>,Iterable<OutputType>> function,
 Class<InputType> inputTypeClass,
 Class<OutputType> outputTypeClass)

Creates a new instance.

MapPartitionsOperator

public MapPartitionsOperator(MapPartitionsDescriptor<InputType,OutputType> functionDescriptor)

Creates a new instance.

MapPartitionsOperator

public MapPartitionsOperator(MapPartitionsDescriptor<InputType,OutputType> functionDescriptor,
 DataSetType<InputType> inputType,
 DataSetType<OutputType> outputType)

Creates a new instance.

MapPartitionsOperator

public MapPartitionsOperator(MapPartitionsOperator<InputType,OutputType> that)

Copies an instance (exclusive of broadcasts).

Parameters:

that - that should be copied

Method Details

getFunctionDescriptor

public MapPartitionsDescriptor<InputType,OutputType> getFunctionDescriptor()

createCardinalityEstimator

public Optional<CardinalityEstimator> createCardinalityEstimator(int outputIndex,
 Configuration configuration)

Description copied from interface: ElementaryOperator

Provide a CardinalityEstimator for the OutputSlot at outputIndex.

Parameters:

outputIndex - index of the OutputSlot for that the CardinalityEstimator is requested

configuration - if the CardinalityEstimator depends on further ones, use this to obtain the latter

Returns:

an Optional that might provide the requested instance