Package org.apache.wayang.spark.operators

Interface SparkExecutionOperator

All Superinterfaces:

ActualOperator, ElementaryOperator, ExecutionOperator, Operator, Serializable

All Known Implementing Classes:

SparkBernoulliSampleOperator, SparkBroadcastOperator, SparkCacheOperator, SparkCartesianOperator, SparkCoGroupOperator, SparkCollectionSource, SparkCollectOperator, SparkCountOperator, SparkDecisionTreeClassificationOperator, SparkDecisionTreeRegressionOperator, SparkDistinctOperator, SparkDoWhileOperator, SparkFilterOperator, SparkFlatMapOperator, SparkGlobalMaterializedGroupOperator, SparkGlobalReduceOperator, SparkIEJoinOperator, SparkIESelfJoinOperator, SparkIntersectOperator, SparkJoinOperator, SparkKafkaTopicSink, SparkKafkaTopicSource, SparkKMeansOperator, SparkLinearRegressionOperator, SparkLinearSVCOperator, SparkLocalCallbackSink, SparkLogisticRegressionOperator, SparkLoopOperator, SparkMapOperator, SparkMapPartitionsOperator, SparkMaterializedGroupByOperator, SparkModelTransformOperator, SparkObjectFileSink, SparkObjectFileSource, SparkParquetSource, SparkPredictOperator, SparkRandomPartitionSampleOperator, SparkReduceByOperator, SparkRepeatOperator, SparkShufflePartitionSampleOperator, SparkSortOperator, SparkTextFileSink, SparkTextFileSource, SparkTsvFileSink, SparkTsvFileSource, SparkUnionAllOperator, SparkZipWithIdOperator, SqlToRddOperator

public interface SparkExecutionOperator
extends ExecutionOperator

Execution operator for the SparkPlatform.

Field Summary

Fields inherited from interface org.apache.wayang.core.plan.wayangplan.Operator

FIRST_EPOCH

Method Summary

Modifier and Type	Method	Description
boolean	containsAction()	Tell whether this instances is a Spark action.
Tuple<Collection<ExecutionLineageNode>,Collection<ChannelInstance>>	evaluate(ChannelInstance[] inputs, ChannelInstance[] outputs, SparkExecutor sparkExecutor, OptimizationContext.OperatorContext operatorContext)	Evaluates this operator.
default SparkPlatform	getPlatform()
default void	name(org.apache.spark.api.java.JavaPairRDD<?,?> rdd)	Utility method to name an RDD according to this instance's name.
default void	name(org.apache.spark.api.java.JavaRDD<?> rdd)	Utility method to name an RDD according to this instance's name.

Methods inherited from interface org.apache.wayang.core.plan.wayangplan.ActualOperator

accept

Methods inherited from interface org.apache.wayang.core.plan.wayangplan.ElementaryOperator

createCardinalityEstimator, getCardinalityEstimator, isAuxiliary, setAuxiliary, setCardinalityEstimator

Methods inherited from interface org.apache.wayang.core.plan.wayangplan.ExecutionOperator

copy, createLoadProfileEstimator, createOutputChannelInstances, getLimitBaseKey, getLoadProfileEstimatorConfigurationKey, getLoadProfileEstimatorConfigurationKeys, getOriginal, getOutputChannelDescriptor, getSupportedInputChannels, getSupportedOutputChannels, isFiltered

Methods inherited from interface org.apache.wayang.core.plan.wayangplan.Operator

addBroadcastInput, addTargetPlatform, broadcastTo, broadcastTo, collectMappedInputSlots, collectMappedOutputSlots, connectTo, connectTo, getAllInputs, getAllOutputs, getCardinalityPusher, getContainer, getEffectiveOccupant, getEffectiveOccupant, getEpoch, getEstimationContextProperties, getForwards, getInnermostLoop, getInput, getInput, getLoopStack, getName, getNumBroadcastInputs, getNumInputs, getNumOutputs, getNumRegularInputs, getOuterInputSlot, getOutermostInputSlot, getOutermostOutputSlots, getOutput, getOutput, getParent, getTargetPlatforms, isAlternative, isConversion, isElementary, isExecutionOperator, isFeedbackInput, isFeedforwardOutput, isLoopHead, isLoopSubplan, isOwnerOf, isReading, isSink, isSource, isSubplan, isSupportingBroadcastInputs, isUnconnected, propagateInputCardinality, propagateOutputCardinality, propagateOutputCardinality, replaceWith, setContainer, setEpoch, setInput, setName, setOutput

Method Details

getPlatform

default SparkPlatform getPlatform()

Specified by:

getPlatform in interface ExecutionOperator

Returns:

the platform that can run this operator

evaluate

Tuple<Collection<ExecutionLineageNode>,Collection<ChannelInstance>> evaluate(ChannelInstance[] inputs,
 ChannelInstance[] outputs,
 SparkExecutor sparkExecutor,
 OptimizationContext.OperatorContext operatorContext)

Evaluates this operator. Takes a set of ChannelInstances according to the operator inputs and manipulates a set of ChannelInstances according to the operator outputs -- unless the operator is a sink, then it triggers execution.

In addition, this method should give feedback of what this instance was doing by wiring the LazyExecutionLineageNodes of input and ouput ChannelInstances and providing a Collection of executed ExecutionLineageNodes.

Parameters:

inputs - ChannelInstances that satisfy the inputs of this operator

outputs - ChannelInstances that accept the outputs of this operator

sparkExecutor - SparkExecutor that executes this instance

operatorContext - optimization information for this instance

Returns:

Collections of what has been executed and produced

containsAction

boolean containsAction()

Tell whether this instances is a Spark action. This is important to keep track on when Spark is actually initialized.

Returns:

whether this instance issues Spark actions

name

default void name(org.apache.spark.api.java.JavaRDD<?> rdd)

Utility method to name an RDD according to this instance's name.

Parameters:

rdd - that should be renamed

See Also:

• Operator.getName()

name

default void name(org.apache.spark.api.java.JavaPairRDD<?,?> rdd)

Utility method to name an RDD according to this instance's name.

Parameters:

rdd - that should be renamed

See Also:

• Operator.getName()