Package org.apache.wayang.spark.execution

Class SparkExecutor

java.lang.Object

org.apache.wayang.core.util.AbstractReferenceCountable

org.apache.wayang.core.platform.ExecutorTemplate

org.apache.wayang.core.platform.PushExecutorTemplate

org.apache.wayang.spark.execution.SparkExecutor

All Implemented Interfaces:

CompositeExecutionResource, ExecutionResource, Executor, ReferenceCountable

public class SparkExecutor
extends PushExecutorTemplate

Executor implementation for the SparkPlatform.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.wayang.core.platform.PushExecutorTemplate

PushExecutorTemplate.StageExecution

Nested classes/interfaces inherited from interface org.apache.wayang.core.platform.Executor

Executor.Factory

Field Summary

Fields

Modifier and Type	Field	Description
FunctionCompiler	compiler	Compiler to create Spark UDFs.
final org.apache.spark.api.java.JavaSparkContext	sc	The JavaSparkContext to be used by this instance.
final org.apache.spark.sql.SparkSession	ss	The SparkSession to be used by this instance.

Fields inherited from class org.apache.wayang.core.platform.PushExecutorTemplate

job

Fields inherited from class org.apache.wayang.core.platform.ExecutorTemplate

logger

Constructor Summary

Constructors

Constructor	Description
SparkExecutor(SparkPlatform platform, Job job)

Method Summary

Modifier and Type	Method	Description
void	dispose()	Releases any instances acquired by this instance to execute ExecutionStages.
protected Tuple<List<ChannelInstance>,PartialExecution>	execute(ExecutionTask task, List<ChannelInstance> inputChannelInstances, OptimizationContext.OperatorContext producerOperatorContext, boolean isRequestEagerExecution)	Executes the given task and return the output ChannelInstances.
void	forward(ChannelInstance input, ChannelInstance output)	Utility method to forward a RddChannel.Instance to another.
FunctionCompiler	getCompiler()	Provide a FunctionCompiler.
int	getNumDefaultPartitions()	Hint to SparkExecutionOperators on how many partitions they should request.
SparkPlatform	getPlatform()

Methods inherited from class org.apache.wayang.core.platform.PushExecutorTemplate

execute, getJob

Methods inherited from class org.apache.wayang.core.platform.ExecutorTemplate

createPartialExecution, disposeUnreferenced, getConfiguration, getCrossPlatformExecutor, register, registerMeasuredCardinalities, registerMeasuredCardinality, toString, unregister

Methods inherited from class org.apache.wayang.core.util.AbstractReferenceCountable

disposeIfUnreferenced, getNumReferences, isDisposed, noteDiscardedReference, noteObtainedReference

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.wayang.core.util.ReferenceCountable

disposeIfUnreferenced, getNumReferences, isDisposed, noteDiscardedReference, noteObtainedReference

Field Details

sc

public final org.apache.spark.api.java.JavaSparkContext sc

The JavaSparkContext to be used by this instance.

See Also:

• sparkContextReference

ss

public final org.apache.spark.sql.SparkSession ss

The SparkSession to be used by this instance.

compiler

public FunctionCompiler compiler

Compiler to create Spark UDFs.

Constructor Details

SparkExecutor

public SparkExecutor(SparkPlatform platform,
 Job job)

Method Details

execute

protected Tuple<List<ChannelInstance>,PartialExecution> execute(ExecutionTask task,
 List<ChannelInstance> inputChannelInstances,
 OptimizationContext.OperatorContext producerOperatorContext,
 boolean isRequestEagerExecution)

Description copied from class: PushExecutorTemplate

Executes the given task and return the output ChannelInstances.

Specified by:

execute in class PushExecutorTemplate

Parameters:

task - that should be executed

inputChannelInstances - inputs into the task

isRequestEagerExecution - whether the ExecutionTask should be executed eagerly if possible

Returns:

the ChannelInstances created as output of task

forward

public void forward(ChannelInstance input,
 ChannelInstance output)

Utility method to forward a RddChannel.Instance to another.

Parameters:

input - that should be forwarded

output - to that should be forwarded

getPlatform

public SparkPlatform getPlatform()

Returns:

the Platform this instance belongs to

getNumDefaultPartitions

public int getNumDefaultPartitions()

Hint to SparkExecutionOperators on how many partitions they should request.

Returns:

the default number of partitions

dispose

public void dispose()

Description copied from interface: Executor

Releases any instances acquired by this instance to execute ExecutionStages.

Specified by:

dispose in interface ExecutionResource

Specified by:

dispose in interface Executor

Overrides:

dispose in class ExecutorTemplate

getCompiler

public FunctionCompiler getCompiler()

Provide a FunctionCompiler.

Returns:

the FunctionCompiler