Package org.apache.wayang.spark.compiler

Class FunctionCompiler

java.lang.Object

org.apache.wayang.spark.compiler.FunctionCompiler

public class FunctionCompiler
extends Object

A compiler translates Wayang functions into executable Java functions.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	FunctionCompiler.KeyExtractor<T,K>	Spark function for building pair RDDs.
static class	FunctionCompiler.Reducer<Type>	Spark function for aggregating data quanta.
static interface	FunctionCompiler.WayangSparkFunction	Describes functions coming from Wayang, designated for Spark.

Constructor Summary

Constructors

Constructor	Description
FunctionCompiler()

Method Summary

Modifier and Type	Method	Description
<I, O> org.apache.spark.api.java.function.FlatMapFunction<I,O>	compile(FlatMapDescriptor<I,O> descriptor, SparkExecutionOperator operator, OptimizationContext.OperatorContext operatorContext, ChannelInstance[] inputs)	Create an appropriate FlatMapFunction for deploying the given FlatMapDescriptor on Apache Spark.
<I, O> org.apache.spark.api.java.function.FlatMapFunction<Iterator<I>,O>	compile(MapPartitionsDescriptor<I,O> descriptor, SparkExecutionOperator operator, OptimizationContext.OperatorContext operatorContext, ChannelInstance[] inputs)	Create an appropriate Function for deploying the given MapPartitionsDescriptor on Apache Spark's AbstractJavaRDDLike.mapPartitions(FlatMapFunction).
<Type> org.apache.spark.api.java.function.Function<Type,Boolean>	compile(PredicateDescriptor<Type> predicateDescriptor, SparkExecutionOperator operator, OptimizationContext.OperatorContext operatorContext, ChannelInstance[] inputs)	Create an appropriate Function-based predicate for deploying the given PredicateDescriptor on Apache Spark.
<T> org.apache.spark.api.java.function.Function2<T,T,T>	compile(ReduceDescriptor<T> descriptor, SparkExecutionOperator operator, OptimizationContext.OperatorContext operatorContext, ChannelInstance[] inputs)	Create an appropriate Function for deploying the given ReduceDescriptor on Apache Spark.
<I, O> org.apache.spark.api.java.function.Function<I,O>	compile(TransformationDescriptor<I,O> descriptor, SparkExecutionOperator operator, OptimizationContext.OperatorContext operatorContext, ChannelInstance[] inputs)	Create an appropriate Function for deploying the given TransformationDescriptor on Apache Spark.
<T, K> FunctionCompiler.KeyExtractor<T,K>	compileToKeyExtractor(TransformationDescriptor<T,K> descriptor)	Compile a key extraction.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

FunctionCompiler

public FunctionCompiler()

Method Details

compile

public <I,
O>
org.apache.spark.api.java.function.Function<I,O> compile(TransformationDescriptor<I,O> descriptor,
 SparkExecutionOperator operator,
 OptimizationContext.OperatorContext operatorContext,
 ChannelInstance[] inputs)

Create an appropriate Function for deploying the given TransformationDescriptor on Apache Spark.

Parameters:

descriptor - describes the transformation function

operator - that executes the Function; only required if the descriptor describes an ExtendedFunction

operatorContext - contains optimization information for the operator

inputs - that feed the operator; only required if the descriptor describes an ExtendedFunction

compile

public <I,
O>
org.apache.spark.api.java.function.FlatMapFunction<Iterator<I>,O> compile(MapPartitionsDescriptor<I,O> descriptor,
 SparkExecutionOperator operator,
 OptimizationContext.OperatorContext operatorContext,
 ChannelInstance[] inputs)

Create an appropriate Function for deploying the given MapPartitionsDescriptor on Apache Spark's AbstractJavaRDDLike.mapPartitions(FlatMapFunction).

Parameters:

descriptor - describes the function

operator - that executes the Function; only required if the descriptor describes an ExtendedFunction

operatorContext - contains optimization information for the operator

inputs - that feed the operator; only required if the descriptor describes an ExtendedFunction

compileToKeyExtractor

public <T,
K> FunctionCompiler.KeyExtractor<T,K> compileToKeyExtractor(TransformationDescriptor<T,K> descriptor)

Compile a key extraction.

Returns:

a compiled function

compile

public <I,
O>
org.apache.spark.api.java.function.FlatMapFunction<I,O> compile(FlatMapDescriptor<I,O> descriptor,
 SparkExecutionOperator operator,
 OptimizationContext.OperatorContext operatorContext,
 ChannelInstance[] inputs)

Create an appropriate FlatMapFunction for deploying the given FlatMapDescriptor on Apache Spark.

Parameters:

descriptor - describes the function

operator - that executes the Function; only required if the descriptor describes an ExtendedFunction

operatorContext - contains optimization information for the operator

inputs - that feed the operator; only required if the descriptor describes an ExtendedFunction

compile

public <T>
org.apache.spark.api.java.function.Function2<T,T,T> compile(ReduceDescriptor<T> descriptor,
 SparkExecutionOperator operator,
 OptimizationContext.OperatorContext operatorContext,
 ChannelInstance[] inputs)

Create an appropriate Function for deploying the given ReduceDescriptor on Apache Spark.

compile

public <Type>
org.apache.spark.api.java.function.Function<Type,Boolean> compile(PredicateDescriptor<Type> predicateDescriptor,
 SparkExecutionOperator operator,
 OptimizationContext.OperatorContext operatorContext,
 ChannelInstance[] inputs)

Create an appropriate Function-based predicate for deploying the given PredicateDescriptor on Apache Spark.

Parameters:

predicateDescriptor - describes the function

operator - that executes the Function; only required if the descriptor describes an ExtendedFunction

operatorContext - contains optimization information for the operator

inputs - that feed the operator; only required if the descriptor describes an ExtendedFunction