Package org.apache.wayang.spark.compiler

Class FunctionCompiler

java.lang.Object

org.apache.wayang.spark.compiler.FunctionCompiler

public class FunctionCompiler
extends java.lang.Object

A compiler translates Wayang functions into executable Java functions.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	FunctionCompiler.KeyExtractor<T,K>	Spark function for building pair RDDs.
static class	FunctionCompiler.Reducer<Type>	Spark function for aggregating data quanta.
static interface	FunctionCompiler.WayangSparkFunction	Describes functions coming from Wayang, designated for Spark.

Constructor Summary

Constructors

Constructor	Description
FunctionCompiler()

Method Summary

Modifier and Type	Method	Description
<I,O> org.apache.spark.api.java.function.FlatMapFunction<I,O>	compile(FlatMapDescriptor<I,O> descriptor, SparkExecutionOperator operator, OptimizationContext.OperatorContext operatorContext, ChannelInstance[] inputs)	Create an appropriate FlatMapFunction for deploying the given FlatMapDescriptor on Apache Spark.
<I,O> org.apache.spark.api.java.function.FlatMapFunction<java.util.Iterator<I>,O>	compile(MapPartitionsDescriptor<I,O> descriptor, SparkExecutionOperator operator, OptimizationContext.OperatorContext operatorContext, ChannelInstance[] inputs)	Create an appropriate Function for deploying the given MapPartitionsDescriptor on Apache Spark's AbstractJavaRDDLike.mapPartitions(FlatMapFunction).
<Type> org.apache.spark.api.java.function.Function<Type,java.lang.Boolean>	compile(PredicateDescriptor<Type> predicateDescriptor, SparkExecutionOperator operator, OptimizationContext.OperatorContext operatorContext, ChannelInstance[] inputs)	Create an appropriate Function-based predicate for deploying the given PredicateDescriptor on Apache Spark.
<T> org.apache.spark.api.java.function.Function2<T,T,T>	compile(ReduceDescriptor<T> descriptor, SparkExecutionOperator operator, OptimizationContext.OperatorContext operatorContext, ChannelInstance[] inputs)	Create an appropriate Function for deploying the given ReduceDescriptor on Apache Spark.
<I,O> org.apache.spark.api.java.function.Function<I,O>	compile(TransformationDescriptor<I,O> descriptor, SparkExecutionOperator operator, OptimizationContext.OperatorContext operatorContext, ChannelInstance[] inputs)	Create an appropriate Function for deploying the given TransformationDescriptor on Apache Spark.
<T,K> FunctionCompiler.KeyExtractor<T,K>	compileToKeyExtractor(TransformationDescriptor<T,K> descriptor)	Compile a key extraction.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FunctionCompiler

public FunctionCompiler()

Method Detail

compile

public <I,O> org.apache.spark.api.java.function.Function<I,O> compile(TransformationDescriptor<I,O> descriptor,
                                                                                  SparkExecutionOperator operator,
                                                                                  OptimizationContext.OperatorContext operatorContext,
                                                                                  ChannelInstance[] inputs)

Create an appropriate Function for deploying the given TransformationDescriptor on Apache Spark.

Parameters:

descriptor - describes the transformation function

operator - that executes the Function; only required if the descriptor describes an ExtendedFunction

operatorContext - contains optimization information for the operator

inputs - that feed the operator; only required if the descriptor describes an ExtendedFunction

compile

public <I,O> org.apache.spark.api.java.function.FlatMapFunction<java.util.Iterator<I>,O> compile(MapPartitionsDescriptor<I,O> descriptor,
                                                                                                             SparkExecutionOperator operator,
                                                                                                             OptimizationContext.OperatorContext operatorContext,
                                                                                                             ChannelInstance[] inputs)

Create an appropriate Function for deploying the given MapPartitionsDescriptor on Apache Spark's AbstractJavaRDDLike.mapPartitions(FlatMapFunction).

Parameters:

descriptor - describes the function

operator - that executes the Function; only required if the descriptor describes an ExtendedFunction

operatorContext - contains optimization information for the operator

inputs - that feed the operator; only required if the descriptor describes an ExtendedFunction

compileToKeyExtractor

public <T,K> FunctionCompiler.KeyExtractor<T,K> compileToKeyExtractor(TransformationDescriptor<T,K> descriptor)

Compile a key extraction.

Returns:

a compiled function

compile

public <I,O> org.apache.spark.api.java.function.FlatMapFunction<I,O> compile(FlatMapDescriptor<I,O> descriptor,
                                                                                         SparkExecutionOperator operator,
                                                                                         OptimizationContext.OperatorContext operatorContext,
                                                                                         ChannelInstance[] inputs)

Create an appropriate FlatMapFunction for deploying the given FlatMapDescriptor on Apache Spark.

Parameters:

descriptor - describes the function

operator - that executes the Function; only required if the descriptor describes an ExtendedFunction

operatorContext - contains optimization information for the operator

inputs - that feed the operator; only required if the descriptor describes an ExtendedFunction

compile

public <T> org.apache.spark.api.java.function.Function2<T,T,T> compile(ReduceDescriptor<T> descriptor,
                                                                                   SparkExecutionOperator operator,
                                                                                   OptimizationContext.OperatorContext operatorContext,
                                                                                   ChannelInstance[] inputs)

Create an appropriate Function for deploying the given ReduceDescriptor on Apache Spark.

compile

public <Type> org.apache.spark.api.java.function.Function<Type,java.lang.Boolean> compile(PredicateDescriptor<Type> predicateDescriptor,
                                                                                                SparkExecutionOperator operator,
                                                                                                OptimizationContext.OperatorContext operatorContext,
                                                                                                ChannelInstance[] inputs)

Create an appropriate Function-based predicate for deploying the given PredicateDescriptor on Apache Spark.

Parameters:

predicateDescriptor - describes the function

operator - that executes the Function; only required if the descriptor describes an ExtendedFunction

operatorContext - contains optimization information for the operator

inputs - that feed the operator; only required if the descriptor describes an ExtendedFunction