All Classes and Interfaces
Class
Description
Abstract
CardinalityPusher
implementation for OperatorAlternative
s and subclasses.Template for
ChannelInstance
implementations.Implements a template for
ReferenceCountable
objects.AbstractTopologicalTraversal<ActivatorType extends AbstractTopologicalTraversal.Activator<ActivationType>,ActivationType extends AbstractTopologicalTraversal.Activation<ActivatorType>>
Encapsulates logic to traverse a
WayangPlan
in a topological, bottom-up manner.Declares that the current traversal should be aborted.
AbstractTopologicalTraversal.Activation<TActivator extends AbstractTopologicalTraversal.Activator<? extends AbstractTopologicalTraversal.Activation<TActivator>>>
Describes a reference to an input of an
AbstractTopologicalTraversal.Activator
.AbstractTopologicalTraversal.Activator<TActivation extends AbstractTopologicalTraversal.Activation<? extends AbstractTopologicalTraversal.Activator<TActivation>>>
Wraps a
CardinalityEstimator
, thereby caching its input CardinalityEstimate
s and keeping track
of its dependent CardinalityEstimator
s.This interface represents any piece of code that takes no input and produces no output but may fail.
Utilities to perform actions.
Operator free of ambiguities.
Created by khayyzy on 5/28/16.
This
OptimizationContext
implementation aggregates several OptimizationContext
s and exposes
their OptimizationContext.OperatorContext
in an aggregated manner.CardinalityEstimator
implementation that can have multiple ways of calculating a CardinalityEstimate
.This descriptor pertains to functions that take multiple data units and aggregate them into a single data unit.
This source reads a blob file stored in Amazon s3 and outputs the lines as data units.
Mapping from
AmazonS3Source
to JavaAmazonS3Source
.This interface represents an application metrics in a Spark cluster.
The
ApplicationEnd
class represents an application that has ended.The ApplicationStart class implements the Application and SerializableObject interfaces.
An atomic execution describes the smallest work unit considered by Wayang's cost model.
This
JsonSerializer
stores the given instances via their Configuration
key, if any, or else by the LoadProfile
that they estimate.JsonSerializer
implementation for AtomicExecutionGroup
s.This source reads a blob file stored in Azure Blob Storage and outputs the lines as data units.
Mapping from
AzureBlobStorageSource
to JavaAzureBlobStorageSource
.A basic data unit type is elementary and not constructed from other data unit types.
Implementation of a battle between to elements.
An operation
Expression
.Wraps a
BinaryOperator
as a Function2
.SparkOperatorProfiler
implementation for SparkExecutionOperator
s with two inputs and one output.This operator has two inputs and a single output.
A mutable bit-mask.
Created by khayyzy on 5/28/16.
Created by khayyzy on 5/28/16.
Describes when to interrupt the execution of an
ExecutionPlan
.Channel
that represents a broadcasted value.build2ListObject<TypeXPivot extends Comparable<TypeXPivot>,TypeXRef extends Comparable<TypeXRef>,Input extends Copyable<Input>>
Created by khayyzy on 5/28/16.
This utility maintains canonical sets of objects.
An estimate of cardinality within a
WayangPlan
expressed as a ProbabilisticIntervalEstimate
.CardinalityEstimator
that subsumes a DAG of operators, each one providing a local CardinalityEstimator
.Computes
CardinalityEstimate
s.Handles the
CardinalityEstimate
s of a WayangPlan
.Stores cardinalities that have been collected by the
CrossPlatformExecutor
.Mapping from
CartesianOperator
to FlinkCartesianOperator
.Mapping from
CartesianOperator
to JavaCartesianOperator
.Mapping from
CartesianOperator
to SparkCartesianOperator
.This operator returns the cartesian product of elements of input datasets.
Models the data movement between to
ExecutionTask
s.This graph contains a set of
ChannelConversion
s.Prefers
ChannelConversionGraph.Tree
s with lower cost.Prefers
ChannelConversionGraph.Tree
s with lower cost.ChannelConversion
s used by the FlinkPlatform
.ChannelConversion
s for the GiraphPlatform
.ChannelConversion
s for the JavaPlatform
.Register for the
ChannelConversion
s supported for this platform.ChannelConversion
s used by the JavaPlatform
.Register for the
ChannelConversion
s supported for this platform.ChannelConversion
s for the TensorflowPlatform
.Describes a certain
Channel
type including further parameters.Represents the actual, allocated resource represented by
Channel
.Encapsulates a
ChannelInstance
in the lazy execution lineage.Mapping from
CoGroupOperator
to FlinkCoGroupOperator
.Mapping from
JoinOperator
to JavaJoinOperator
.Mapping from
CoGroupOperator
to SparkCoGroupOperator
.This operator groups both inputs by some key and then matches groups with the same key.
Provides a
Collection
of objects.This source takes as input a Java
Collection
.Mapping from
CollectionSource
to FlinkCollectionSource
.Mapping from
CollectionSource
to JavaCollectionSource
.Mapping from
CollectionSource
to SparkCollectionSource
.Mapping from
CollectionSource
to TensorflowCollectionSource
.Expression
implementation that represents a function with a static implementation.ExecutionResource
that itself contains ExecutionResource
s.A composite operator can be decomposed into smaller operators.
Created by zoi on 22/1/15.
Describes both the configuration of a
WayangContext
and Job
s.Breakpoint
implementation that disrupts execution if all aggregated Breakpoint
s request a disruption.A constant
Expression
.LoadProfileEstimator
that estimates a predefined LoadProfile
.Used by
Configuration
s to provide some value.Created by bertty on 13-07-17.
Utility to expose interfaces that accept a callback as an
Iterator
.Provides contextual variables and functions in order to evaluate an
Expression
.Implementing objects must be able to provide copies of themselves.
This measurement captures execution costs w.r.t. to Wayang's cost model.
This utility helps to count elements.
Mapping from
CountOperator
to FlinkCountOperator
.Mapping from
CountOperator
to JavaCountOperator
.Mapping from
CountOperator
to SparkCountOperator
.This operator returns the count of elements in this stream.
Executes a (cross-platform)
ExecutionPlan
.Iterates all combinations, i.e., the Cartesian product, of given
Iterable
s.Based on Calcite's CSV enumerator.
Created by khayyzy on 5/28/16.
Created by khayyzy on 5/28/16.
This class describes a database.
Created by khayyzy on 5/28/16.
Utility to create common data generators.
Describes the situation where one
DataSet
is operated on, producing a further DataSet
.A data set is an abstraction of the Wayang programming model.
A grouped data unit type describes just the structure of data units within a grouped dataset.
The type of data types designate the kind of data that are being passed between operators.
Mapping from
DecisionTreeRegressionOperator
to SparkDecisionTreeRegressionOperator
.Default implementation of the
CardinalityEstimator
.Default
CardinalityPusher
implementation.Default implementation of the
ChannelConversion
.Default
Context
implementation that can be configured.Implementation of
LoadEstimator
that uses a single-point cost function.This implementation of
OptimizationContext
represents a direct mapping from OptimizationContext.OperatorContext
to executions of the respective Operator
s.Profiles the reading and writing speed to some directory.
Mapping from
DistinctOperator
to FlinkDistinctOperator
.Mapping from
DistinctOperator
to JavaDistinctOperator
.Mapping from
DistinctOperator
to SparkDistinctOperator
.This operator returns the distinct elements in this dataset.
Mapping from
DLTrainingOperator
to TensorflowDLTrainingOperator
.Mapping from
DoWhileOperator
to FlinkDoWhileOperator
.Mapping from
LoopOperator
to JavaLoopOperator
.Mapping from
LoopOperator
to SparkLoopOperator
.This operator has three inputs and two outputs.
Create a that remove the elements null
Class create a that genereta only null as convertion
EstimationContext
implementation for DynamicLoadEstimator
s.Adjustable
LoadProfileEstimator
implementation.Adjustable
LoadProfileEstimator
implementation.Utility to create
DynamicLoadProfileEstimator
s.This
Plugin
can be arbitrarily customized.An enumeration alternative is embedded within an
EnumerationBranch
.An enumeration branch is the basic unit for enumeration, i.e., translation from a
WayangPlan
to an
ExecutionPlan
.EstimatableCost defines an interface for optimizer cost in terms of
methods to estimate a PlanImplementation's cost.
Provides parameters required by
LoadProfileEstimator
s.Marks a property of an
Operator
that is relevant to the estimation process, i.e., should be provided
in EstimationContext
s.This exception signals a failed
Expression
evaluation.While a function is executed on a certain
Platform
, allows access to some information of the context in
which the function is being executed.Encapsulates
AtomicExecution
s with a common OptimizationContext.OperatorContext
in a lazy execution lineage.Stores execution data have been collected by the
CrossPlatformExecutor
.An execution operator is handled by a certain platform.
Represents an executable, cross-platform data flow.
This
Measurement
encapsulates an ExecutionPlan
.Encapsulates a
Channel
of the ExecutionPlan
.A directed link between an
ExecutionPlanMeasurement.OperatorNode
and a ExecutionPlanMeasurement.ChannelNode
(in any order).Encapsulates a
Channel
of the ExecutionPlan
.Represents a resource allocated for execution that needs to be released manually.
Implements various functionalities of an
ExecutionResource
.Resides within a
PlatformExecution
and represents the minimum execution unit that is controlled by Wayang.This class models the execution equivalent of
LoopSubplan
s.Contains a state of the execution of an
ExecutionTaskFlow
.Serves as an adapter to include
ExecutionOperator
s, which are usually parts of WayangPlan
s, in
ExecutionPlan
s.Graph of
ExecutionTask
s and Channel
s.Creates an
ExecutionTaskFlow
from a PlanImplementation
.Propagates a
Channel
to its consumers.An executor executes
ExecutionOperator
s.The Executor interface represents an executor in a Spark cluster.
Factory for
Executor
s.The ExecutorAdded class represents an executor added event in a distributed computing system.
An event class representing the removal of an executor.
Implements the
ExecutionResource
handling as defined by Executor
.An implementation of the Executor interface that represents an updated executor.
An experiment comprises
Measurement
s from one specific Subject
execution.CollectionProvider
implementation based on a blacklist and a whitelist.A mathematical expression that can be evaluated.
This utility builds
Expression
s from an input String
.Implements a
Function2
that calls ExtendedFunction.open(ExecutionContext)
of its implementation before delegating the very first Function2.call(Object, Object)
.Implements a
FlatMapFunction
that calls ExtendedFunction.open(ExecutionContext)
of its implementation before delegating the very first Function.call(Object)
.Used to enrich regular functions with additional life-cycle methods of
Operator
s.Implements a
Function
that calls ExtendedFunction.open(ExecutionContext)
of its implementation before delegating the very first Function.call(Object)
.Implements a
Function
that calls ExtendedFunction.open(ExecutionContext)
of its implementation before delegating the very first Function.call(Object)
.Wraps a
Function
as a FlatMapFunction
.Implements a
Function
that calls ExtendedFunction.open(ExecutionContext)
of its implementation before delegating the very first Function.call(Object)
.Created by khayyzy on 5/28/16.
Created by khayyzy on 5/28/16.
Assumes with a confidence of 50% that the output cardinality will be somewhere between 1 and the product of
all 10*input estimates.
Represents a
Channel
that is realized via a file/set of files.ChannelDescriptor
for FileChannel
s.Utilities for
FileChannel
s.Abstraction for accessing a file system.
Tool to work with
FileSystem
s.Mapping from
FilterOperator
to FlinkFilterOperator
.Mapping from
FilterOperator
to GenericJdbcFilterOperator
.Mapping from
FilterOperator
to JavaFilterOperator
.Mapping from
FilterOperator
to PostgresFilterOperator
.Mapping from
FilterOperator
to SparkFilterOperator
.Mapping from
FilterOperator
to Sqlite3FilterOperator
.This operator returns a new dataset after filtering by applying predicateDescriptor.
Created by khayyzy on 5/28/16.
Describes when to interrupt the execution of an
ExecutionPlan
.CardinalityEstimator
implementation for Operator
s with a fix-sized output.This descriptor pertains to functions that consume a single data unit and output a group of data units.
Wraps a
Function
as a FlatMapFunction
.Mapping from
FlatMapOperator
to FlinkFlatMapOperator
.Mapping from
FlatMapOperator
to JavaFlatMapOperator
.Mapping from
FlatMapOperator
to SparkFlatMapOperator
.A flatmap operator represents semantics as they are known from frameworks, such as Spark and Flink.
Register for relevant components of this module.
Flink implementation of the
CartesianOperator
.Wrapper of of Flink for use in Wayang
Flink implementation of the
CoGroupOperator
.Converts
DataSetChannel
into a CollectionChannel
This is execution operator implements the
CollectionSource
.Wraps and manages a Flink
ExecutionEnvironment
to avoid steady re-creation.Flink implementation of the
CountOperator
.Flink implementation of the
DistinctOperator
.Flink implementation of the
DoWhileOperator
.ExecutionContext
implementation for the FlinkPlatform
.Execution operator for the Flink platform.
Executor
implementation for the FlinkPlatform
.Flink implementation of the
FilterOperator
.Flink implementation of the
FlatMapOperator
.Flink implementation of the
GlobalMaterializedGroupOperator
.Flink implementation of the
GlobalReduceOperator
.Flink implementation of the
GroupByOperator
.Flink implementation of the
IntersectOperator
.Flink implementation of the
JoinOperator
.Implementation of the
LocalCallbackSink
operator for the Flink platform.Flink implementation of the
RepeatOperator
.Flink implementation of the
MapOperator
.Flink implementation of the
MapPartitionsOperator
.Flink implementation of the
MaterializedGroupByOperator
.Operator
for the FlinkPlatform
that creates a sequence file.Operator
for the FlinkPlatform
that creates a sequence file.Flink implementation of the
PageRankOperator
.A reduce function that takes a sequence of edges and builds the adjacency list for the vertex where the edges
originate.
The function that applies the page rank dampening formula.
Filter that filters vertices where the rank difference is below a threshold.
Join function that distributes a fraction of a vertex's rank to all neighbors.
A map function that assigns an initial rank to all pages.
Platform
for Apache Flink.Flink implementation of the
ReduceByOperator
.Flink implementation of the
RepeatOperator
.Flink implementation of the
RepeatOperator
.Flink implementation of the
SampleOperator
.Flink implementation of the
SortOperator
.Implementation of the
TextFileSink
operator for the Flink platform.Provides a
Collection
to a Flink job.Created by bertty on 31-10-17.
Flink implementation of the
UnionAllOperator
.Flink implementation of the
MapOperator
.Formats different general purpose objects.
Instruments only outbound
Channel
s.CollectionProvider
implementation based on a blacklist and a whitelist.Implementation of
KeyValueProvider
that uses a Function
to provide a value.Used by
Configuration
s to provide some value.A compiler translates Wayang functions into executable Flink functions.
A compiler translates Wayang functions into executable Java functions.
Compiles
FunctionDescriptor
s to SQL clauses.A compiler translates Wayang functions into executable Java functions.
Spark function for building pair RDDs.
Spark function for aggregating data quanta.
Describes functions coming from Wayang, designated for Spark.
A function operates on single data units or collections of those.
Extends a
FunctionDescriptor.SerializableBinaryOperator
to an ExtendedFunction
.Extends a
FunctionDescriptor.SerializableConsumer
to an ExtendedFunction
.Extends a
FunctionDescriptor.SerializableFunction
to an ExtendedFunction
.Decorates the default
Function
with Serializable
, which is required by some distributed frameworks.Decorates the default
Function
with Serializable
, which is required by some distributed frameworks.Decorates the default
Consumer
with Serializable
, which is required by some distributed frameworks.Decorates the default
Function
with Serializable
, which is required by some distributed frameworks.Register for the
ChannelConversion
s supported for this platform.Register for relevant components of this module.
Executor
implementation for the GenericJdbcPlatform
.GenericJdbc implementation of the
FilterOperator
.Platform
implementation for GenericJdbc.GenericJdbc implementation of the
FilterOperator
.Implementation of the genetic optimization technique for finding good
LoadProfileEstimator
s.This app tries to infer good
LoadProfileEstimator
s for ExecutionOperator
s using data from an
ExecutionLog
.Register for relevant components of this module.
ExecutionOperator
that can be run by the GiraphExecutor
.Executor
for the GiraphPlatform
.PageRank
Operator
implementation for the GiraphPlatform
.Giraph
Platform
for Wayang.This
Plugin
activates default capabilities of the GiraphPlatform
.Mapping from
GlobalMaterializedGroupOperator
to FlinkGlobalMaterializedGroupOperator
.Mapping from
GlobalMaterializedGroupOperator
to JavaGlobalMaterializedGroupOperator
.Mapping from
GlobalMaterializedGroupOperator
to SparkGlobalMaterializedGroupOperator
.This operator groups the elements of a data set into a single data quantum.
This mapping detects combinations of the
GroupByOperator
and ReduceOperator
and merges them into
a single ReduceByOperator
.Mapping from
GlobalReduceOperator
to FlinkGlobalReduceOperator
.Mapping from
GlobalReduceOperator
to JavaGlobalReduceOperator
.Mapping from
GlobalReduceOperator
to SparkGlobalReduceOperator
.This operator groups the elements of a data set and aggregates the groups.
Mapping from
GoogleCloudStorageSource
to JavaGoogleClodStorageSource
.Mapping from
GroupByOperator
to FlinkGroupByOperator
.This is the auxiliary GroupBy operator, i.e., it behaves differently depending on its context.
Grouping key used in Query 1.
FileSystem
immplementation for the HDFS.TODO: Implement
Mapping from
IEJoinOperator
to JavaIEJoinOperator
.Mapping from
IEJoinOperator
to SparkIEJoinOperator
.This operator decides the correct sorting orders for IEJoin
Created by khayyzy on 5/19/16.
This operator applies inequality join on elements of input datasets.
Mapping from
IESelfJoinOperator
to JavaIESelfJoinOperator
.Mapping from
IESelfJoinOperator
to SparkIESelfJoinOperator
.This operator applies inequality self join on elements of input datasets.
Context for the optimization of
LoadProfileEstimator
s.An input slot declares an input of an
Operator
.Determines, which
Channel
s in an ExecutionPlan
should be instrumented.Mapping from
IntersectOperator
to FlinkIntersectOperator
.Mapping from
IntersectOperator
to JavaIntersectOperator
.Mapping from
IntersectOperator
to SparkIntersectOperator
.This operator returns the set intersection of elements of input datasets.
Implementation of
LoadEstimator
that uses a interval-based cost function.Utilities for the work with
Iterator
s.Register for relevant components of this module.
This is execution operator that implements the
AmazonS3Source
.This is execution operator that implements the
AzureBlobStorageSource
.Java implementation of the
CartesianOperator
.Defines execution logic to handle a
Channel
.Java implementation of the
CoGroupOperator
.This is execution operator implements the
TextFileSource
.Converts
StreamChannel
into a CollectionChannel
Java implementation of the
CountOperator
.Java implementation of the
DistinctOperator
.Java implementation of the
DoWhileOperator
.ExecutionContext
implementation for the JavaPlatform
.Execution operator for the Java platform.
Executor
implementation for the JavaPlatform
.Java implementation of the
FilterOperator
.Java implementation of the
FlatMapOperator
.Java implementation of the
GlobalMaterializedGroupOperator
.Java implementation of the
GlobalReduceOperator
.Java implementation of the
IEJoinOperator
.Java implementation of the
IESelfJoinOperator
.Java implementation of the
IntersectOperator
.Java implementation of the
JoinOperator
.Implementation fo the
KafkaTopicSink
for the JavaPlatform
.Implementation of the
LocalCallbackSink
operator for the Java platform.Java implementation of the
LoopOperator
.Java implementation of the
MapOperator
.Java implementation of the
MapPartitionsOperator
.Java implementation of the
MaterializedGroupByOperator
.Operator
for the JavaPlatform
that creates a sequence file.Utility to chunk a
Stream
into portions of fixed size.Operator
for the JavaPlatform
that creates a sequence file.Java implementation of the
PageRankOperator
.This is execution operator implements the
ParquetSource
.Java implementation of the
JavaRandomSampleOperator
.Java implementation of the
ReduceByOperator
.Java implementation of the
DoWhileOperator
.Java implementation of the
JavaReservoirSampleOperator
.Java implementation of the
SortOperator
.Implementation fo the
TextFileSink
for the JavaPlatform
.This is execution operator implements the
TextFileSource
.OperatorProfiler
for sources.Operator
for the JavaPlatform
that creates a TSV file.Operator
for the JavaPlatform
that creates a sequence file.Java implementation of the
UnionAllOperator
.Executor
implementation for the JdbcPlatformTemplate
.Template for JDBC-based
FilterOperator
.PostgreSQL implementation for the
JoinOperator
.Platform
implementation for a JDBC-accessible database.Projects the fields of
Record
s.Implementation of
Schema
that is backed by a JDBC data source.Schema factory that creates a
JdbcSchema
.Do not use.
Queryable that gets its data from a table within a JDBC connection.
PostgreSQL implementation for the
TableSource
.Describes a job that is to be executed using Wayang.
The Job interface represents a job to be executed in a distributed system.
JobEnd class represents a job's end in a system.
JobStart class represents a job's start in a system.
Mapping from
JoinOperator
to FlinkJoinOperator
.Mapping from
JoinOperator
to JavaJoinOperator
.Mapping from
JoinOperator
to PostgresJoinOperator
.Mapping from
JoinOperator
to SparkJoinOperator
.This operator returns the cartesian product of elements of input datasets.
This interface prescribes implementing instances to be able to provide itself as a
WayangJsonObj
.A
JsonSerializer
implementation to serialize JsonSerializable
s.Utility to deal with
JsonSerializable
s.Alternative to
JsonSerializable
: Externalizes serialization logic.Utilities to deal with JUEL expressions.
Describes the implementation of one
OutputSlot
to its occupied InputSlot
s.This
UnarySink
writes all incoming data quanta to a single Kafka topic.Mapping from
KafkaTopicSink
to JavaKafkaTopicSink
.Mapping from
KafkaTopicSink
to SparkKafkaTopicSink
.This source reads messages from a Kafka topic and outputs the messages as data units.
Mapping from
KafkaTopicSource
to JavaKafkaTopicSource
.Mapping from
KafkaTopicSource
to SparkKafkaTopicSource
.Wrapper for
Wrapper for
KeySelector
Used by
Configuration
s to provide some value.Mapping from
KMeansOperator
to SparkKMeansOperator
.This
PlanEnumerationPruningStrategy
follows the idea that we can prune a
PlanImplementation
, when there is a further one that is (i) better and (ii) has the exact same
operators with still-to-be-connected Slot
s.A node wraps a
ChannelInstance
and keeps track of predecessor nodes.Callback interface for traversals of
LazyExecutionLineageNode
s, thereby accumulating the callback return values.LazyExecutionLineageNode.Aggregator
implementation that collects all visited LazyExecutionLineageNode
contents.InputStream
that is trimmed to a specified size.Mapping from
LinearRegressionOperator
to SparkLinearRegressionOperator
.Mapping from
LinearSVCOperator
to SparkLinearSVCOperator
.A tuple of the lineitem table.
Parses a
LineItemTuple
from a given CSV line (double quoted, comma-separated).Created by khayyzy on 5/28/16.
An estimate of costs of some executable code expressed as a
ProbabilisticIntervalEstimate
.Describes the resource utilization of something executable within an
EstimationContext
, such as an
ExecutionOperator
or a FunctionDescriptor
.Functional interface for lambda expressions to express single-point load estimation functions.
Reflects the (estimated) required resources of an
Operator
or FunctionDescriptor
.Estimates the
LoadProfile
of some executable artifact that takes some input data quanta and produces them.Utilities to deal with
LoadProfileEstimator
s.Calculates a
TimeEstimate
for a link LoadProfile
.Converts a
LoadEstimate
into a time estimate.This sink executes a callback on each received data unit into a Java
Collection
.Mapping from
LocalCallbackSink
to FlinkLocalCallbackSink
.Mapping from
LocalCallbackSink
to JavaLocalCallbackSink
.Mapping from
LocalCallbackSink
to SparkLocalCallbackSink
.FileSystem
implementation for the local file system.Evaluates a
Configuration
on a ExecutionLog
.Provide logging via
Logger
.An ML4all logical operator
Logical Operator that uses the [[org.apache.wayang.ml4all.abstraction.plan.ML4allModel]] as input besides the data flowing from its input slot.
Mapping from
LogisticRegressionOperator
to SparkLogisticRegressionOperator
.Enumerator for
LoopSubplan
s.Special
OperatorAlternative
for LoopHeadOperator
s.CardinalityPusher
implementation for LoopHeadAlternative
s.Head of a
LoopSubplan
.LoopHeadOperator
s can be stateful because they might be executed mulitple times.Describes the enumeration of a
LoopSubplan
.Goes over a
WayangPlan
and isolates its loops.Mapping from
LoopOperator
to FlinkLoopOperator
.Mapping from
LoopOperator
to JavaLoopOperator
.Mapping from
LoopOperator
to SparkLoopOperator
.This operator has three inputs and three outputs.
Wraps a loop of
Operator
s.CardinalityPusher
implementation for LoopSubplan
s.Key-value cache with "least recently used" eviction strategy.
Starts a profiling run of Spark.
Implementation of
KeyValueProvider
that uses a Map
to provide a value.Wraps a
Function
as a Function
.Mapping from
MapOperator
to FlinkMapOperator
.Mapping from
MapOperator
to JavaMapOperator
.Mapping from
MapOperator
to SparkMapOperator
.A map operator represents semantics as they are known from frameworks, such as Spark and Flink.
This descriptor pertains to functions that consume and output multiple data quanta.
Wraps a
Function
as a FlatMapFunction
.Mapping from
MapPartitionsOperator
to FlinkMapPartitionsOperator
.Mapping from
MapPartitionsOperator
to JavaMapPartitionsOperator
.Mapping from
MapPartitionsOperator
to SparkMapPartitionsOperator
.This operator takes as input potentially multiple input data quanta and outputs multiple input data quanta.
A mapping issues a set of
PlanTransformation
s that make up the complete mapping.Register for the components provided in the basic plugin.
Register for
Mapping
s for this platform.Register for the
Mapping
s supported for this platform.Register for
Mapping
s for this platform.Mapping
s for the IEJoinOperator
.Register for
Mapping
s for this platform.Register for the
Mapping
s supported for this platform.Register for
Mapping
s for this platform.Register for the
Mapping
s supported for this platform.Register for
Mapping
s for this platform.This mapping translates the
GroupByOperator
into the MaterializedGroupByOperator
.Mapping from
MaterializedGroupByOperator
to FlinkMaterializedGroupByOperator
.Mapping from
MaterializedGroupByOperator
to JavaMaterializedGroupByOperator
.Mapping from
MaterializedGroupByOperator
to SparkMaterializedGroupByOperator
.This operator collocates the data units in a data set w.r.t. a key function.
This class provides an empty implementation of
MathExListener
,
which can be extended to create a listener which only needs to handle a subset
of the available methods.This class provides an empty implementation of
MathExVisitor
,
which can be extended to create a visitor which only needs to handle a subset
of the available methods.This exception signals a failed
Expression
evaluation.This interface defines a complete listener for a parse tree produced by
MathExParser
.This interface defines a complete generic visitor for a parse tree produced
by
MathExParser
.Measurement captures the value of a metric at a specific time
Custom deserializer for
Measurement
s
Detects actual subclass of serialized instances and then delegates the deserialization to that subtype.Custom serializer for
Measurement
s
Detects actual subclass of given instances, encodes this class membership, and then delegates serialization to that subtype.For keeping global variables to access in the logical operators.
Wayang physical plan for ML4all algorithms
A Type that represents a ML model
Deprecated.
Deprecated.
Flattens Tuple2<Record, Record> to Record
Maps keys to multiple values.
Created by khayyzy on 5/28/16.
Created by khayyzy on 5/28/16.
Expression
implementation that represents a function that is identified
via its name.LoadProfileEstimator
that can host further LoadProfileEstimator
s.Instruments only outbound
Channel
s.This
Breakpoint
implementation always requests a break unless inside of ExecutionStageLoop
s.This
UnarySink
writes all incoming data quanta to a Object file.Mapping from
ObjectFileSink
to FlinkObjectFileSink
.Mapping from
ObjectFileSink
to JavaObjectFileSink
.Mapping from
ObjectFileSink
to SparkObjectFileSink
.This source reads a text file and outputs the lines as data units.
Mapping from
ObjectFileSource
to FlinkObjectFileSource
.Mapping from
ObjectFileSource
to JavaObjectFileSource
.Mapping from
ObjectFileSource
to SparkObjectFileSource
.Method wrapper that ensures that it is called only once.
An operator is any node that within a
WayangPlan
.This operator encapsulates operators that are alternative to each other.
CardinalityPusher
implementation for OperatorAlternative
s.Helper class for the implementation of the
Operator
interface.Utility to de/serialize
Operator
s.This is not an
Operator
in its own right.Utilities to deal with
OperatorContainer
s.An operator match correlates an
OperatorPattern
to an actually matched Operator
.An operator pattern matches to a class of operator instances.
Allows to instrument an
JavaExecutionOperator
.The result of a single profiling run.
Utilities to create
OperatorProfiler
instances.Utilities to create
SparkOperatorProfiler
instances.Utility class for
Operator
s.Manages contextual information required during the optimization of a
WayangPlan
.Context for the optimization of
LoadProfileEstimator
s.Utility methods for the optimization process.
Utility similar to
Optional
.Instruments only outbound
Channel
s.Wrapper for
An output slot declares an output of an
Operator
.Basic PageRank implementation.
Master compute associated with
PageRankAlgorithm
.Simple VertexInputFormat that supports
PageRankAlgorithm
Simple VertexOutputFormat that supports
PageRankAlgorithm
Simple VertexReader that supports
PageRankAlgorithm
Worker context used with
PageRankAlgorithm
.Mapping from
PageRankOperator
to FlinkPageRankOperator
.Maps
PageRankOperator
s to GiraphPageRankOperator
s.Mapping from
PageRankOperator
to JavaPageRankOperator
.Mapping from
PageRankOperator
to org.apache.wayang.spark.operators.graph.SparkPageRankOperator .Creates intial page ranks.
Operator
for the PageRank algorithm.Parameters for Basic PageRank implementation.
This source reads a parquet file and outputs the lines as
Record
units.Mapping from
ParquetSource
to JavaParquetSource
.Mapping from
CollectionSource
to SparkCollectionSource
.This exception signals a failed
Expression
evaluation.Captures data of a execution of a set of
ExecutionOperator
s.JsonSerializer
implementation for PartialExecution
s.This
Measurement
adapts a PartialExecutionMeasurement
.Represents a collection of
PlanImplementation
s that all implement the same section of a WayangPlan
(which
is assumed to contain OperatorAlternative
s in general).A strategy to prune
PlanImplementation
s from a PlanEnumeration
.The plan partitioner recursively dissects a
WayangPlan
into PlanEnumeration
s and then assembles
them.An
Operator
can be activated as soon as all of its inputs are available.Represents a partial execution plan.
This class collects metrics for
WayangPlan
s.Traverse a plan.
A callback can be invoked during a plan traversal on each traversed node.
A callback can be invoked during a plan traversal on each traversed node.
A platform describes an execution engine that executes
ExecutionOperator
s.Complete data flow on a single platform, that consists of multiple
ExecutionStage
s.A plugin contributes the following components to a
WayangContext
:
Mapping
s,
ChannelConversion
s, and
Configuration
properties.
In turn, it may require several Platform
s for its operation.Register for relevant components of this module.
PostgreSQL implementation of the
FilterOperator
.PostgreSQL implementation of the
JoinOperator
.Platform
implementation for SQLite3.PostgreSQL implementation of the
FilterOperator
.PostgreSQL implementation for the
TableSource
.Wraps a
Predicate
as a Function
.This descriptor pertains to predicates that consume a single data unit.
Mapping from
PredictOperator
to SparkPredictOperator
.Mapping from
PredictOperator
to TensorflowPredictOperator
.An value representation that is capable of expressing uncertainty.
An estimate that is capable of expressing uncertainty.
This class provides facilities to save and load
Experiment
s.Utilities to work with
ProfileDB
s.Utility to support finding reasonable
LoadProfileEstimator
s for JavaExecutionOperator
s.Utilities to fake Wayang internals etc..
This descriptor pertains to projections.
/**
Mapping from
MapOperator
to GenericJdbcProjectionOperator
./**
Mapping from
MapOperator
to PostgresProjectionOperator
.Mapping from
MapOperator
to Sqlite3ProjectionOperator
.Executor
implementation that employs a push model, i.e., data quanta are "pushed"
through the ExecutionStage
.Describes the situation where one
JavaRDD
is operated on, producing a further JavaRDD
.A Type that represents a record with a schema, might be replaced with
something standard like JPA entity.
This is a specific
BasicDataUnitType
for Record
s.This mapping detects combinations of the
GroupByOperator
and ReduceOperator
and merges them into
a single ReduceByOperator
.Mapping from
ReduceByOperator
to FlinkReduceByOperator
.Mapping from
ReduceByOperator
to JavaReduceByOperator
.Mapping from
ReduceByOperator
to SparkReduceByOperator
.This operator groups the elements of a data set and aggregates the groups.
This descriptor pertains to functions that take multiple data units and aggregate them into a single data unit
by means of a tree-like fold, i.e., using a commutative, associative function..
This operator is context dependent: after a
GroupByOperator
, it is meant to be a ReduceByOperator
;
otherwise, it is a GlobalReduceOperator
.This interface provides a reference-counting scheme, e.g., to manage allocated external resources.
Utilities for reflection code.
Mapping from
RepeatOperator
to FlinkRepeatOperator
.Mapping from
RepeatOperator
to JavaRepeatOperator
.Mapping from
RepeatOperator
to SparkRepeatOperator
.This factory takes an
SubplanMatch
and derives a replacement Subplan
from it.Implementation of the
ReplacementSubplanFactory
that replaces exactly one Operator
with exactly one Operator
,
where both have the exact same InputSlot
s and OutputSlot
in the exact same order.
Utility to do reservoir sampling with competitions over shared space.
Tuple that is returned by Query 1.
Utility to read from an RRD file.
Mapping from
SampleOperator
to FlinkSampleOperator
.Mapping from
SampleOperator
to JavaRandomSampleOperator
.Mapping from
SampleOperator
to SparkRandomPartitionSampleOperator
.A random sample operator randomly selects its inputs from the input slot and pushes that element to the output slot.
Interface that describes a sampling algorithm that can give bias towards certain elements..
This class checks a
WayangPlan
for several sanity criteria:
Subplan
s must only be used as top-level Operator
of OperatorAlternative.Alternative
Subplan
s must contain more than one Operator
The
SerializableObject
interface is a marker interface that indicates
that its implementing classes are serializable.This class executes a stochastic gradient descent optimization on Apache Wayang (incubating).
This class executes a stochastic gradient descent optimization on Apache Wayang (incubating), just like
SGDImpl
.This
EstimationContext
implementation just stores all required variables without any further logic.This
PlanEnumerationPruningStrategy
retains only the best PlanImplementation
s employing a single
Platform
only.OperatorProfiler
implementation for sinks.SparkOperatorProfiler
implementation for SparkExecutionOperator
s with one input and no outputs.Abstract class for inputs and outputs to operators.
This mapping can be used to encapsulate subplans by connecting slots (usually against the data flow direction,
i.e., outer output slot to inner output slot, inner input slot to outer input slot).
Mapping from
SortOperator
to FlinkSortOperator
.Mapping from
SortOperator
to JavaSortOperator
.Mapping from
SortOperator
to SparkSortOperator
.This operator sorts the elements in this dataset.
OperatorProfiler
implementation for sinks.Register for relevant components of this module.
Spark implementation of the
SparkBernoulliSampleOperator
.Takes care of creating a
Broadcast
that can be used later on.Converts an uncached
RddChannel
into a cached RddChannel
.Spark implementation of the
CartesianOperator
.Spark implementation of the
JoinOperator
.Provides a
Collection
to a Spark job.SparkOperatorProfiler
for the SparkTextFileSource
.Wraps and manages a
JavaSparkContext
to avoid steady re-creation.This
Plugin
enables to create RDD
s.Spark implementation of the
CountOperator
.Spark implementation of the
DistinctOperator
.Spark implementation of the
DoWhileOperator
.ExecutionContext
implementation for the SparkPlatform
.Execution operator for the
SparkPlatform
.Executor
implementation for the SparkPlatform
.Spark implementation of the
FilterOperator
.Spark implementation of the
FlatMapOperator
.Spark implementation of the
GlobalMaterializedGroupOperator
.Spark implementation of the
GlobalReduceOperator
.SparkIEJoinOperator<Type0 extends Comparable<Type0>,Type1 extends Comparable<Type1>,Input extends Copyable<Input>>
Spark implementation of the
IEJoinOperator
.SparkIESelfJoinOperator<Type0 extends Comparable<Type0>,Type1 extends Comparable<Type1>,Input extends Copyable<Input>>
Spark implementation of the
IESelfJoinOperator
.Spark implementation of the
JoinOperator
.Spark implementation of the
JoinOperator
.Implementation of the
KafkaTopicSink
operator for the Spark platform.Provides a
Collection
to a Spark job.A Spark listener implementation that captures events generated during the Spark job execution
and sends them to a Kafka topic for further processing.
Implementation of the
LocalCallbackSink
operator for the Spark platform.Spark implementation of the
LoopOperator
.Spark implementation of the
MapOperator
.Spark implementation of the
MapPartitionsOperator
.Spark implementation of the
MaterializedGroupByOperator
.Deprecated.
Operator
for the SparkPlatform
that creates a sequence file.Operator
for the SparkPlatform
that creates a sequence file.Allows to instrument an
SparkExecutionOperator
.The result of a single profiling run.
Provides a
Collection
to a Spark job.Platform
for Apache Spark.Spark implementation of the
SampleOperator
.Spark implementation of the
ReduceByOperator
.Spark implementation of the
RepeatOperator
.Spark implementation of the
SparkShufflePartitionSampleOperator
.Spark implementation of the
SortOperator
.SparkOperatorProfiler
implementation for SparkExecutionOperator
s with one input and one output.Implementation of the
TextFileSink
operator for the Spark platform.Provides a
Collection
to a Spark job.SparkOperatorProfiler
for the SparkTextFileSource
.Operator
for the SparkPlatform
that creates a TSV file.Operator
for the SparkPlatform
that creates a sequence file.SparkOperatorProfiler
implementation for SparkExecutionOperator
s with one input and one output.Spark implementation of the
UnionAllOperator
.Spark implementation of the
MapOperator
.Register for relevant components of this module.
Implementation of the
FilterOperator
for the Sqlite3Platform
.Platform
implementation for SQLite3.Implementation of the
JdbcProjectionOperator
for the Sqlite3Platform
.Implementation of the
TableSource
for the Sqlite3Platform
.Implementation of a
Channel
that is given by a SQL query.Describes a specific class of
SqlQueryChannel
s belonging to a certain JdbcPlatformTemplate
.This interface represents a stage in a data processing pipeline.
Builds an
ExecutionPlan
from a ExecutionTaskFlow
.Criterion to futher split
StageAssignmentTraversal.InterimStage
besides precedence.Represents a completed stage in a distributed computing system.
The StageExecutorMetrics class implements the Stage and SerializableObject interfaces and represents
the metrics associated with a stage executed by an executor.
The StageSubmitted class implements the Stage and SerializableObject interfaces to represent a submitted stage in a distributed system.
Utility to create
TimeMeasurement
s for an Experiment
.Controls how conducted experiments will be persisted and loaded
The subject of an
Experiment
, e.g., an application or algorithm.A subplan encapsulates connected operators as a single operator.
A subplan match correlates a
SubplanPattern
with its actually matched .A subplan pattern describes a class of subplans in a
WayangPlan
.UnarySource
that provides the tuples from a database table.The Task interface represents a task in a distributed computing or data processing system.
The TaskStatusForRunning enum represents the possible statuses of a task while it is running.
Represents the end status of a task execution.
Represents the getting results of a task execution.
The TaskMetric interface defines the methods used to set and retrieve
performance metrics for a given task.
This class represents the metrics for a task in the Apache Wayang monitoring system.
Represents the start status of a task execution.
Register for relevant components of this module.
TensorflowBatchNorm2D<T extends org.tensorflow.types.family.TNumber>
TensorflowBatchNorm3D<T extends org.tensorflow.types.family.TNumber>
Provides a
Collection
to a Tensorflow job.Converts
TensorChannel
into a CollectionChannel
TensorflowConv2D<T extends org.tensorflow.types.family.TNumber>
TensorflowConv3D<T extends org.tensorflow.types.family.TNumber>
TensorflowConvLSTM2D<T extends org.tensorflow.types.family.TNumber>
TensorflowConvLSTM2D.Cell<T extends org.tensorflow.types.family.TNumber>
Execution operator for the
TensorflowPlatform
.Executor
implementation for the TensorflowPlatform
.TensorflowLinear<T extends org.tensorflow.types.family.TNumber>
Platform
for TensorflowThis
UnarySink
writes all incoming data quanta to a text file.Mapping from
TextFileSink
to FlinkTextFileSink
.Mapping from
TextFileSink
to JavaTextFileSink
.Mapping from
CollectionSource
to SparkCollectionSource
.This source reads a text file and outputs the lines as data units.
Mapping from
TextFileSource
to FlinkTextFileSource
.Mapping from
TextFileSource
to JavaTextFileSource
.Mapping from
CollectionSource
to SparkCollectionSource
.An estimate of time (in milliseconds) expressed as a
ProbabilisticIntervalEstimate
.A
Measurement
that captures a certain amount of time in milliseconds.This (linear) converter turns
TimeEstimate
s into cost estimates.Visitor (as in the Visitor Pattern) for
WayangPlan
s.Sampling strategy that simulates a tournament between elements.
Main class for the TPC-H app based on Apache Wayang (incubating).
This descriptor pertains to functions that consume a single data unit and output a single data unit.
A helper data structure to manage two values without creating new classes.
A type for tuples.
A type for tuples.
An operation
Expression
.Abstract base-class for sinks with a single input.
Abstract base class for sources with a single output.
This operator has a single input and a single output.
Mapping from
UnionAllOperator
to FlinkUnionAllOperator
.Mapping from
UnionAllOperator
to JavaUnionAllOperator
.Mapping from
UnionAllOperator
to SparkUnionAllOperator
.This
Operator
creates the union (bag semantics) of two .Used by
Configuration
s to provide some value.A variable
Expression
A variable that can be altered by an optimization algorithm.
Class create a that generate aggregatorWrapper
Utility for handling arrays.
Activator for the basic Wayang package.
Activator for graph operations being executed with Wayang's basic operators.
Register for plugins in the module.
Utilities to operate
Collection
s.This is the entry point for users to work with Wayang.
Class create a that generate aggregatorWrapper
Exception that declares a problem of Wayang.
Wrapper for
FileOutputFormat
Class create a for use inside of the LoopOperators
JSONArray is the wrapper for the
ArrayNode
to enable the
easy access to Json data
TODO: the java doc is not done because is missing implementation and it performed some
modification on the codeJSONObject is the wrapper for the
ObjectNode
to enable the
easy access to the json data
TODO: the java doc is not done because is missing implementation and it performed some
modification on the codeIs a Wrapper for used in the criterion of the Loops
A Wayang plan consists of a set of
Operator
s.Rule that converts
LogicalSort
to Wayang convention
WayangSort
Implementation of
Value
of flink for use in WayangExample Apache Wayang (incubating) App that does a word count -- the Hello World of Map/Reduce-like systems.
TODO: Implement
Mapping from
ZipWithIdOperator
to FlinkZipWithIdOperator
.Mapping from
ZipWithIdMapping
to a subplan.Mapping from
ZipWithIdOperator
to SparkZipWithIdOperator
.This operators attaches a unique ID to each input data quantum.