Package org.apache.wayang.basic.operators

Class KafkaTopicSource

java.lang.Object

org.apache.wayang.core.plan.wayangplan.OperatorBase

org.apache.wayang.core.plan.wayangplan.UnarySource<String>

org.apache.wayang.basic.operators.KafkaTopicSource

All Implemented Interfaces:

Serializable, ActualOperator, ElementaryOperator, Operator

Direct Known Subclasses:

JavaKafkaTopicSource, SparkKafkaTopicSource

public class KafkaTopicSource
extends UnarySource<String>

This source reads messages from a Kafka topic and outputs the messages as data units.

See Also:

• Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected class	KafkaTopicSource.CardinalityEstimator

Nested classes/interfaces inherited from class org.apache.wayang.core.plan.wayangplan.OperatorBase

OperatorBase.GsonSerializer

Field Summary

Fields inherited from class org.apache.wayang.core.plan.wayangplan.OperatorBase

inputSlots, outputSlots, STANDARD_OPERATOR_ARGS

Fields inherited from interface org.apache.wayang.core.plan.wayangplan.Operator

FIRST_EPOCH

Constructor Summary

Constructors

Constructor	Description
KafkaTopicSource(String topicName)
KafkaTopicSource(String topicName, String encoding)
KafkaTopicSource(KafkaTopicSource that)	Copies an instance (exclusive of broadcasts).

Method Summary

Modifier and Type	Method	Description
Optional<CardinalityEstimator>	createCardinalityEstimator(int outputIndex, Configuration configuration)	Provide a CardinalityEstimator for the OutputSlot at outputIndex.
org.apache.kafka.clients.consumer.KafkaConsumer<String,String>	getConsumer()
org.apache.kafka.clients.consumer.KafkaConsumer<String,String>	getConsumer(Properties props)
static Properties	getDefaultProperties()
String	getEncoding()
String	getTopicName()
void	initConsumer(KafkaTopicSource kts)
static Properties	loadConfig(String propertiesFilePath)	Load properties from a properties file or alternatively use the default properties with some sensitive values from environment variables.
void	startConsuming()

Methods inherited from class org.apache.wayang.core.plan.wayangplan.UnarySource

getOutput, getType

Methods inherited from class org.apache.wayang.core.plan.wayangplan.OperatorBase

accept, addBroadcastInput, addTargetPlatform, at, collectMappedInputSlots, collectMappedOutputSlots, copy, createCopy, getAllInputs, getAllOutputs, getCardinalityEstimator, getContainer, getEpoch, getName, getOriginal, getSimpleClassName, getTargetPlatforms, isAuxiliary, isSupportingBroadcastInputs, propagateInputCardinality, propagateOutputCardinality, setAuxiliary, setCardinalityEstimator, setContainer, setEpoch, setName, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.wayang.core.plan.wayangplan.ActualOperator

accept

Methods inherited from interface org.apache.wayang.core.plan.wayangplan.ElementaryOperator

getCardinalityEstimator, isAuxiliary, setAuxiliary, setCardinalityEstimator

Methods inherited from interface org.apache.wayang.core.plan.wayangplan.Operator

addBroadcastInput, addTargetPlatform, broadcastTo, broadcastTo, collectMappedInputSlots, collectMappedOutputSlots, connectTo, connectTo, getAllInputs, getAllOutputs, getCardinalityPusher, getContainer, getEffectiveOccupant, getEffectiveOccupant, getEpoch, getEstimationContextProperties, getForwards, getInnermostLoop, getInput, getInput, getLoopStack, getName, getNumBroadcastInputs, getNumInputs, getNumOutputs, getNumRegularInputs, getOuterInputSlot, getOutermostInputSlot, getOutermostOutputSlots, getOutput, getOutput, getParent, getTargetPlatforms, isAlternative, isConversion, isElementary, isExecutionOperator, isFeedbackInput, isFeedforwardOutput, isLoopHead, isLoopSubplan, isOwnerOf, isReading, isSink, isSource, isSubplan, isSupportingBroadcastInputs, isUnconnected, propagateInputCardinality, propagateOutputCardinality, propagateOutputCardinality, replaceWith, setContainer, setEpoch, setInput, setName, setOutput

Constructor Details

KafkaTopicSource

public KafkaTopicSource(String topicName)

KafkaTopicSource

public KafkaTopicSource(String topicName,
 String encoding)

KafkaTopicSource

public KafkaTopicSource(KafkaTopicSource that)

Copies an instance (exclusive of broadcasts).

Parameters:

that - that should be copied

Method Details

initConsumer

public void initConsumer(KafkaTopicSource kts)

getTopicName

public String getTopicName()

getEncoding

public String getEncoding()

getConsumer

public org.apache.kafka.clients.consumer.KafkaConsumer<String,String> getConsumer()

getConsumer

public org.apache.kafka.clients.consumer.KafkaConsumer<String,String> getConsumer(Properties props)

loadConfig

public static Properties loadConfig(String propertiesFilePath)

Load properties from a properties file or alternatively use the default properties with some sensitive values from environment variables.

Parameters:

propertiesFilePath - - File path or null.

Returns:

Properties object

getDefaultProperties

public static Properties getDefaultProperties()

createCardinalityEstimator

public Optional<CardinalityEstimator> createCardinalityEstimator(int outputIndex,
 Configuration configuration)

Description copied from interface: ElementaryOperator

Provide a CardinalityEstimator for the OutputSlot at outputIndex.

Parameters:

outputIndex - index of the OutputSlot for that the CardinalityEstimator is requested

configuration - if the CardinalityEstimator depends on further ones, use this to obtain the latter

Returns:

an Optional that might provide the requested instance

startConsuming

public void startConsuming()