Package org.apache.wayang.spark.util

Class DatasetConverters

java.lang.Object

org.apache.wayang.spark.util.DatasetConverters

public final class DatasetConverters
extends Object

Utility methods to convert Record-backed RDDs into Spark Datasets.

Method Summary

Modifier and Type	Method	Description
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	recordsToDataset(org.apache.spark.api.java.JavaRDD<Record> records, DataSetType<Record> dataSetType, org.apache.spark.sql.SparkSession sparkSession)	Convert an RDD of Records into a Spark Dataset.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

recordsToDataset

public static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> recordsToDataset(org.apache.spark.api.java.JavaRDD<Record> records,
 DataSetType<Record> dataSetType,
 org.apache.spark.sql.SparkSession sparkSession)

Convert an RDD of Records into a Spark Dataset.

Parameters:

records - the records to convert

dataSetType - type information about the records (field names etc.)

sparkSession - the SparkSession used to create the Dataset

Returns:

a Dataset view over records