Package org.apache.wayang.flink.compiler

Class WayangFileOutputFormat<IT>

java.lang.Object
org.apache.flink.api.common.io.RichOutputFormat<IT>
org.apache.flink.api.common.io.FileOutputFormat<IT>
org.apache.wayang.flink.compiler.WayangFileOutputFormat<IT>
All Implemented Interfaces:
Serializable, org.apache.flink.api.common.io.CleanupWhenUnsuccessful, org.apache.flink.api.common.io.InitializeOnMaster, org.apache.flink.api.common.io.OutputFormat<IT>
public class WayangFileOutputFormat<IT> extends org.apache.flink.api.common.io.FileOutputFormat<IT> implements org.apache.flink.api.common.io.InitializeOnMaster, org.apache.flink.api.common.io.CleanupWhenUnsuccessful
Wrapper for FileOutputFormat
See Also:

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    protected class 
    WayangFileOutputFormat.BlockBasedOutput
    Writes a block info at the end of the blocks.
    Current implementation uses only int and not long.

    Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileOutputFormat

    org.apache.flink.api.common.io.FileOutputFormat.OutputDirectoryMode

    Nested classes/interfaces inherited from interface org.apache.flink.api.common.io.OutputFormat

    org.apache.flink.api.common.io.OutputFormat.InitializationContext

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String
    BLOCK_SIZE_PARAMETER_KEY
    The config parameter which defines the fixed length of a record.
    static final String
    FILE_PARAMETER_KEY
    The key under which the name of the target path is stored in the configuration.
    static final long
    NATIVE_BLOCK_SIZE
     
    protected org.apache.flink.core.fs.Path
    outputFilePath
    The path of the file to be written.
    protected org.apache.flink.core.fs.FSDataOutputStream
    stream
    The stream to which the data is written;

  • Constructor Summary

    Constructors
    Constructor
    Description
    WayangFileOutputFormat()
     
    WayangFileOutputFormat(String path)
     
    WayangFileOutputFormat(org.apache.flink.core.fs.Path outputPath)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    close()
     
    protected void
    complementBlockInfo(org.apache.flink.api.common.io.BlockInfo blockInfo)
     
    void
    configure(org.apache.flink.configuration.Configuration parameters)
     
    protected org.apache.flink.api.common.io.BlockInfo
    createBlockInfo()
     
    protected String
    getDirectoryFileName(int taskNumber)
     
    org.apache.flink.api.common.io.FileOutputFormat.OutputDirectoryMode
    getOutputDirectoryMode()
     
    org.apache.flink.core.fs.Path
    getOutputFilePath()
     
    org.apache.flink.core.fs.FileSystem.WriteMode
    getWriteMode()
     
    static void
    initDefaultsFromConfiguration(org.apache.flink.configuration.Configuration configuration)
    Initialize defaults for output format.
    void
    initializeGlobal(int parallelism)
    Initialization of the distributed file system if it is used.
    void
    open(int taskNumber, int numTasks)
     
    void
    setOutputDirectoryMode(org.apache.flink.api.common.io.FileOutputFormat.OutputDirectoryMode mode)
     
    void
    setOutputFilePath(org.apache.flink.core.fs.Path path)
     
    void
    setWriteMode(org.apache.flink.core.fs.FileSystem.WriteMode mode)
     
    void
    tryCleanupOnError()
     
    void
    writeRecord(IT record)
     

    Methods inherited from class org.apache.flink.api.common.io.RichOutputFormat

    getRuntimeContext, setRuntimeContext

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.flink.api.common.io.OutputFormat

    open

  • Field Details

    • FILE_PARAMETER_KEY

      public static final String FILE_PARAMETER_KEY
      The key under which the name of the target path is stored in the configuration.
      See Also:

    • outputFilePath

      protected org.apache.flink.core.fs.Path outputFilePath
      The path of the file to be written.

    • stream

      protected transient org.apache.flink.core.fs.FSDataOutputStream stream
      The stream to which the data is written;

    • BLOCK_SIZE_PARAMETER_KEY

      public static final String BLOCK_SIZE_PARAMETER_KEY
      The config parameter which defines the fixed length of a record.
      See Also:

    • NATIVE_BLOCK_SIZE

      public static final long NATIVE_BLOCK_SIZE
      See Also:

  • Constructor Details

    • WayangFileOutputFormat

      public WayangFileOutputFormat()

    • WayangFileOutputFormat

      public WayangFileOutputFormat(String path)

    • WayangFileOutputFormat

      public WayangFileOutputFormat(org.apache.flink.core.fs.Path outputPath)

  • Method Details

    • initDefaultsFromConfiguration

      public static void initDefaultsFromConfiguration(org.apache.flink.configuration.Configuration configuration)
      Initialize defaults for output format. Needs to be a static method because it is configured for local cluster execution, see LocalFlinkMiniCluster.
      Parameters:
      configuration - The configuration to load defaults from

    • setOutputFilePath

      public void setOutputFilePath(org.apache.flink.core.fs.Path path)
      Overrides:
      setOutputFilePath in class org.apache.flink.api.common.io.FileOutputFormat<IT>

    • getOutputFilePath

      public org.apache.flink.core.fs.Path getOutputFilePath()
      Overrides:
      getOutputFilePath in class org.apache.flink.api.common.io.FileOutputFormat<IT>

    • setWriteMode

      public void setWriteMode(org.apache.flink.core.fs.FileSystem.WriteMode mode)
      Overrides:
      setWriteMode in class org.apache.flink.api.common.io.FileOutputFormat<IT>

    • getWriteMode

      public org.apache.flink.core.fs.FileSystem.WriteMode getWriteMode()
      Overrides:
      getWriteMode in class org.apache.flink.api.common.io.FileOutputFormat<IT>

    • setOutputDirectoryMode

      public void setOutputDirectoryMode(org.apache.flink.api.common.io.FileOutputFormat.OutputDirectoryMode mode)
      Overrides:
      setOutputDirectoryMode in class org.apache.flink.api.common.io.FileOutputFormat<IT>

    • getOutputDirectoryMode

      public org.apache.flink.api.common.io.FileOutputFormat.OutputDirectoryMode getOutputDirectoryMode()
      Overrides:
      getOutputDirectoryMode in class org.apache.flink.api.common.io.FileOutputFormat<IT>

    • configure

      public void configure(org.apache.flink.configuration.Configuration parameters)
      Specified by:
      configure in interface org.apache.flink.api.common.io.OutputFormat<IT>
      Overrides:
      configure in class org.apache.flink.api.common.io.FileOutputFormat<IT>

    • open

      public void open(int taskNumber, int numTasks) throws IOException
      Specified by:
      open in interface org.apache.flink.api.common.io.OutputFormat<IT>
      Overrides:
      open in class org.apache.flink.api.common.io.FileOutputFormat<IT>
      Throws:
      IOException

    • writeRecord

      public void writeRecord(IT record) throws IOException
      Specified by:
      writeRecord in interface org.apache.flink.api.common.io.OutputFormat<IT>
      Throws:
      IOException

    • getDirectoryFileName

      protected String getDirectoryFileName(int taskNumber)
      Overrides:
      getDirectoryFileName in class org.apache.flink.api.common.io.FileOutputFormat<IT>

    • close

      public void close() throws IOException
      Specified by:
      close in interface org.apache.flink.api.common.io.OutputFormat<IT>
      Overrides:
      close in class org.apache.flink.api.common.io.FileOutputFormat<IT>
      Throws:
      IOException

    • initializeGlobal

      public void initializeGlobal(int parallelism) throws IOException
      Initialization of the distributed file system if it is used.
      Specified by:
      initializeGlobal in interface org.apache.flink.api.common.io.InitializeOnMaster
      Overrides:
      initializeGlobal in class org.apache.flink.api.common.io.FileOutputFormat<IT>
      Parameters:
      parallelism - The task parallelism.
      Throws:
      IOException

    • tryCleanupOnError

      public void tryCleanupOnError()
      Specified by:
      tryCleanupOnError in interface org.apache.flink.api.common.io.CleanupWhenUnsuccessful
      Overrides:
      tryCleanupOnError in class org.apache.flink.api.common.io.FileOutputFormat<IT>

    • createBlockInfo

      protected org.apache.flink.api.common.io.BlockInfo createBlockInfo()

    • complementBlockInfo

      protected void complementBlockInfo(org.apache.flink.api.common.io.BlockInfo blockInfo)