Package org.apache.wayang.flink.compiler

Class WayangFileOutputFormat<IT>

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.flink.api.common.io.CleanupWhenUnsuccessful, org.apache.flink.api.common.io.InitializeOnMaster, org.apache.flink.api.common.io.OutputFormat<IT>
    public class WayangFileOutputFormat<IT>
extends org.apache.flink.api.common.io.FileOutputFormat<IT>
implements org.apache.flink.api.common.io.InitializeOnMaster, org.apache.flink.api.common.io.CleanupWhenUnsuccessful
    Wrapper for FileOutputFormat
    See Also:
    Serialized Form

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      protected class  WayangFileOutputFormat.BlockBasedOutput
      Writes a block info at the end of the blocks.
      Current implementation uses only int and not long.

      • Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileOutputFormat

        org.apache.flink.api.common.io.FileOutputFormat.OutputDirectoryMode

      • Nested classes/interfaces inherited from interface org.apache.flink.api.common.io.OutputFormat

        org.apache.flink.api.common.io.OutputFormat.InitializationContext

    • Field Summary

      Fields 
      Modifier and Type Field Description
      static java.lang.String BLOCK_SIZE_PARAMETER_KEY
      The config parameter which defines the fixed length of a record.
      static java.lang.String FILE_PARAMETER_KEY
      The key under which the name of the target path is stored in the configuration.
      static long NATIVE_BLOCK_SIZE  
      protected org.apache.flink.core.fs.Path outputFilePath
      The path of the file to be written.
      protected org.apache.flink.core.fs.FSDataOutputStream stream
      The stream to which the data is written;

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()  
      protected void complementBlockInfo​(org.apache.flink.api.common.io.BlockInfo blockInfo)  
      void configure​(org.apache.flink.configuration.Configuration parameters)  
      protected org.apache.flink.api.common.io.BlockInfo createBlockInfo()  
      protected java.lang.String getDirectoryFileName​(int taskNumber)  
      org.apache.flink.api.common.io.FileOutputFormat.OutputDirectoryMode getOutputDirectoryMode()  
      org.apache.flink.core.fs.Path getOutputFilePath()  
      org.apache.flink.core.fs.FileSystem.WriteMode getWriteMode()  
      static void initDefaultsFromConfiguration​(org.apache.flink.configuration.Configuration configuration)
      Initialize defaults for output format.
      void initializeGlobal​(int parallelism)
      Initialization of the distributed file system if it is used.
      void open​(int taskNumber, int numTasks)  
      void setOutputDirectoryMode​(org.apache.flink.api.common.io.FileOutputFormat.OutputDirectoryMode mode)  
      void setOutputFilePath​(org.apache.flink.core.fs.Path path)  
      void setWriteMode​(org.apache.flink.core.fs.FileSystem.WriteMode mode)  
      void tryCleanupOnError()  
      void writeRecord​(IT record)  

      • Methods inherited from class org.apache.flink.api.common.io.RichOutputFormat

        getRuntimeContext, setRuntimeContext

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface org.apache.flink.api.common.io.OutputFormat

        open

    • Field Detail

      • FILE_PARAMETER_KEY

        public static final java.lang.String FILE_PARAMETER_KEY
        The key under which the name of the target path is stored in the configuration.
        See Also:
        Constant Field Values

      • outputFilePath

        protected org.apache.flink.core.fs.Path outputFilePath
        The path of the file to be written.

      • stream

        protected transient org.apache.flink.core.fs.FSDataOutputStream stream
        The stream to which the data is written;

      • BLOCK_SIZE_PARAMETER_KEY

        public static final java.lang.String BLOCK_SIZE_PARAMETER_KEY
        The config parameter which defines the fixed length of a record.
        See Also:
        Constant Field Values

    • Constructor Detail

      • WayangFileOutputFormat

        public WayangFileOutputFormat()

      • WayangFileOutputFormat

        public WayangFileOutputFormat​(java.lang.String path)

      • WayangFileOutputFormat

        public WayangFileOutputFormat​(org.apache.flink.core.fs.Path outputPath)

    • Method Detail

      • initDefaultsFromConfiguration

        public static void initDefaultsFromConfiguration​(org.apache.flink.configuration.Configuration configuration)
        Initialize defaults for output format. Needs to be a static method because it is configured for local cluster execution, see LocalFlinkMiniCluster.
        Parameters:
        configuration - The configuration to load defaults from

      • setOutputFilePath

        public void setOutputFilePath​(org.apache.flink.core.fs.Path path)
        Overrides:
        setOutputFilePath in class org.apache.flink.api.common.io.FileOutputFormat<IT>

      • getOutputFilePath

        public org.apache.flink.core.fs.Path getOutputFilePath()
        Overrides:
        getOutputFilePath in class org.apache.flink.api.common.io.FileOutputFormat<IT>

      • setWriteMode

        public void setWriteMode​(org.apache.flink.core.fs.FileSystem.WriteMode mode)
        Overrides:
        setWriteMode in class org.apache.flink.api.common.io.FileOutputFormat<IT>

      • getWriteMode

        public org.apache.flink.core.fs.FileSystem.WriteMode getWriteMode()
        Overrides:
        getWriteMode in class org.apache.flink.api.common.io.FileOutputFormat<IT>

      • setOutputDirectoryMode

        public void setOutputDirectoryMode​(org.apache.flink.api.common.io.FileOutputFormat.OutputDirectoryMode mode)
        Overrides:
        setOutputDirectoryMode in class org.apache.flink.api.common.io.FileOutputFormat<IT>

      • getOutputDirectoryMode

        public org.apache.flink.api.common.io.FileOutputFormat.OutputDirectoryMode getOutputDirectoryMode()
        Overrides:
        getOutputDirectoryMode in class org.apache.flink.api.common.io.FileOutputFormat<IT>

      • configure

        public void configure​(org.apache.flink.configuration.Configuration parameters)
        Specified by:
        configure in interface org.apache.flink.api.common.io.OutputFormat<IT>
        Overrides:
        configure in class org.apache.flink.api.common.io.FileOutputFormat<IT>

      • open

        public void open​(int taskNumber,
                 int numTasks)
          throws java.io.IOException
        Specified by:
        open in interface org.apache.flink.api.common.io.OutputFormat<IT>
        Overrides:
        open in class org.apache.flink.api.common.io.FileOutputFormat<IT>
        Throws:
        java.io.IOException

      • writeRecord

        public void writeRecord​(IT record)
                 throws java.io.IOException
        Specified by:
        writeRecord in interface org.apache.flink.api.common.io.OutputFormat<IT>
        Throws:
        java.io.IOException

      • getDirectoryFileName

        protected java.lang.String getDirectoryFileName​(int taskNumber)
        Overrides:
        getDirectoryFileName in class org.apache.flink.api.common.io.FileOutputFormat<IT>

      • close

        public void close()
           throws java.io.IOException
        Specified by:
        close in interface org.apache.flink.api.common.io.OutputFormat<IT>
        Overrides:
        close in class org.apache.flink.api.common.io.FileOutputFormat<IT>
        Throws:
        java.io.IOException

      • initializeGlobal

        public void initializeGlobal​(int parallelism)
                      throws java.io.IOException
        Initialization of the distributed file system if it is used.
        Specified by:
        initializeGlobal in interface org.apache.flink.api.common.io.InitializeOnMaster
        Overrides:
        initializeGlobal in class org.apache.flink.api.common.io.FileOutputFormat<IT>
        Parameters:
        parallelism - The task parallelism.
        Throws:
        java.io.IOException

      • tryCleanupOnError

        public void tryCleanupOnError()
        Specified by:
        tryCleanupOnError in interface org.apache.flink.api.common.io.CleanupWhenUnsuccessful
        Overrides:
        tryCleanupOnError in class org.apache.flink.api.common.io.FileOutputFormat<IT>

      • createBlockInfo

        protected org.apache.flink.api.common.io.BlockInfo createBlockInfo()

      • complementBlockInfo

        protected void complementBlockInfo​(org.apache.flink.api.common.io.BlockInfo blockInfo)