org.apache.flink.runtime.fs.hdfs

Class HadoopFileSystem

java.lang.Object

org.apache.flink.core.fs.FileSystem

org.apache.flink.runtime.fs.hdfs.HadoopFileSystem

All Implemented Interfaces:

HadoopFileSystemWrapper

public final class HadoopFileSystem
extends FileSystem
implements HadoopFileSystemWrapper

Concrete implementation of the FileSystem base class for the Hadoop File System. The class is a wrapper class which encapsulated the original Hadoop HDFS API. If no file system class is specified, the wrapper will automatically load the Hadoop distributed file system (HDFS).

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.core.fs.FileSystem

FileSystem.WriteMode

Constructor Summary

Constructors

Constructor and Description
HadoopFileSystem(Class<? extends org.apache.hadoop.fs.FileSystem> fsClass) Creates a new DistributedFileSystem object to access HDFS, based on a class name and picking up the configuration from the class path or the Flink configuration.
HadoopFileSystem(org.apache.hadoop.conf.Configuration hadoopConfig, org.apache.hadoop.fs.FileSystem hadoopFileSystem) Creates a new DistributedFileSystem that uses the given Hadoop FileSystem under the hood.

Method Summary

Methods

Modifier and Type	Method and Description
HadoopDataOutputStream	create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize)
HadoopDataOutputStream	create(Path f, FileSystem.WriteMode overwrite)
boolean	delete(Path f, boolean recursive)
long	getDefaultBlockSize()
BlockLocation[]	getFileBlockLocations(FileStatus file, long start, long len)
FileStatus	getFileStatus(Path f)
static org.apache.hadoop.conf.Configuration	getHadoopConfiguration() Returns a new Hadoop Configuration object using the path to the hadoop conf configured in the main configuration (flink-conf.yaml).
org.apache.hadoop.fs.FileSystem	getHadoopFileSystem() Gets the underlying Hadoop FileSystem.
Class<?>	getHadoopWrapperClassNameForFileSystem(String scheme)
Path	getHomeDirectory()
URI	getUri()
Path	getWorkingDirectory()
void	initialize(URI path)
boolean	isDistributedFS()
FileStatus[]	listStatus(Path f)
boolean	mkdirs(Path f)
HadoopDataInputStream	open(Path f)
HadoopDataInputStream	open(Path f, int bufferSize)
boolean	rename(Path src, Path dst)

Methods inherited from class org.apache.flink.core.fs.FileSystem

create, exists, get, getLocalFileSystem, getUnguardedFileSystem, initOutPathDistFS, initOutPathLocalFS, isFlinkSupportedScheme, setDefaultScheme

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopFileSystem

public HadoopFileSystem(Class<? extends org.apache.hadoop.fs.FileSystem> fsClass)
                 throws IOException

Creates a new DistributedFileSystem object to access HDFS, based on a class name and picking up the configuration from the class path or the Flink configuration.

Throws:

IOException - throw if the required HDFS classes cannot be instantiated

HadoopFileSystem

public HadoopFileSystem(org.apache.hadoop.conf.Configuration hadoopConfig,
                org.apache.hadoop.fs.FileSystem hadoopFileSystem)

Creates a new DistributedFileSystem that uses the given Hadoop FileSystem under the hood.

Parameters:

hadoopConfig - The Hadoop configuration that the FileSystem is based on.

hadoopFileSystem - The Hadoop FileSystem that will be used under the hood.

Method Detail

getHadoopConfiguration

public static org.apache.hadoop.conf.Configuration getHadoopConfiguration()

Returns a new Hadoop Configuration object using the path to the hadoop conf configured in the main configuration (flink-conf.yaml). This method is public because its being used in the HadoopDataSource.

getWorkingDirectory

public Path getWorkingDirectory()

Specified by:

getWorkingDirectory in class FileSystem

getHomeDirectory

public Path getHomeDirectory()

Specified by:

getHomeDirectory in class FileSystem

getUri

public URI getUri()

Specified by:

getUri in class FileSystem

getHadoopFileSystem

public org.apache.hadoop.fs.FileSystem getHadoopFileSystem()

Gets the underlying Hadoop FileSystem.

Returns:

The underlying Hadoop FileSystem.

initialize

public void initialize(URI path)
                throws IOException

Specified by:

initialize in class FileSystem

Throws:

IOException

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws IOException

Specified by:

getFileStatus in class FileSystem

Throws:

IOException

getFileBlockLocations

public BlockLocation[] getFileBlockLocations(FileStatus file,
                                    long start,
                                    long len)
                                      throws IOException

Specified by:

getFileBlockLocations in class FileSystem

Throws:

IOException

open

public HadoopDataInputStream open(Path f,
                         int bufferSize)
                           throws IOException

Specified by:

open in class FileSystem

Throws:

IOException

open

public HadoopDataInputStream open(Path f)
                           throws IOException

Specified by:

open in class FileSystem

Throws:

IOException

create

public HadoopDataOutputStream create(Path f,
                            boolean overwrite,
                            int bufferSize,
                            short replication,
                            long blockSize)
                              throws IOException

Specified by:

create in class FileSystem

Throws:

IOException

create

public HadoopDataOutputStream create(Path f,
                            FileSystem.WriteMode overwrite)
                              throws IOException

Specified by:

create in class FileSystem

Throws:

IOException

delete

public boolean delete(Path f,
             boolean recursive)
               throws IOException

Specified by:

delete in class FileSystem

Throws:

IOException

listStatus

public FileStatus[] listStatus(Path f)
                        throws IOException

Specified by:

listStatus in class FileSystem

Throws:

IOException

mkdirs

public boolean mkdirs(Path f)
               throws IOException

Specified by:

mkdirs in class FileSystem

Throws:

IOException

rename

public boolean rename(Path src,
             Path dst)
               throws IOException

Specified by:

rename in class FileSystem

Throws:

IOException

getDefaultBlockSize

public long getDefaultBlockSize()

Overrides:

getDefaultBlockSize in class FileSystem

isDistributedFS

public boolean isDistributedFS()

Specified by:

isDistributedFS in class FileSystem

getHadoopWrapperClassNameForFileSystem

public Class<?> getHadoopWrapperClassNameForFileSystem(String scheme)

Specified by:

getHadoopWrapperClassNameForFileSystem in interface HadoopFileSystemWrapper