org.apache.flink.runtime.state.filesystem

Class FsStateBackend

java.lang.Object

org.apache.flink.runtime.state.AbstractStateBackend

org.apache.flink.runtime.state.filesystem.FsStateBackend

All Implemented Interfaces:

Serializable, StateBackend

public class FsStateBackend
extends AbstractStateBackend

The file state backend is a state backend that stores the state of streaming jobs in a file system.

The state backend has one core directory into which it puts all checkpoint data. Inside that directory, it creates a directory per job, inside which each checkpoint gets a directory, with files for each state, for example: hdfs://namenode:port/flink-checkpoints/<job-id>/chk-17/6ba7b810-9dad-11d1-80b4-00c04fd430c8

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_FILE_STATE_THRESHOLD By default, state smaller than 1024 bytes will not be written to files, but will be stored directly with the metadata

Fields inherited from class org.apache.flink.runtime.state.AbstractStateBackend

FS_STATE_BACKEND_NAME, MEMORY_STATE_BACKEND_NAME, ROCKSDB_STATE_BACKEND_NAME

Constructor Summary

Constructors

Constructor and Description
FsStateBackend(Path checkpointDataUri) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.
FsStateBackend(Path checkpointDataUri, boolean asynchronousSnapshots) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.
FsStateBackend(String checkpointDataUri) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.
FsStateBackend(String checkpointDataUri, boolean asynchronousSnapshots) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.
FsStateBackend(URI checkpointDataUri) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.
FsStateBackend(URI checkpointDataUri, boolean asynchronousSnapshots) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.
FsStateBackend(URI checkpointDataUri, int fileStateSizeThreshold) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.
FsStateBackend(URI checkpointDataUri, int fileStateSizeThreshold, boolean asynchronousSnapshots) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

Method Summary

Methods

Modifier and Type	Method and Description
<K> AbstractKeyedStateBackend<K>	createKeyedStateBackend(Environment env, JobID jobID, String operatorIdentifier, TypeSerializer<K> keySerializer, int numberOfKeyGroups, KeyGroupRange keyGroupRange, TaskKvStateRegistry kvStateRegistry) Creates a new AbstractKeyedStateBackend that is responsible for holding keyed state and checkpointing it.
OperatorStateBackend	createOperatorStateBackend(Environment env, String operatorIdentifier) Creates a new OperatorStateBackend that can be used for storing operator state.
CheckpointStreamFactory	createSavepointStreamFactory(JobID jobId, String operatorIdentifier, String targetLocation) Creates a CheckpointStreamFactory that can be used to create streams that should end up in a savepoint.
CheckpointStreamFactory	createStreamFactory(JobID jobId, String operatorIdentifier) Creates a CheckpointStreamFactory that can be used to create streams that should end up in a checkpoint.
Path	getBasePath() Gets the base directory where all state-containing files are stored.
int	getMinFileSizeThreshold() Gets the threshold below which state is stored as part of the metadata, rather than in files.
String	toString()
static Path	validateAndNormalizeUri(URI checkpointDataUri) Checks and normalizes the checkpoint data URI.

Methods inherited from class org.apache.flink.runtime.state.AbstractStateBackend

loadStateBackendFromConfig, loadStateBackendFromConfigOrCreateDefault

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_FILE_STATE_THRESHOLD

public static final int DEFAULT_FILE_STATE_THRESHOLD

By default, state smaller than 1024 bytes will not be written to files, but will be stored directly with the metadata

See Also:

Constant Field Values

Constructor Detail

FsStateBackend

public FsStateBackend(String checkpointDataUri)
               throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to the checkpoint data directory.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

FsStateBackend

public FsStateBackend(String checkpointDataUri,
              boolean asynchronousSnapshots)
               throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to the checkpoint data directory.

asynchronousSnapshots - Switch to enable asynchronous snapshots.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

FsStateBackend

public FsStateBackend(Path checkpointDataUri)
               throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to the checkpoint data directory.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

FsStateBackend

public FsStateBackend(Path checkpointDataUri,
              boolean asynchronousSnapshots)
               throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to the checkpoint data directory.

asynchronousSnapshots - Switch to enable asynchronous snapshots.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

FsStateBackend

public FsStateBackend(URI checkpointDataUri)
               throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to the checkpoint data directory.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

FsStateBackend

public FsStateBackend(URI checkpointDataUri,
              boolean asynchronousSnapshots)
               throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to the checkpoint data directory.

asynchronousSnapshots - Switch to enable asynchronous snapshots.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

FsStateBackend

public FsStateBackend(URI checkpointDataUri,
              int fileStateSizeThreshold)
               throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to the checkpoint data directory.

fileStateSizeThreshold - State up to this size will be stored as part of the metadata, rather than in files

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

IllegalArgumentException - Thrown, if the fileStateSizeThreshold is out of bounds.

FsStateBackend

public FsStateBackend(URI checkpointDataUri,
              int fileStateSizeThreshold,
              boolean asynchronousSnapshots)
               throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to the checkpoint data directory.

fileStateSizeThreshold - State up to this size will be stored as part of the metadata, rather than in files

asynchronousSnapshots - Switch to enable asynchronous snapshots.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

Method Detail

getBasePath

public Path getBasePath()

Gets the base directory where all state-containing files are stored. The job specific directory is created inside this directory.

Returns:

The base directory.

getMinFileSizeThreshold

public int getMinFileSizeThreshold()

Gets the threshold below which state is stored as part of the metadata, rather than in files. This threshold ensures that the backend does not create a large amount of very small files, where potentially the file pointers are larger than the state itself.

By default, this threshold is 1024.

Returns:

The file size threshold, in bytes.

createStreamFactory

public CheckpointStreamFactory createStreamFactory(JobID jobId,
                                          String operatorIdentifier)
                                            throws IOException

Description copied from interface: StateBackend

Creates a CheckpointStreamFactory that can be used to create streams that should end up in a checkpoint.

Specified by:

createStreamFactory in interface StateBackend

Specified by:

createStreamFactory in class AbstractStateBackend

Parameters:

jobId - The JobID of the job for which we are creating checkpoint streams.

operatorIdentifier - An identifier of the operator for which we create streams.

Throws:

IOException

createSavepointStreamFactory

public CheckpointStreamFactory createSavepointStreamFactory(JobID jobId,
                                                   String operatorIdentifier,
                                                   String targetLocation)
                                                     throws IOException

Description copied from interface: StateBackend

Creates a CheckpointStreamFactory that can be used to create streams that should end up in a savepoint.

This is only called if the triggered checkpoint is a savepoint. Commonly this will return the same factory as for regular checkpoints, but maybe slightly adjusted.

Specified by:

createSavepointStreamFactory in interface StateBackend

Specified by:

createSavepointStreamFactory in class AbstractStateBackend

Parameters:

jobId - The JobID of the job for which we are creating checkpoint streams.

operatorIdentifier - An identifier of the operator for which we create streams.

targetLocation - An optional custom location for the savepoint stream.

Returns:

The stream factory for savepoints.

Throws:

IOException - Failures during stream creation are forwarded.

createKeyedStateBackend

public <K> AbstractKeyedStateBackend<K> createKeyedStateBackend(Environment env,
                                                       JobID jobID,
                                                       String operatorIdentifier,
                                                       TypeSerializer<K> keySerializer,
                                                       int numberOfKeyGroups,
                                                       KeyGroupRange keyGroupRange,
                                                       TaskKvStateRegistry kvStateRegistry)
                                                     throws IOException

Description copied from interface: StateBackend

Creates a new AbstractKeyedStateBackend that is responsible for holding keyed state and checkpointing it.

Keyed State is state where each value is bound to a key.

Specified by:

createKeyedStateBackend in interface StateBackend

Specified by:

createKeyedStateBackend in class AbstractStateBackend

Type Parameters:

K - The type of the keys by which the state is organized.

Returns:

The Keyed State Backend for the given job, operator, and key group range.

Throws:

IOException

createOperatorStateBackend

public OperatorStateBackend createOperatorStateBackend(Environment env,
                                              String operatorIdentifier)
                                                throws Exception

Description copied from interface: StateBackend

Creates a new OperatorStateBackend that can be used for storing operator state.

Operator state is state that is associated with parallel operator (or function) instances, rather than with keys.

Specified by:

createOperatorStateBackend in interface StateBackend

Specified by:

createOperatorStateBackend in class AbstractStateBackend

Parameters:

env - The runtime environment of the executing task.

operatorIdentifier - The identifier of the operator whose state should be stored.

Returns:

The OperatorStateBackend for operator identified by the job and operator identifier.

Throws:

Exception - This method may forward all exceptions that occur while instantiating the backend.

toString

public String toString()

Overrides:

toString in class Object

validateAndNormalizeUri

public static Path validateAndNormalizeUri(URI checkpointDataUri)
                                    throws IOException

Checks and normalizes the checkpoint data URI. This method first checks the validity of the URI (scheme, path, availability of a matching file system) and then normalizes the URI to a path.

If the URI does not include an authority, but the file system configured for the URI has an authority, then the normalized path will include this authority.

Parameters:

checkpointDataUri - The URI to check and normalize.

Returns:

A normalized URI as a Path.

Throws:

IllegalArgumentException - Thrown, if the URI misses scheme or path.

IOException - Thrown, if no file system can be found for the URI's scheme.