org.apache.flink.runtime.state.filesystem

Class FsStateBackend

java.lang.Object

org.apache.flink.runtime.state.AbstractStateBackend

org.apache.flink.runtime.state.filesystem.FsStateBackend

All Implemented Interfaces:

Serializable

public class FsStateBackend
extends AbstractStateBackend

The file state backend is a state backend that stores the state of streaming jobs in a file system.

The state backend has one core directory into which it puts all checkpoint data. Inside that directory, it creates a directory per job, inside which each checkpoint gets a directory, with files for each state, for example: hdfs://namenode:port/flink-checkpoints/<job-id>/chk-17/6ba7b810-9dad-11d1-80b4-00c04fd430c8

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FsStateBackend.FsCheckpointStateOutputStream A CheckpointStateOutputStream that writes into a file and returns the path to that file upon closing.

Nested classes/interfaces inherited from class org.apache.flink.runtime.state.AbstractStateBackend

AbstractStateBackend.CheckpointStateOutputStream, AbstractStateBackend.CheckpointStateOutputView

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_FILE_STATE_THRESHOLD By default, state smaller than 1024 bytes will not be written to files, but will be stored directly with the metadata
static int	MAX_FILE_STATE_THRESHOLD Maximum size of state that is stored with the metadata, rather than in files

Fields inherited from class org.apache.flink.runtime.state.AbstractStateBackend

currentKey, keySerializer, keyValueStatesByName, userCodeClassLoader

Constructor Summary

Constructors

Constructor and Description
FsStateBackend(org.apache.flink.core.fs.Path checkpointDataUri) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.
FsStateBackend(String checkpointDataUri) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.
FsStateBackend(URI checkpointDataUri) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.
FsStateBackend(URI checkpointDataUri, int fileStateSizeThreshold) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

Method Summary

Modifier and Type	Method and Description
<S extends Serializable> StateHandle<S>	checkpointStateSerializable(S state, long checkpointID, long timestamp) Writes the given state into the checkpoint, and returns a handle that can retrieve the state back.
void	close() Closes the state backend, releasing all internal resources, but does not delete any persistent checkpoint data.
FsStateBackend.FsCheckpointStateOutputStream	createCheckpointStateOutputStream(long checkpointID, long timestamp) Creates an output stream that writes into the state of the given checkpoint.
protected <N,T,ACC> org.apache.flink.api.common.state.FoldingState<T,ACC>	createFoldingState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.FoldingStateDescriptor<T,ACC> stateDesc) Creates and returns a new FoldingState.
<N,T> org.apache.flink.api.common.state.ListState<T>	createListState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.ListStateDescriptor<T> stateDesc) Creates and returns a new ListState.
<N,T> org.apache.flink.api.common.state.ReducingState<T>	createReducingState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.ReducingStateDescriptor<T> stateDesc) Creates and returns a new ReducingState.
<N,V> org.apache.flink.api.common.state.ValueState<V>	createValueState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.ValueStateDescriptor<V> stateDesc) Creates and returns a new ValueState.
void	disposeAllStateForCurrentJob() Disposes all state associated with the current job.
org.apache.flink.core.fs.Path	getBasePath() Gets the base directory where all state-containing files are stored.
org.apache.flink.core.fs.Path	getCheckpointDirectory() Gets the directory where this state backend stores its checkpoint data.
int	getFileStateSizeThreshold() Gets the size (in bytes) above which the state will written to files.
org.apache.flink.core.fs.FileSystem	getFileSystem() Gets the file system handle for the file system that stores the state for this backend.
void	initializeForJob(Environment env, String operatorIdentifier, org.apache.flink.api.common.typeutils.TypeSerializer<?> keySerializer) This method is called by the task upon deployment to initialize the state backend for data for a specific job.
boolean	isInitialized() Checks whether this state backend is initialized.
String	toString()
static org.apache.flink.core.fs.Path	validateAndNormalizeUri(URI checkpointDataUri) Checks and normalizes the checkpoint data URI.

Methods inherited from class org.apache.flink.runtime.state.AbstractStateBackend

createCheckpointStateOutputView, dispose, getCurrentKey, getPartitionedState, injectKeyValueStateSnapshots, mergePartitionedStates, notifyOfCompletedCheckpoint, setCurrentKey, snapshotPartitionedState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_FILE_STATE_THRESHOLD

public static final int DEFAULT_FILE_STATE_THRESHOLD

By default, state smaller than 1024 bytes will not be written to files, but will be stored directly with the metadata

See Also:

Constant Field Values

MAX_FILE_STATE_THRESHOLD

public static final int MAX_FILE_STATE_THRESHOLD

Maximum size of state that is stored with the metadata, rather than in files

See Also:

Constant Field Values

Constructor Detail

FsStateBackend

public FsStateBackend(String checkpointDataUri)
               throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to the checkpoint data directory.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

FsStateBackend

public FsStateBackend(org.apache.flink.core.fs.Path checkpointDataUri)
               throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to the checkpoint data directory.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

FsStateBackend

public FsStateBackend(URI checkpointDataUri)
               throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to the checkpoint data directory.

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

FsStateBackend

public FsStateBackend(URI checkpointDataUri,
                      int fileStateSizeThreshold)
               throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to the checkpoint data directory.

fileStateSizeThreshold - State up to this size will be stored as part of the metadata, rather than in files

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

Method Detail

getBasePath

public org.apache.flink.core.fs.Path getBasePath()

Gets the base directory where all state-containing files are stored. The job specific directory is created inside this directory.

Returns:

The base directory.

getCheckpointDirectory

public org.apache.flink.core.fs.Path getCheckpointDirectory()

Gets the directory where this state backend stores its checkpoint data. Will be null if the state backend has not been initialized.

Returns:

The directory where this state backend stores its checkpoint data.

getFileStateSizeThreshold

public int getFileStateSizeThreshold()

Gets the size (in bytes) above which the state will written to files. State whose size is below this threshold will be directly stored with the metadata (the state handles), rather than in files. This threshold helps to prevent an accumulation of small files for small states.

Returns:

The threshold (in bytes) above which state is written to files.

isInitialized

public boolean isInitialized()

Checks whether this state backend is initialized. Note that initialization does not carry across serialization. After each serialization, the state backend needs to be initialized.

Returns:

True, if the file state backend has been initialized, false otherwise.

getFileSystem

public org.apache.flink.core.fs.FileSystem getFileSystem()

Gets the file system handle for the file system that stores the state for this backend.

Returns:

This backend's file system handle.

initializeForJob

public void initializeForJob(Environment env,
                             String operatorIdentifier,
                             org.apache.flink.api.common.typeutils.TypeSerializer<?> keySerializer)
                      throws Exception

Description copied from class: AbstractStateBackend

This method is called by the task upon deployment to initialize the state backend for data for a specific job.

Overrides:

initializeForJob in class AbstractStateBackend

Parameters:

env - The Environment of the task that instantiated the state backend

operatorIdentifier - Unique identifier for naming states created by this backend

Throws:

Exception - Overwritten versions of this method may throw exceptions, in which case the job that uses the state backend is considered failed during deployment.

disposeAllStateForCurrentJob

public void disposeAllStateForCurrentJob()
                                  throws Exception

Description copied from class: AbstractStateBackend

Disposes all state associated with the current job.

Specified by:

disposeAllStateForCurrentJob in class AbstractStateBackend

Throws:

Exception - Exceptions may occur during disposal of the state and should be forwarded.

close

public void close()
           throws Exception

Description copied from class: AbstractStateBackend

Closes the state backend, releasing all internal resources, but does not delete any persistent checkpoint data.

Specified by:

close in class AbstractStateBackend

Throws:

Exception - Exceptions can be forwarded and will be logged by the system

createValueState

public <N,V> org.apache.flink.api.common.state.ValueState<V> createValueState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                              org.apache.flink.api.common.state.ValueStateDescriptor<V> stateDesc)
                                                                       throws Exception

Description copied from class: AbstractStateBackend

Creates and returns a new ValueState.

Specified by:

createValueState in class AbstractStateBackend

Type Parameters:

N - The type of the namespace.

V - The type of the value that the ValueState can store.

Parameters:

namespaceSerializer - TypeSerializer for the state namespace.

stateDesc - The StateDescriptor that contains the name of the state.

Throws:

Exception

createListState

public <N,T> org.apache.flink.api.common.state.ListState<T> createListState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                            org.apache.flink.api.common.state.ListStateDescriptor<T> stateDesc)
                                                                     throws Exception

Description copied from class: AbstractStateBackend

Creates and returns a new ListState.

Specified by:

createListState in class AbstractStateBackend

Type Parameters:

N - The type of the namespace.

T - The type of the values that the ListState can store.

Parameters:

namespaceSerializer - TypeSerializer for the state namespace.

stateDesc - The StateDescriptor that contains the name of the state.

Throws:

Exception

createReducingState

public <N,T> org.apache.flink.api.common.state.ReducingState<T> createReducingState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                                    org.apache.flink.api.common.state.ReducingStateDescriptor<T> stateDesc)
                                                                             throws Exception

Description copied from class: AbstractStateBackend

Creates and returns a new ReducingState.

Specified by:

createReducingState in class AbstractStateBackend

Type Parameters:

N - The type of the namespace.

T - The type of the values that the ListState can store.

Parameters:

namespaceSerializer - TypeSerializer for the state namespace.

stateDesc - The StateDescriptor that contains the name of the state.

Throws:

Exception

createFoldingState

protected <N,T,ACC> org.apache.flink.api.common.state.FoldingState<T,ACC> createFoldingState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                                             org.apache.flink.api.common.state.FoldingStateDescriptor<T,ACC> stateDesc)
                                                                                      throws Exception

Description copied from class: AbstractStateBackend

Creates and returns a new FoldingState.

Specified by:

createFoldingState in class AbstractStateBackend

Type Parameters:

N - The type of the namespace.

T - Type of the values folded into the state

ACC - Type of the value in the state *

Parameters:

namespaceSerializer - TypeSerializer for the state namespace.

stateDesc - The StateDescriptor that contains the name of the state.

Throws:

Exception

checkpointStateSerializable

public <S extends Serializable> StateHandle<S> checkpointStateSerializable(S state,
                                                                           long checkpointID,
                                                                           long timestamp)
                                                                    throws Exception

Description copied from class: AbstractStateBackend

Writes the given state into the checkpoint, and returns a handle that can retrieve the state back.

Specified by:

checkpointStateSerializable in class AbstractStateBackend

Type Parameters:

S - The type of the state.

Parameters:

state - The state to be checkpointed.

checkpointID - The ID of the checkpoint.

timestamp - The timestamp of the checkpoint.

Returns:

A state handle that can retrieve the checkpoined state.

Throws:

Exception - Exceptions may occur during serialization / storing the state and should be forwarded.

createCheckpointStateOutputStream

public FsStateBackend.FsCheckpointStateOutputStream createCheckpointStateOutputStream(long checkpointID,
                                                                                      long timestamp)
                                                                               throws Exception

Description copied from class: AbstractStateBackend

Creates an output stream that writes into the state of the given checkpoint. When the stream is closes, it returns a state handle that can retrieve the state back.

Specified by:

createCheckpointStateOutputStream in class AbstractStateBackend

Parameters:

checkpointID - The ID of the checkpoint.

timestamp - The timestamp of the checkpoint.

Returns:

An output stream that writes state for the given checkpoint.

Throws:

Exception - Exceptions may occur while creating the stream and should be forwarded.

toString

public String toString()

Overrides:

toString in class Object

validateAndNormalizeUri

public static org.apache.flink.core.fs.Path validateAndNormalizeUri(URI checkpointDataUri)
                                                             throws IOException

Checks and normalizes the checkpoint data URI. This method first checks the validity of the URI (scheme, path, availability of a matching file system) and then normalizes the URI to a path.

If the URI does not include an authority, but the file system configured for the URI has an authority, then the normalized path will include this authority.

Parameters:

checkpointDataUri - The URI to check and normalize.

Returns:

A normalized URI as a Path.

Throws:

IllegalArgumentException - Thrown, if the URI misses scheme or path.

IOException - Thrown, if no file system can be found for the URI's scheme.