org.apache.flink.runtime.state.filesystem

Class FsCheckpointStreamFactory

java.lang.Object

org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory

All Implemented Interfaces:

CheckpointStreamFactory

Direct Known Subclasses:

FsSavepointStreamFactory

public class FsCheckpointStreamFactory
extends Object
implements CheckpointStreamFactory

CheckpointStreamFactory that produces streams that write to a FileSystem.

The factory has one core directory into which it puts all checkpoint data. Inside that directory, it creates a directory per job, inside which each checkpoint gets a directory, with files for each state, for example: hdfs://namenode:port/flink-checkpoints/<job-id>/chk-17/6ba7b810-9dad-11d1-80b4-00c04fd430c8

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FsCheckpointStreamFactory.FsCheckpointStateOutputStream A CheckpointStreamFactory.CheckpointStateOutputStream that writes into a file and returns a StreamStateHandle upon closing.

Nested classes/interfaces inherited from interface org.apache.flink.runtime.state.CheckpointStreamFactory

CheckpointStreamFactory.CheckpointStateOutputStream

Constructor Summary

Constructors

Constructor and Description
FsCheckpointStreamFactory(Path checkpointDataUri, JobID jobId, int fileStateSizeThreshold) Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Closes the stream factory, releasing all internal resources, but does not delete any persistent checkpoint data.
protected Path	createBasePath(FileSystem fs, Path checkpointDirectory, JobID jobID)
protected Path	createCheckpointDirPath(Path checkpointDirectory, long checkpointID)
FsCheckpointStreamFactory.FsCheckpointStateOutputStream	createCheckpointStateOutputStream(long checkpointID, long timestamp) Creates an new CheckpointStreamFactory.CheckpointStateOutputStream.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

FsCheckpointStreamFactory

public FsCheckpointStreamFactory(Path checkpointDataUri,
                         JobID jobId,
                         int fileStateSizeThreshold)
                          throws IOException

Creates a new state backend that stores its checkpoint data in the file system and location defined by the given URI.

A file system for the file system scheme in the URI (e.g., 'file://', 'hdfs://', or 'S3://') must be accessible via FileSystem.get(URI).

For a state backend targeting HDFS, this means that the URI must either specify the authority (host and port), or that the Hadoop configuration that describes that information must be in the classpath.

Parameters:

checkpointDataUri - The URI describing the filesystem (scheme and optionally authority), and the path to the checkpoint data directory.

fileStateSizeThreshold - State up to this size will be stored as part of the metadata, rather than in files

Throws:

IOException - Thrown, if no file system can be found for the scheme in the URI.

Method Detail

close

public void close()
           throws Exception

Description copied from interface: CheckpointStreamFactory

Closes the stream factory, releasing all internal resources, but does not delete any persistent checkpoint data.

Specified by:

close in interface CheckpointStreamFactory

Throws:

Exception - Exceptions can be forwarded and will be logged by the system

createCheckpointStateOutputStream

public FsCheckpointStreamFactory.FsCheckpointStateOutputStream createCheckpointStateOutputStream(long checkpointID,
                                                                                        long timestamp)
                                                                                          throws Exception

Description copied from interface: CheckpointStreamFactory

Creates an new CheckpointStreamFactory.CheckpointStateOutputStream. When the stream is closed, it returns a state handle that can retrieve the state back.

Specified by:

createCheckpointStateOutputStream in interface CheckpointStreamFactory

Parameters:

checkpointID - The ID of the checkpoint.

timestamp - The timestamp of the checkpoint.

Returns:

An output stream that writes state for the given checkpoint.

Throws:

Exception - Exceptions may occur while creating the stream and should be forwarded.

createBasePath

protected Path createBasePath(FileSystem fs,
                  Path checkpointDirectory,
                  JobID jobID)
                       throws IOException

Throws:

IOException

createCheckpointDirPath

protected Path createCheckpointDirPath(Path checkpointDirectory,
                           long checkpointID)

toString

public String toString()

Overrides:

toString in class Object