org.apache.beam.runners.spark.io

Class MicrobatchSource<T,CheckpointMarkT extends UnboundedSource.CheckpointMark>

java.lang.Object

org.apache.beam.sdk.io.Source<T>

org.apache.beam.runners.spark.io.MicrobatchSource<T,CheckpointMarkT>

All Implemented Interfaces:

Serializable, HasDisplayData

public class MicrobatchSource<T,CheckpointMarkT extends UnboundedSource.CheckpointMark>
extends Source<T>

A Source that accommodates Spark's micro-batch oriented nature and wraps an UnboundedSource.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	MicrobatchSource.Reader Mostly based on BoundedReadFromUnboundedSource's UnboundedToBoundedSourceAdapter, with some adjustments for Spark specifics.

Method Summary

Modifier and Type	Method and Description
static void	clearCache()
boolean	equals(Object o)
Coder<CheckpointMarkT>	getCheckpointMarkCoder()
Coder<T>	getDefaultOutputCoder()
String	getId()
Source.Reader<T>	getOrCreateReader(PipelineOptions options, CheckpointMarkT checkpointMark)
int	hashCode()
void	validate()

Methods inherited from class org.apache.beam.sdk.io.Source

populateDisplayData

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Method Detail

getOrCreateReader

public Source.Reader<T> getOrCreateReader(PipelineOptions options,
                                          CheckpointMarkT checkpointMark)
                                   throws IOException

Throws:

IOException

validate

public void validate()

Specified by:

validate in class Source<T>

getDefaultOutputCoder

public Coder<T> getDefaultOutputCoder()

Specified by:

getDefaultOutputCoder in class Source<T>

getCheckpointMarkCoder

public Coder<CheckpointMarkT> getCheckpointMarkCoder()

getId

public String getId()

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

clearCache

public static void clearCache()