org.apache.beam.runners.spark.io

Class MicrobatchSource.Reader

java.lang.Object

org.apache.beam.sdk.io.Source.Reader<T>

org.apache.beam.runners.spark.io.MicrobatchSource.Reader

All Implemented Interfaces:

AutoCloseable

Enclosing class:

MicrobatchSource<T,CheckpointMarkT extends UnboundedSource.CheckpointMark>

public class MicrobatchSource.Reader
extends Source.Reader<T>

Mostly based on BoundedReadFromUnboundedSource's UnboundedToBoundedSourceAdapter, with some adjustments for Spark specifics.

This Reader reads until one of the following thresholds has been reached:

1. max records (per batch)
2. max read duration (per batch)

Method Summary

Modifier and Type	Method and Description
boolean	advance()
void	close()
CheckpointMarkT	getCheckpointMark()
T	getCurrent()
Source<T>	getCurrentSource()
org.joda.time.Instant	getCurrentTimestamp()
org.joda.time.Instant	getWatermark()
boolean	start()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

start

public boolean start()
              throws IOException

Specified by:

start in class Source.Reader<T>

Throws:

IOException

advance

public boolean advance()
                throws IOException

Specified by:

advance in class Source.Reader<T>

Throws:

IOException

getCurrent

public T getCurrent()
             throws NoSuchElementException

Specified by:

getCurrent in class Source.Reader<T>

Throws:

NoSuchElementException

getCurrentTimestamp

public org.joda.time.Instant getCurrentTimestamp()
                                          throws NoSuchElementException

Specified by:

getCurrentTimestamp in class Source.Reader<T>

Throws:

NoSuchElementException

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in class Source.Reader<T>

Throws:

IOException

getCurrentSource

public Source<T> getCurrentSource()

Specified by:

getCurrentSource in class Source.Reader<T>

getCheckpointMark

public CheckpointMarkT getCheckpointMark()

getWatermark

public org.joda.time.Instant getWatermark()