org.apache.beam.runners.spark.translation

Class EvaluationContext

java.lang.Object

org.apache.beam.runners.spark.translation.EvaluationContext

public class EvaluationContext
extends Object

The EvaluationContext allows us to define pipeline instructions and translate between PObject<T>s or PCollection<T>s and Ts or DStreams/RDDs of Ts.

Constructor Summary

Constructors

Constructor and Description
EvaluationContext(org.apache.spark.api.java.JavaSparkContext jsc, Pipeline pipeline, PipelineOptions options)
EvaluationContext(org.apache.spark.api.java.JavaSparkContext jsc, Pipeline pipeline, PipelineOptions options, org.apache.spark.streaming.api.java.JavaStreamingContext jssc)

Method Summary

Modifier and Type	Method and Description
Dataset	borrowDataset(PTransform<? extends PValue,?> transform)
Dataset	borrowDataset(PValue pvalue)
void	computeOutputs() Computes the outputs for all RDDs that are leaves in the DAG and do not have any actions (like saving to a file) registered on them (i.e.
<T> T	get(PValue value) Retrieve an object of Type T associated with the PValue passed in.
Map<PCollection,Long>	getCacheCandidates() Get the map of cache candidates hold by the evaluation context.
AppliedPTransform<?,?,?>	getCurrentTransform()
<T extends PValue> T	getInput(PTransform<T,?> transform)
<T> Map<TupleTag<?>,PValue>	getInputs(PTransform<?,?> transform)
PipelineOptions	getOptions()
<T extends PValue> T	getOutput(PTransform<?,T> transform)
Map<TupleTag<?>,PValue>	getOutputs(PTransform<?,?> transform)
Pipeline	getPipeline()
SparkPCollectionView	getPViews() Retrun the current views creates in the pipepline.
SparkRuntimeContext	getRuntimeContext()
org.apache.spark.api.java.JavaSparkContext	getSparkContext()
org.apache.spark.streaming.api.java.JavaStreamingContext	getStreamingContext()
void	putDataset(PTransform<?,? extends PValue> transform, Dataset dataset)
void	putDataset(PValue pvalue, Dataset dataset)
void	putPView(PCollectionView<?> view, Iterable<WindowedValue<?>> value, Coder<Iterable<WindowedValue<?>>> coder) Adds/Replaces a view to the current views creates in the pipepline.
void	setCurrentTransform(AppliedPTransform<?,?,?> transform)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EvaluationContext

public EvaluationContext(org.apache.spark.api.java.JavaSparkContext jsc,
                         Pipeline pipeline,
                         PipelineOptions options)

EvaluationContext

public EvaluationContext(org.apache.spark.api.java.JavaSparkContext jsc,
                         Pipeline pipeline,
                         PipelineOptions options,
                         org.apache.spark.streaming.api.java.JavaStreamingContext jssc)

Method Detail

getSparkContext

public org.apache.spark.api.java.JavaSparkContext getSparkContext()

getStreamingContext

public org.apache.spark.streaming.api.java.JavaStreamingContext getStreamingContext()

getPipeline

public Pipeline getPipeline()

getOptions

public PipelineOptions getOptions()

getRuntimeContext

public SparkRuntimeContext getRuntimeContext()

setCurrentTransform

public void setCurrentTransform(AppliedPTransform<?,?,?> transform)

getCurrentTransform

public AppliedPTransform<?,?,?> getCurrentTransform()

getInput

public <T extends PValue> T getInput(PTransform<T,?> transform)

getInputs

public <T> Map<TupleTag<?>,PValue> getInputs(PTransform<?,?> transform)

getOutput

public <T extends PValue> T getOutput(PTransform<?,T> transform)

getOutputs

public Map<TupleTag<?>,PValue> getOutputs(PTransform<?,?> transform)

putDataset

public void putDataset(PTransform<?,? extends PValue> transform,
                       Dataset dataset)

putDataset

public void putDataset(PValue pvalue,
                       Dataset dataset)

borrowDataset

public Dataset borrowDataset(PTransform<? extends PValue,?> transform)

borrowDataset

public Dataset borrowDataset(PValue pvalue)

computeOutputs

public void computeOutputs()

Computes the outputs for all RDDs that are leaves in the DAG and do not have any actions (like saving to a file) registered on them (i.e. they are performed for side effects).

get

public <T> T get(PValue value)

Retrieve an object of Type T associated with the PValue passed in.

Type Parameters:

T - Type of object to return.

Parameters:

value - PValue to retrieve associated data for.

Returns:

Native object.

getPViews

public SparkPCollectionView getPViews()

Retrun the current views creates in the pipepline.

Returns:

SparkPCollectionView

putPView

public void putPView(PCollectionView<?> view,
                     Iterable<WindowedValue<?>> value,
                     Coder<Iterable<WindowedValue<?>>> coder)

Adds/Replaces a view to the current views creates in the pipepline.

Parameters:

view - - Identifier of the view

value - - Actual value of the view

coder - - Coder of the value

getCacheCandidates

public Map<PCollection,Long> getCacheCandidates()

Get the map of cache candidates hold by the evaluation context.

Returns:

The current Map of cache candidates.