org.apache.beam.runners.spark.translation

Class MultiDoFnFunction<InputT,OutputT>

java.lang.Object

org.apache.beam.runners.spark.translation.MultiDoFnFunction<InputT,OutputT>

Type Parameters:

InputT - Input type for DoFunction.

OutputT - Output type for DoFunction.

All Implemented Interfaces:

Serializable, org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<WindowedValue<InputT>>,TupleTag<?>,WindowedValue<?>>

public class MultiDoFnFunction<InputT,OutputT>
extends Object
implements org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<WindowedValue<InputT>>,TupleTag<?>,WindowedValue<?>>

DoFunctions ignore outputs that are not the main output. MultiDoFunctions deal with additional outputs by enriching the underlying data with multiple TupleTags.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description

MultiDoFnFunction(org.apache.spark.Accumulator<NamedAggregators> aggAccum, org.apache.spark.Accumulator<MetricsContainerStepMap> metricsAccum, String stepName, DoFn<InputT,OutputT> doFn, SparkRuntimeContext runtimeContext, TupleTag<OutputT> mainOutputTag, List<TupleTag<?>> additionalOutputTags, Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs, WindowingStrategy<?,?> windowingStrategy, boolean stateful)

Method Summary

Modifier and Type	Method and Description
Iterable<scala.Tuple2<TupleTag<?>,WindowedValue<?>>>	call(Iterator<WindowedValue<InputT>> iter)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MultiDoFnFunction

public MultiDoFnFunction(org.apache.spark.Accumulator<NamedAggregators> aggAccum,
                         org.apache.spark.Accumulator<MetricsContainerStepMap> metricsAccum,
                         String stepName,
                         DoFn<InputT,OutputT> doFn,
                         SparkRuntimeContext runtimeContext,
                         TupleTag<OutputT> mainOutputTag,
                         List<TupleTag<?>> additionalOutputTags,
                         Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs,
                         WindowingStrategy<?,?> windowingStrategy,
                         boolean stateful)

Parameters:

aggAccum - The Spark Accumulator that backs the Beam Aggregators.

metricsAccum - The Spark Accumulator that backs the Beam metrics.

doFn - The DoFn to be wrapped.

runtimeContext - The SparkRuntimeContext.

mainOutputTag - The main output TupleTag.

additionalOutputTags - Additional output tags.

sideInputs - Side inputs used in this DoFn.

windowingStrategy - Input WindowingStrategy.

stateful - Stateful DoFn.

Method Detail

call

public Iterable<scala.Tuple2<TupleTag<?>,WindowedValue<?>>> call(Iterator<WindowedValue<InputT>> iter)
                                                          throws Exception

Specified by:

call in interface org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<WindowedValue<InputT>>,TupleTag<?>,WindowedValue<?>>

Throws:

Exception