org.apache.beam.runners.spark.translation

Class SparkAbstractCombineFn

java.lang.Object

org.apache.beam.runners.spark.translation.SparkAbstractCombineFn

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

SparkGlobalCombineFn, SparkKeyedCombineFn

public class SparkAbstractCombineFn
extends Object
implements Serializable

An abstract for the SparkRunner implementation of Combine.CombineFn.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected SparkRuntimeContext	runtimeContext
protected Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>>	sideInputs
protected WindowingStrategy<?,BoundedWindow>	windowingStrategy

Constructor Summary

Constructors

Constructor and Description
SparkAbstractCombineFn(SparkRuntimeContext runtimeContext, Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs, WindowingStrategy<?,?> windowingStrategy)

Method Summary

Modifier and Type	Method and Description
protected org.apache.beam.runners.spark.translation.SparkAbstractCombineFn.SparkCombineContext	ctxtForInput(WindowedValue<?> input)
protected static boolean	isIntersecting(IntervalWindow union, IntervalWindow window)
protected static IntervalWindow	merge(IntervalWindow union, IntervalWindow window)
protected static <T> Iterable<WindowedValue<T>>	sortByWindows(Iterable<WindowedValue<T>> iter)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

runtimeContext

protected final SparkRuntimeContext runtimeContext

sideInputs

protected final Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs

windowingStrategy

protected final WindowingStrategy<?,BoundedWindow> windowingStrategy

Constructor Detail

SparkAbstractCombineFn

public SparkAbstractCombineFn(SparkRuntimeContext runtimeContext,
                              Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>> sideInputs,
                              WindowingStrategy<?,?> windowingStrategy)

Method Detail

ctxtForInput

protected org.apache.beam.runners.spark.translation.SparkAbstractCombineFn.SparkCombineContext ctxtForInput(WindowedValue<?> input)

sortByWindows

protected static <T> Iterable<WindowedValue<T>> sortByWindows(Iterable<WindowedValue<T>> iter)

isIntersecting

protected static boolean isIntersecting(IntervalWindow union,
                                        IntervalWindow window)

merge

protected static IntervalWindow merge(IntervalWindow union,
                                      IntervalWindow window)