org.apache.beam.runners.spark.translation

Class TranslationUtils

java.lang.Object

org.apache.beam.runners.spark.translation.TranslationUtils

public final class TranslationUtils
extends Object

A set of utilities to help translating Beam transformations into Spark transformations.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TranslationUtils.CombineGroupedValues<K,InputT,OutputT> A SparkKeyedCombineFn function applied to grouped KVs.
static class	TranslationUtils.TupleTagFilter<V> A utility class to filter TupleTags.

Method Summary

Modifier and Type	Method and Description
static <T1,T2> org.apache.spark.streaming.api.java.JavaDStream<T2>	dStreamValues(org.apache.spark.streaming.api.java.JavaPairDStream<T1,T2> pairDStream) Transform a pair stream into a value stream.
static <T> org.apache.spark.api.java.function.VoidFunction<T>	emptyVoidFunction()
static <InputT,OutputT> org.apache.spark.api.java.function.FlatMapFunction<Iterator<InputT>,OutputT>	functionToFlatMapFunction(org.apache.spark.api.java.function.Function<InputT,OutputT> func) A utility method that adapts Function to a FlatMapFunction with an Iterator input.
static Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>>	getSideInputs(List<PCollectionView<?>> views, org.apache.spark.api.java.JavaSparkContext context, SparkPCollectionView pviews) Create SideInputs as Broadcast variables.
static <T,K,V> org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<T>,K,V>	pairFunctionToPairFlatMapFunction(org.apache.spark.api.java.function.PairFunction<T,K,V> pairFunction) A utility method that adapts PairFunction to a PairFlatMapFunction with an Iterator input.
static void	rejectSplittable(DoFn<?,?> doFn)
static void	rejectStateAndTimers(DoFn<?,?> doFn) Reject state and timers DoFn.
static <T,W extends BoundedWindow> boolean	skipAssignWindows(Window.Assign<T> transform, EvaluationContext context) Checks if the window transformation should be applied or skipped.
static <K,V> org.apache.spark.api.java.function.PairFunction<WindowedValue<KV<K,V>>,K,WindowedValue<KV<K,V>>>	toPairByKeyInWindowedValue() Extract key from a WindowedValue KV into a pair.
static <K,V> org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<KV<K,V>>,K,V>	toPairFlatMapFunction() KV to pair flatmap function.
static <K,V> org.apache.spark.api.java.function.PairFunction<KV<K,V>,K,V>	toPairFunction() KV to pair function.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

skipAssignWindows

public static <T,W extends BoundedWindow> boolean skipAssignWindows(Window.Assign<T> transform,
                                                                    EvaluationContext context)

Checks if the window transformation should be applied or skipped.

Avoid running assign windows if both source and destination are global window or if the user has not specified the WindowFn (meaning they are just messing with triggering or allowed lateness).

Type Parameters:

T - PCollection type.

W - BoundedWindow type.

Parameters:

transform - The Window.Assign transformation.

context - The EvaluationContext.

Returns:

if to apply the transformation.

dStreamValues

public static <T1,T2> org.apache.spark.streaming.api.java.JavaDStream<T2> dStreamValues(org.apache.spark.streaming.api.java.JavaPairDStream<T1,T2> pairDStream)

Transform a pair stream into a value stream.

toPairFunction

public static <K,V> org.apache.spark.api.java.function.PairFunction<KV<K,V>,K,V> toPairFunction()

KV to pair function.

toPairFlatMapFunction

public static <K,V> org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<KV<K,V>>,K,V> toPairFlatMapFunction()

KV to pair flatmap function.

toPairByKeyInWindowedValue

public static <K,V> org.apache.spark.api.java.function.PairFunction<WindowedValue<KV<K,V>>,K,WindowedValue<KV<K,V>>> toPairByKeyInWindowedValue()

Extract key from a WindowedValue KV into a pair.

getSideInputs

public static Map<TupleTag<?>,KV<WindowingStrategy<?,?>,SideInputBroadcast<?>>> getSideInputs(List<PCollectionView<?>> views,
                                                                                              org.apache.spark.api.java.JavaSparkContext context,
                                                                                              SparkPCollectionView pviews)

Create SideInputs as Broadcast variables.

Parameters:

views - The PCollectionViews.

context - The JavaSparkContext.

pviews - The SparkPCollectionView.

Returns:

a map of tagged SideInputBroadcasts and their WindowingStrategy.

rejectSplittable

public static void rejectSplittable(DoFn<?,?> doFn)

rejectStateAndTimers

public static void rejectStateAndTimers(DoFn<?,?> doFn)

Reject state and timers DoFn.

Parameters:

doFn - the DoFn to possibly reject.

emptyVoidFunction

public static <T> org.apache.spark.api.java.function.VoidFunction<T> emptyVoidFunction()

pairFunctionToPairFlatMapFunction

public static <T,K,V> org.apache.spark.api.java.function.PairFlatMapFunction<Iterator<T>,K,V> pairFunctionToPairFlatMapFunction(org.apache.spark.api.java.function.PairFunction<T,K,V> pairFunction)

A utility method that adapts PairFunction to a PairFlatMapFunction with an Iterator input. This is particularly useful because it allows to use functions written for mapToPair functions in flatmapToPair functions.

Type Parameters:

T - the input type.

K - the output key type.

V - the output value type.

Parameters:

pairFunction - the PairFunction to adapt.

Returns:

a PairFlatMapFunction that accepts an Iterator as an input and applies the PairFunction on every element.

functionToFlatMapFunction

public static <InputT,OutputT> org.apache.spark.api.java.function.FlatMapFunction<Iterator<InputT>,OutputT> functionToFlatMapFunction(org.apache.spark.api.java.function.Function<InputT,OutputT> func)

A utility method that adapts Function to a FlatMapFunction with an Iterator input. This is particularly useful because it allows to use functions written for map functions in flatmap functions.

Type Parameters:

InputT - the input type.

OutputT - the output type.

Parameters:

func - the Function to adapt.

Returns:

a FlatMapFunction that accepts an Iterator as an input and applies the Function on every element.