org.apache.beam.runners.spark.translation

Class GroupCombineFunctions

java.lang.Object

org.apache.beam.runners.spark.translation.GroupCombineFunctions

public class GroupCombineFunctions
extends Object

A set of group/combine functions to apply to Spark RDDs.

Constructor Summary

Constructors

Constructor and Description
GroupCombineFunctions()

Method Summary

Modifier and Type	Method and Description
static <InputT,AccumT> com.google.common.base.Optional<Iterable<WindowedValue<AccumT>>>	combineGlobally(org.apache.spark.api.java.JavaRDD<WindowedValue<InputT>> rdd, SparkGlobalCombineFn<InputT,AccumT,?> sparkCombineFn, Coder<InputT> iCoder, Coder<AccumT> aCoder, WindowingStrategy<?,?> windowingStrategy) Apply a composite Combine.Globally transformation.
static <K,InputT,AccumT> org.apache.spark.api.java.JavaPairRDD<K,Iterable<WindowedValue<KV<K,AccumT>>>>	combinePerKey(org.apache.spark.api.java.JavaRDD<WindowedValue<KV<K,InputT>>> rdd, SparkKeyedCombineFn<K,InputT,AccumT,?> sparkCombineFn, Coder<K> keyCoder, Coder<InputT> iCoder, Coder<AccumT> aCoder, WindowingStrategy<?,?> windowingStrategy) Apply a composite Combine.PerKey transformation.
static <K,V> org.apache.spark.api.java.JavaRDD<WindowedValue<KV<K,Iterable<WindowedValue<V>>>>>	groupByKeyOnly(org.apache.spark.api.java.JavaRDD<WindowedValue<KV<K,V>>> rdd, Coder<K> keyCoder, WindowedValue.WindowedValueCoder<V> wvCoder) An implementation of GroupByKeyViaGroupByKeyOnly.GroupByKeyOnly for the Spark runner.
static <K,V> org.apache.spark.api.java.JavaRDD<WindowedValue<KV<K,V>>>	reshuffle(org.apache.spark.api.java.JavaRDD<WindowedValue<KV<K,V>>> rdd, Coder<K> keyCoder, WindowedValue.WindowedValueCoder<V> wvCoder) An implementation of Reshuffle for the Spark runner.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GroupCombineFunctions

public GroupCombineFunctions()

Method Detail

groupByKeyOnly

public static <K,V> org.apache.spark.api.java.JavaRDD<WindowedValue<KV<K,Iterable<WindowedValue<V>>>>> groupByKeyOnly(org.apache.spark.api.java.JavaRDD<WindowedValue<KV<K,V>>> rdd,
                                                                                                                      Coder<K> keyCoder,
                                                                                                                      WindowedValue.WindowedValueCoder<V> wvCoder)

An implementation of GroupByKeyViaGroupByKeyOnly.GroupByKeyOnly for the Spark runner.

combineGlobally

public static <InputT,AccumT> com.google.common.base.Optional<Iterable<WindowedValue<AccumT>>> combineGlobally(org.apache.spark.api.java.JavaRDD<WindowedValue<InputT>> rdd,
                                                                                                               SparkGlobalCombineFn<InputT,AccumT,?> sparkCombineFn,
                                                                                                               Coder<InputT> iCoder,
                                                                                                               Coder<AccumT> aCoder,
                                                                                                               WindowingStrategy<?,?> windowingStrategy)

Apply a composite Combine.Globally transformation.

combinePerKey

public static <K,InputT,AccumT> org.apache.spark.api.java.JavaPairRDD<K,Iterable<WindowedValue<KV<K,AccumT>>>> combinePerKey(org.apache.spark.api.java.JavaRDD<WindowedValue<KV<K,InputT>>> rdd,
                                                                                                                             SparkKeyedCombineFn<K,InputT,AccumT,?> sparkCombineFn,
                                                                                                                             Coder<K> keyCoder,
                                                                                                                             Coder<InputT> iCoder,
                                                                                                                             Coder<AccumT> aCoder,
                                                                                                                             WindowingStrategy<?,?> windowingStrategy)

Apply a composite Combine.PerKey transformation.

This aggregation will apply Beam's Combine.CombineFn via Spark's JavaPairRDD.combineByKey(Function, Function2, Function2) aggregation.

For streaming, this will be called from within a serialized context (DStream's transform callback), so passed arguments need to be Serializable.

reshuffle

public static <K,V> org.apache.spark.api.java.JavaRDD<WindowedValue<KV<K,V>>> reshuffle(org.apache.spark.api.java.JavaRDD<WindowedValue<KV<K,V>>> rdd,
                                                                                        Coder<K> keyCoder,
                                                                                        WindowedValue.WindowedValueCoder<V> wvCoder)

An implementation of Reshuffle for the Spark runner.