org.apache.beam.runners.spark.translation

Class SparkContextFactory

java.lang.Object

org.apache.beam.runners.spark.translation.SparkContextFactory

public final class SparkContextFactory
extends Object

The Spark context factory.

Field Summary

Fields

Modifier and Type	Field and Description
static String	TEST_REUSE_SPARK_CONTEXT If the property beam.spark.test.reuseSparkContext is set to true then the Spark context will be reused for beam pipelines.

Method Summary

Modifier and Type	Method and Description
static org.apache.spark.api.java.JavaSparkContext	getSparkContext(SparkPipelineOptions options)
static void	stopSparkContext(org.apache.spark.api.java.JavaSparkContext context)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEST_REUSE_SPARK_CONTEXT

public static final String TEST_REUSE_SPARK_CONTEXT

If the property beam.spark.test.reuseSparkContext is set to true then the Spark context will be reused for beam pipelines. This property should only be enabled for tests.

See Also:

Constant Field Values

Method Detail

getSparkContext

public static org.apache.spark.api.java.JavaSparkContext getSparkContext(SparkPipelineOptions options)

stopSparkContext

public static void stopSparkContext(org.apache.spark.api.java.JavaSparkContext context)