org.nd4j.parameterserver.distributed

Class VoidParameterServer

java.lang.Object

org.nd4j.parameterserver.distributed.VoidParameterServer

public class VoidParameterServer
extends Object

This is "special case" distributed P2P parameter server implementation, suitable for Spark Word2Vec/ParagraphVectors/DeepWalk implementations. Aeron is used as backbone for messaging system here. Highlights: a) It does ONLY one-way messaging. Clients are NOT getting anything back from ParamServer. b) It works sharded. Amount of shards is defined in runtime. c) Data replication strategy is defined in runtime. d) It's supposed to be used as singleton in Spark environment ONLY.

Author:

raver119@gmail.com

Field Summary

Fields

Modifier and Type	Field and Description
protected Clipboard	clipboard
protected ThreadPoolExecutor	executor
protected Map<String,Frame<TrainingMessage>>	frames
protected AtomicBoolean	initFinished
protected AtomicBoolean	initLocker
protected AtomicBoolean	manualMode
protected static double	MAX_EXP
protected NodeRole	nodeRole
protected static int	numThreads
protected Runnable[]	processingRunnables
protected Thread[]	processingThreads
protected AtomicBoolean	runner
protected short	shardIndex
protected AtomicBoolean	shutdownFinished
protected AtomicBoolean	shutdownLocker
protected Storage	storage
protected TrainingDriver<? extends TrainingMessage>	trainer
protected Transport	transport
protected VoidConfiguration	voidConfiguration

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	VoidParameterServer()
protected	VoidParameterServer(boolean manualMode)
protected	VoidParameterServer(NodeRole nodeRole)

Method Summary

Modifier and Type	Method and Description
void	execDistributed(Frame<? extends TrainingMessage> messages)
void	execDistributed(TrainingMessage message) This method dispatches TrainingMessage to ParameterServer network PLEASE NOTE: This method is synchronized and *periodically* becomes blocking by design
void	execDistributedImmediately(TrainingMessage message)
protected org.nd4j.linalg.api.ndarray.INDArray	getExpTable()
static VoidParameterServer	getInstance()
static Set<String>	getLocalAddresses() This method returns set of local IP addresses available in system.
protected org.nd4j.linalg.api.ndarray.INDArray	getNegTable()
protected org.nd4j.linalg.primitives.Pair<NodeRole,String>	getRole(VoidConfiguration voidConfiguration, Collection<String> localIPs) This method checks for designated role, according to local IP addresses and configuration passed into method
short	getShardIndex() This method returns shardIndex value.
protected org.nd4j.linalg.api.ndarray.INDArray	getSyn0()
protected org.nd4j.linalg.api.ndarray.INDArray	getSyn1()
protected org.nd4j.linalg.api.ndarray.INDArray	getSyn1Neg()
protected Transport	getTransport()
org.nd4j.linalg.api.ndarray.INDArray	getVector(int rowIdx)
org.nd4j.linalg.api.ndarray.INDArray	getVector(Integer key, int rowIdx) This method returns INDArray matching requested storageId value PLEASE NOTE: This method IS blocking
protected void	handleMessage(VoidMessage message)
protected void	init(VoidConfiguration voidConfiguration)
void	init(VoidConfiguration voidConfiguration, Transport transport, TrainingDriver<? extends TrainingMessage> trainer) This method starts ParameterServer instance PLEASE NOTE: This method is blocking for first caller only
void	initializeSeqVec(int vectorLength, int numWords, long seed, int columnsPerShard, boolean useHs, boolean useNegSampling) This method handles Shards initialization PLEASE NOTE: This method is blocking
boolean	isInit() This method returns True if initialization was started AND was finished, false otherwise
void	sendMessageToAllClients(VoidMessage message) This method sends given message to all Clients
void	sendMessageToAllClients(VoidMessage message, Long... exclusions) This method sends given message to all Clients, excluding
void	sendMessageToAllShards(VoidMessage message) This method sends given message to all Shards
protected void	setIpPortForShard(String ip, int port)
protected void	setShardIndex(short idx)
void	setTrainingDriver(TrainingDriver<? extends TrainingMessage> trainer)
void	shutdown() This method initiates shutdown sequence for this instance.
protected VoidParameterServer	toggleManualMode(boolean mode) This method is available for debug purposes only

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

nodeRole

protected volatile NodeRole nodeRole

voidConfiguration

protected volatile VoidConfiguration voidConfiguration

initLocker

protected AtomicBoolean initLocker

initFinished

protected AtomicBoolean initFinished

shutdownLocker

protected AtomicBoolean shutdownLocker

shutdownFinished

protected AtomicBoolean shutdownFinished

transport

protected transient Transport transport

manualMode

protected transient AtomicBoolean manualMode

runner

protected transient AtomicBoolean runner

processingThreads

protected transient Thread[] processingThreads

processingRunnables

protected transient Runnable[] processingRunnables

trainer

protected transient TrainingDriver<? extends TrainingMessage> trainer

shardIndex

protected short shardIndex

clipboard

protected Clipboard clipboard

storage

protected Storage storage

frames

protected Map<String,Frame<TrainingMessage>> frames

numThreads

protected static final int numThreads

executor

protected ThreadPoolExecutor executor

MAX_EXP

protected static double MAX_EXP

Constructor Detail

VoidParameterServer

protected VoidParameterServer()

VoidParameterServer

protected VoidParameterServer(@NonNull
                              NodeRole nodeRole)

VoidParameterServer

protected VoidParameterServer(boolean manualMode)

Method Detail

getInstance

public static VoidParameterServer getInstance()

setTrainingDriver

public void setTrainingDriver(@NonNull
                              TrainingDriver<? extends TrainingMessage> trainer)

getShardIndex

public short getShardIndex()

This method returns shardIndex value. If current node is Shard - it reutrns it's value If current node is client - it returns Shard index of paired Shard

Returns:

setIpPortForShard

protected void setIpPortForShard(String ip,
                                 int port)

setShardIndex

protected void setShardIndex(short idx)

getTransport

protected Transport getTransport()

getSyn0

protected org.nd4j.linalg.api.ndarray.INDArray getSyn0()

getSyn1

protected org.nd4j.linalg.api.ndarray.INDArray getSyn1()

getSyn1Neg

protected org.nd4j.linalg.api.ndarray.INDArray getSyn1Neg()

getExpTable

protected org.nd4j.linalg.api.ndarray.INDArray getExpTable()

getNegTable

protected org.nd4j.linalg.api.ndarray.INDArray getNegTable()

init

protected void init(@NonNull
                    VoidConfiguration voidConfiguration)

isInit

public boolean isInit()

This method returns True if initialization was started AND was finished, false otherwise

Returns:

init

public void init(@NonNull
                 VoidConfiguration voidConfiguration,
                 @NonNull
                 Transport transport,
                 TrainingDriver<? extends TrainingMessage> trainer)

This method starts ParameterServer instance PLEASE NOTE: This method is blocking for first caller only

toggleManualMode

protected VoidParameterServer toggleManualMode(boolean mode)

This method is available for debug purposes only

Parameters:

mode -

getRole

protected org.nd4j.linalg.primitives.Pair<NodeRole,String> getRole(@NonNull
                                                                   VoidConfiguration voidConfiguration,
                                                                   @NonNull
                                                                   Collection<String> localIPs)

This method checks for designated role, according to local IP addresses and configuration passed into method

Parameters:

voidConfiguration -

localIPs -

Returns:

shutdown

public void shutdown()

This method initiates shutdown sequence for this instance. PLEASE NOTE: This method is blocking for first caller only

getLocalAddresses

public static Set<String> getLocalAddresses()

This method returns set of local IP addresses available in system. PLEASE NOTE: loopback, disabled interfaces, IPv6 addresses are ignored here.

Returns:

handleMessage

protected void handleMessage(@NonNull
                             VoidMessage message)

initializeSeqVec

public void initializeSeqVec(int vectorLength,
                             int numWords,
                             long seed,
                             int columnsPerShard,
                             boolean useHs,
                             boolean useNegSampling)

This method handles Shards initialization PLEASE NOTE: This method is blocking

execDistributed

public void execDistributed(@NonNull
                            TrainingMessage message)

This method dispatches TrainingMessage to ParameterServer network PLEASE NOTE: This method is synchronized and *periodically* becomes blocking by design

Parameters:

message -

execDistributedImmediately

public void execDistributedImmediately(@NonNull
                                       TrainingMessage message)

execDistributed

public void execDistributed(@NonNull
                            Frame<? extends TrainingMessage> messages)

getVector

public org.nd4j.linalg.api.ndarray.INDArray getVector(int rowIdx)

getVector

public org.nd4j.linalg.api.ndarray.INDArray getVector(@NonNull
                                                      Integer key,
                                                      int rowIdx)

This method returns INDArray matching requested storageId value PLEASE NOTE: This method IS blocking

Parameters:

rowIdx -

Returns:

sendMessageToAllShards

public void sendMessageToAllShards(@NonNull
                                   VoidMessage message)

This method sends given message to all Shards

Parameters:

message -

sendMessageToAllClients

public void sendMessageToAllClients(@NonNull
                                    VoidMessage message)

This method sends given message to all Clients

Parameters:

message -

sendMessageToAllClients

public void sendMessageToAllClients(@NonNull
                                    VoidMessage message,
                                    Long... exclusions)

This method sends given message to all Clients, excluding

Parameters:

message -