org.apache.hadoop.yarn.server.nodemanager

Class ContainerExecutor

java.lang.Object

org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

Direct Known Subclasses:

DefaultContainerExecutor, DockerContainerExecutor, LinuxContainerExecutor

public abstract class ContainerExecutor
extends Object
implements org.apache.hadoop.conf.Configurable

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ContainerExecutor.DelayedProcessKiller
static class	ContainerExecutor.ExitCode
static class	ContainerExecutor.Signal The constants for the signals.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.permission.FsPermission	TASK_LAUNCH_SCRIPT_PERMISSION

Constructor Summary

Constructors

Constructor and Description
ContainerExecutor()

Method Summary

Modifier and Type	Method and Description
void	activateContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.fs.Path pidFilePath) Mark the container as active
void	deactivateContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) Mark the container as inactive.
abstract void	deleteAsUser(String user, org.apache.hadoop.fs.Path subDir, org.apache.hadoop.fs.Path... basedirs)
org.apache.hadoop.conf.Configuration	getConf()
protected org.apache.hadoop.fs.Path	getPidFilePath(org.apache.hadoop.yarn.api.records.ContainerId containerId) Get the pidFile of the container.
String	getProcessId(org.apache.hadoop.yarn.api.records.ContainerId containerID) Get the process-identifier for the container
protected String[]	getRunCommand(String command, String groupId, String userName, org.apache.hadoop.fs.Path pidFile, org.apache.hadoop.conf.Configuration conf)
protected String[]	getRunCommand(String command, String groupId, String userName, org.apache.hadoop.fs.Path pidFile, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.api.records.Resource resource) Return a command to execute the given command in OS shell.
abstract void	init() Run the executor initialization steps.
protected boolean	isContainerActive(org.apache.hadoop.yarn.api.records.ContainerId containerId) Is the container still active?
abstract boolean	isContainerProcessAlive(String user, String pid)
abstract int	launchContainer(Container container, org.apache.hadoop.fs.Path nmPrivateContainerScriptPath, org.apache.hadoop.fs.Path nmPrivateTokensPath, String user, String appId, org.apache.hadoop.fs.Path containerWorkDir, List<String> localDirs, List<String> logDirs) Launch the container on the node.
org.apache.hadoop.fs.Path	localizeClasspathJar(org.apache.hadoop.fs.Path classPathJar, org.apache.hadoop.fs.Path pwd, String owner) On Windows the ContainerLaunch creates a temporary special jar manifest of other jars to workaround the CLASSPATH length.
protected void	logOutput(String output)
int	reacquireContainer(String user, org.apache.hadoop.yarn.api.records.ContainerId containerId) Recover an already existing container.
void	setConf(org.apache.hadoop.conf.Configuration conf)
abstract boolean	signalContainer(String user, String pid, ContainerExecutor.Signal signal)
abstract void	startLocalizer(org.apache.hadoop.fs.Path nmPrivateContainerTokens, InetSocketAddress nmAddr, String user, String appId, String locId, LocalDirsHandlerService dirsHandler) Prepare the environment for containers in this application to execute.
void	writeLaunchEnv(OutputStream out, Map<String,String> environment, Map<org.apache.hadoop.fs.Path,List<String>> resources, List<String> command)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TASK_LAUNCH_SCRIPT_PERMISSION

public static final org.apache.hadoop.fs.permission.FsPermission TASK_LAUNCH_SCRIPT_PERMISSION

Constructor Detail

ContainerExecutor

public ContainerExecutor()

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

init

public abstract void init()
                   throws IOException

Run the executor initialization steps. Verify that the necessary configs, permissions are in place.

Throws:

IOException

localizeClasspathJar

public org.apache.hadoop.fs.Path localizeClasspathJar(org.apache.hadoop.fs.Path classPathJar,
                                                      org.apache.hadoop.fs.Path pwd,
                                                      String owner)
                                               throws IOException

On Windows the ContainerLaunch creates a temporary special jar manifest of other jars to workaround the CLASSPATH length. In a secure cluster this jar must be localized so that the container has access to it. This function localizes on-demand the jar.

Parameters:

classPathJar -

owner -

Throws:

IOException

startLocalizer

public abstract void startLocalizer(org.apache.hadoop.fs.Path nmPrivateContainerTokens,
                                    InetSocketAddress nmAddr,
                                    String user,
                                    String appId,
                                    String locId,
                                    LocalDirsHandlerService dirsHandler)
                             throws IOException,
                                    InterruptedException

Prepare the environment for containers in this application to execute.

 For $x in local.dirs
   create $x/$user/$appId
 Copy $nmLocal/appTokens  -> $N/$user/$appId
 For $rsrc in private resources
   Copy $rsrc  -> $N/$user/filecache/[idef]
 For $rsrc in job resources
   Copy $rsrc  -> $N/$user/$appId/filecache/idef
 

Parameters:

user - user name of application owner

appId - id of the application

nmPrivateContainerTokens - path to localized credentials, rsrc by NM

nmAddr - RPC address to contact NM

dirsHandler - NM local dirs service, for nm-local-dirs and nm-log-dirs

Throws:

IOException - For most application init failures

InterruptedException - If application init thread is halted by NM

launchContainer

public abstract int launchContainer(Container container,
                                    org.apache.hadoop.fs.Path nmPrivateContainerScriptPath,
                                    org.apache.hadoop.fs.Path nmPrivateTokensPath,
                                    String user,
                                    String appId,
                                    org.apache.hadoop.fs.Path containerWorkDir,
                                    List<String> localDirs,
                                    List<String> logDirs)
                             throws IOException

Launch the container on the node. This is a blocking call and returns only when the container exits.

Parameters:

container - the container to be launched

nmPrivateContainerScriptPath - the path for launch script

nmPrivateTokensPath - the path for tokens for the container

user - the user of the container

appId - the appId of the container

containerWorkDir - the work dir for the container

localDirs - nm-local-dirs to be used for this container

logDirs - nm-log-dirs to be used for this container

Returns:

the return status of the launch

Throws:

IOException

signalContainer

public abstract boolean signalContainer(String user,
                                        String pid,
                                        ContainerExecutor.Signal signal)
                                 throws IOException

Throws:

IOException

deleteAsUser

public abstract void deleteAsUser(String user,
                                  org.apache.hadoop.fs.Path subDir,
                                  org.apache.hadoop.fs.Path... basedirs)
                           throws IOException,
                                  InterruptedException

Throws:

IOException

InterruptedException

isContainerProcessAlive

public abstract boolean isContainerProcessAlive(String user,
                                                String pid)
                                         throws IOException

Throws:

IOException

reacquireContainer

public int reacquireContainer(String user,
                              org.apache.hadoop.yarn.api.records.ContainerId containerId)
                       throws IOException,
                              InterruptedException

Recover an already existing container. This is a blocking call and returns only when the container exits. Note that the container must have been activated prior to this call.

Parameters:

user - the user of the container

containerId - The ID of the container to reacquire

Returns:

The exit code of the pre-existing container

Throws:

IOException

InterruptedException

writeLaunchEnv

public void writeLaunchEnv(OutputStream out,
                           Map<String,String> environment,
                           Map<org.apache.hadoop.fs.Path,List<String>> resources,
                           List<String> command)
                    throws IOException

Throws:

IOException

logOutput

protected void logOutput(String output)

getPidFilePath

protected org.apache.hadoop.fs.Path getPidFilePath(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Get the pidFile of the container.

Parameters:

containerId -

Returns:

the path of the pid-file for the given containerId.

getRunCommand

protected String[] getRunCommand(String command,
                                 String groupId,
                                 String userName,
                                 org.apache.hadoop.fs.Path pidFile,
                                 org.apache.hadoop.conf.Configuration conf)

getRunCommand

protected String[] getRunCommand(String command,
                                 String groupId,
                                 String userName,
                                 org.apache.hadoop.fs.Path pidFile,
                                 org.apache.hadoop.conf.Configuration conf,
                                 org.apache.hadoop.yarn.api.records.Resource resource)

Return a command to execute the given command in OS shell. On Windows, the passed in groupId can be used to launch and associate the given groupId in a process group. On non-Windows, groupId is ignored.

isContainerActive

protected boolean isContainerActive(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Is the container still active?

Parameters:

containerId -

Returns:

true if the container is active else false.

activateContainer

public void activateContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                              org.apache.hadoop.fs.Path pidFilePath)

Mark the container as active

Parameters:

containerId - the ContainerId

pidFilePath - Path where the executor should write the pid of the launched process

deactivateContainer

public void deactivateContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Mark the container as inactive. Done iff the container is still active. Else treat it as a no-op

getProcessId

public String getProcessId(org.apache.hadoop.yarn.api.records.ContainerId containerID)

Get the process-identifier for the container

Parameters:

containerID -

Returns:

the processid of the container if it has already launched, otherwise return null