org.apache.hadoop.yarn.server.nodemanager

Class DockerContainerExecutor

java.lang.Object

org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

public class DockerContainerExecutor
extends ContainerExecutor

This executor will launch a docker container and run the task inside the container.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

ContainerExecutor.DelayedProcessKiller, ContainerExecutor.ExitCode, ContainerExecutor.Signal

Field Summary

Fields

Modifier and Type	Field and Description
static String	DOCKER_CONTAINER_EXECUTOR_SCRIPT
static String	DOCKER_CONTAINER_EXECUTOR_SESSION_SCRIPT
static String	DOCKER_IMAGE_PATTERN

Fields inherited from class org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

TASK_LAUNCH_SCRIPT_PERMISSION

Constructor Summary

Constructors

Constructor and Description
DockerContainerExecutor()

Method Summary

Modifier and Type	Method and Description
static boolean	containerIsAlive(String pid) Returns true if the process with the specified pid is alive.
protected void	copyFile(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst, String owner)
protected void	createDir(org.apache.hadoop.fs.Path dirPath, org.apache.hadoop.fs.permission.FsPermission perms, boolean createParent, String user)
void	deleteAsUser(String user, org.apache.hadoop.fs.Path subDir, org.apache.hadoop.fs.Path... baseDirs)
protected org.apache.hadoop.fs.Path	getWorkingDir(List<String> localDirs, String user, String appId)
void	init() Run the executor initialization steps.
boolean	isContainerProcessAlive(String user, String pid)
protected void	killContainer(String pid, ContainerExecutor.Signal signal) Send a specified signal to the specified pid
int	launchContainer(Container container, org.apache.hadoop.fs.Path nmPrivateContainerScriptPath, org.apache.hadoop.fs.Path nmPrivateTokensPath, String userName, String appId, org.apache.hadoop.fs.Path containerWorkDir, List<String> localDirs, List<String> logDirs) Launch the container on the node.
boolean	signalContainer(String user, String pid, ContainerExecutor.Signal signal)
void	startLocalizer(org.apache.hadoop.fs.Path nmPrivateContainerTokensPath, InetSocketAddress nmAddr, String user, String appId, String locId, LocalDirsHandlerService dirsHandler) Prepare the environment for containers in this application to execute.
void	writeLaunchEnv(OutputStream out, Map<String,String> environment, Map<org.apache.hadoop.fs.Path,List<String>> resources, List<String> command)

Methods inherited from class org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor

activateContainer, deactivateContainer, getConf, getPidFilePath, getProcessId, getRunCommand, getRunCommand, isContainerActive, localizeClasspathJar, logOutput, reacquireContainer, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DOCKER_CONTAINER_EXECUTOR_SCRIPT

public static final String DOCKER_CONTAINER_EXECUTOR_SCRIPT

See Also:

Constant Field Values

DOCKER_CONTAINER_EXECUTOR_SESSION_SCRIPT

public static final String DOCKER_CONTAINER_EXECUTOR_SESSION_SCRIPT

See Also:

Constant Field Values

DOCKER_IMAGE_PATTERN

public static final String DOCKER_IMAGE_PATTERN

See Also:

Constant Field Values

Constructor Detail

DockerContainerExecutor

public DockerContainerExecutor()

Method Detail

copyFile

protected void copyFile(org.apache.hadoop.fs.Path src,
                        org.apache.hadoop.fs.Path dst,
                        String owner)
                 throws IOException

Throws:

IOException

init

public void init()
          throws IOException

Description copied from class: ContainerExecutor

Run the executor initialization steps. Verify that the necessary configs, permissions are in place.

Specified by:

init in class ContainerExecutor

Throws:

IOException

startLocalizer

public void startLocalizer(org.apache.hadoop.fs.Path nmPrivateContainerTokensPath,
                           InetSocketAddress nmAddr,
                           String user,
                           String appId,
                           String locId,
                           LocalDirsHandlerService dirsHandler)
                    throws IOException,
                           InterruptedException

Description copied from class: ContainerExecutor

Prepare the environment for containers in this application to execute.

 For $x in local.dirs
   create $x/$user/$appId
 Copy $nmLocal/appTokens  -> $N/$user/$appId
 For $rsrc in private resources
   Copy $rsrc  -> $N/$user/filecache/[idef]
 For $rsrc in job resources
   Copy $rsrc  -> $N/$user/$appId/filecache/idef
 

Specified by:

startLocalizer in class ContainerExecutor

Parameters:

nmPrivateContainerTokensPath - path to localized credentials, rsrc by NM

nmAddr - RPC address to contact NM

user - user name of application owner

appId - id of the application

dirsHandler - NM local dirs service, for nm-local-dirs and nm-log-dirs

Throws:

IOException - For most application init failures

InterruptedException - If application init thread is halted by NM

launchContainer

public int launchContainer(Container container,
                           org.apache.hadoop.fs.Path nmPrivateContainerScriptPath,
                           org.apache.hadoop.fs.Path nmPrivateTokensPath,
                           String userName,
                           String appId,
                           org.apache.hadoop.fs.Path containerWorkDir,
                           List<String> localDirs,
                           List<String> logDirs)
                    throws IOException

Description copied from class: ContainerExecutor

Launch the container on the node. This is a blocking call and returns only when the container exits.

Specified by:

launchContainer in class ContainerExecutor

Parameters:

container - the container to be launched

nmPrivateContainerScriptPath - the path for launch script

nmPrivateTokensPath - the path for tokens for the container

userName - the user of the container

appId - the appId of the container

containerWorkDir - the work dir for the container

localDirs - nm-local-dirs to be used for this container

logDirs - nm-log-dirs to be used for this container

Returns:

the return status of the launch

Throws:

IOException

writeLaunchEnv

public void writeLaunchEnv(OutputStream out,
                           Map<String,String> environment,
                           Map<org.apache.hadoop.fs.Path,List<String>> resources,
                           List<String> command)
                    throws IOException

Overrides:

writeLaunchEnv in class ContainerExecutor

Throws:

IOException

signalContainer

public boolean signalContainer(String user,
                               String pid,
                               ContainerExecutor.Signal signal)
                        throws IOException

Specified by:

signalContainer in class ContainerExecutor

Throws:

IOException

isContainerProcessAlive

public boolean isContainerProcessAlive(String user,
                                       String pid)
                                throws IOException

Specified by:

isContainerProcessAlive in class ContainerExecutor

Throws:

IOException

containerIsAlive

public static boolean containerIsAlive(String pid)
                                throws IOException

Returns true if the process with the specified pid is alive.

Parameters:

pid - String pid

Returns:

boolean true if the process is alive

Throws:

IOException

killContainer

protected void killContainer(String pid,
                             ContainerExecutor.Signal signal)
                      throws IOException

Send a specified signal to the specified pid

Parameters:

pid - the pid of the process [group] to signal.

signal - signal to send (for logging).

Throws:

IOException

deleteAsUser

public void deleteAsUser(String user,
                         org.apache.hadoop.fs.Path subDir,
                         org.apache.hadoop.fs.Path... baseDirs)
                  throws IOException,
                         InterruptedException

Specified by:

deleteAsUser in class ContainerExecutor

Throws:

IOException

InterruptedException

createDir

protected void createDir(org.apache.hadoop.fs.Path dirPath,
                         org.apache.hadoop.fs.permission.FsPermission perms,
                         boolean createParent,
                         String user)
                  throws IOException

Throws:

IOException

getWorkingDir

protected org.apache.hadoop.fs.Path getWorkingDir(List<String> localDirs,
                                                  String user,
                                                  String appId)
                                           throws IOException

Throws:

IOException