org.apache.hadoop.yarn.server.nodemanager.recovery

Class NMLeveldbStateStoreService

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService

org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service

public class NMLeveldbStateStoreService
extends NMStateStoreService

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService

NMStateStoreService.LocalResourceTrackerState, NMStateStoreService.RecoveredApplicationsState, NMStateStoreService.RecoveredContainerState, NMStateStoreService.RecoveredContainerStatus, NMStateStoreService.RecoveredContainerTokensState, NMStateStoreService.RecoveredDeletionServiceState, NMStateStoreService.RecoveredLocalizationState, NMStateStoreService.RecoveredLogDeleterState, NMStateStoreService.RecoveredNMTokensState, NMStateStoreService.RecoveredUserResources

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.commons.logging.Log	LOG

Constructor Summary

Constructors

Constructor and Description
NMLeveldbStateStoreService()

Method Summary

Modifier and Type	Method and Description
protected void	checkVersion() 1) Versioning scheme: major.minor.
protected void	closeStorage()
void	finishResourceLocalization(String user, org.apache.hadoop.yarn.api.records.ApplicationId appId, YarnServerNodemanagerRecoveryProtos.LocalizedResourceProto proto) Record the completion of a resource localization
protected void	initStorage(org.apache.hadoop.conf.Configuration conf)
boolean	isNewlyCreated()
NMStateStoreService.RecoveredApplicationsState	loadApplicationsState() Load the state of applications
List<NMStateStoreService.RecoveredContainerState>	loadContainersState() Load the state of containers
NMStateStoreService.RecoveredContainerTokensState	loadContainerTokensState() Load the state of container tokens
NMStateStoreService.RecoveredDeletionServiceState	loadDeletionServiceState() Load the state of the deletion service
NMStateStoreService.RecoveredLocalizationState	loadLocalizationState() Load the state of localized resources
NMStateStoreService.RecoveredLogDeleterState	loadLogDeleterState() Load the state of log deleters
NMStateStoreService.RecoveredNMTokensState	loadNMTokensState() Load the state of NM tokens
protected org.iq80.leveldb.DB	openDatabase(org.apache.hadoop.conf.Configuration conf)
void	removeApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId) Remove records corresponding to an application
void	removeContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) Remove records corresponding to a container
void	removeContainerToken(org.apache.hadoop.yarn.api.records.ContainerId containerId) Remove records for a container token
void	removeDeletionTask(int taskId) Remove records corresponding to a deletion task
void	removeLocalizedResource(String user, org.apache.hadoop.yarn.api.records.ApplicationId appId, org.apache.hadoop.fs.Path localPath) Remove records related to a resource localization
void	removeLogDeleter(org.apache.hadoop.yarn.api.records.ApplicationId appId) Remove the state of a log deleter
void	removeNMTokenApplicationMasterKey(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attempt) Remove a master key corresponding to an application
void	startResourceLocalization(String user, org.apache.hadoop.yarn.api.records.ApplicationId appId, org.apache.hadoop.yarn.proto.YarnProtos.LocalResourceProto proto, org.apache.hadoop.fs.Path localPath) Record the start of localization for a resource
protected void	startStorage()
void	storeApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId, YarnServerNodemanagerRecoveryProtos.ContainerManagerApplicationProto p) Record the start of an application
void	storeContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.protocolrecords.StartContainerRequest startRequest) Record a container start request
void	storeContainerCompleted(org.apache.hadoop.yarn.api.records.ContainerId containerId, int exitCode) Record that a container has completed
void	storeContainerDiagnostics(org.apache.hadoop.yarn.api.records.ContainerId containerId, StringBuilder diagnostics) Record diagnostics for a container
void	storeContainerKilled(org.apache.hadoop.yarn.api.records.ContainerId containerId) Record a request to kill a container
void	storeContainerLaunched(org.apache.hadoop.yarn.api.records.ContainerId containerId) Record that a container has been launched
void	storeContainerToken(org.apache.hadoop.yarn.api.records.ContainerId containerId, Long expTime) Record the expiration time for a container token
void	storeContainerTokenCurrentMasterKey(org.apache.hadoop.yarn.server.api.records.MasterKey key) Record the current container token master key
void	storeContainerTokenPreviousMasterKey(org.apache.hadoop.yarn.server.api.records.MasterKey key) Record the previous container token master key
void	storeDeletionTask(int taskId, YarnServerNodemanagerRecoveryProtos.DeletionServiceDeleteTaskProto taskProto) Record a deletion task
void	storeLogDeleter(org.apache.hadoop.yarn.api.records.ApplicationId appId, YarnServerNodemanagerRecoveryProtos.LogDeleterProto proto) Store the state of a log deleter
void	storeNMTokenApplicationMasterKey(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attempt, org.apache.hadoop.yarn.server.api.records.MasterKey key) Record a master key corresponding to an application
void	storeNMTokenCurrentMasterKey(org.apache.hadoop.yarn.server.api.records.MasterKey key) Record the current NM token master key
void	storeNMTokenPreviousMasterKey(org.apache.hadoop.yarn.server.api.records.MasterKey key) Record the previous NM token master key

Methods inherited from class org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService

canRecover, serviceInit, serviceStart, serviceStop

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

Constructor Detail

NMLeveldbStateStoreService

public NMLeveldbStateStoreService()

Method Detail

startStorage

protected void startStorage()
                     throws IOException

Specified by:

startStorage in class NMStateStoreService

Throws:

IOException

closeStorage

protected void closeStorage()
                     throws IOException

Specified by:

closeStorage in class NMStateStoreService

Throws:

IOException

isNewlyCreated

public boolean isNewlyCreated()

Overrides:

isNewlyCreated in class NMStateStoreService

loadContainersState

public List<NMStateStoreService.RecoveredContainerState> loadContainersState()
                                                                      throws IOException

Description copied from class: NMStateStoreService

Load the state of containers

Specified by:

loadContainersState in class NMStateStoreService

Returns:

recovered state for containers

Throws:

IOException

storeContainer

public void storeContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                           org.apache.hadoop.yarn.api.protocolrecords.StartContainerRequest startRequest)
                    throws IOException

Description copied from class: NMStateStoreService

Record a container start request

Specified by:

storeContainer in class NMStateStoreService

Parameters:

containerId - the container ID

startRequest - the container start request

Throws:

IOException

storeContainerDiagnostics

public void storeContainerDiagnostics(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                      StringBuilder diagnostics)
                               throws IOException

Description copied from class: NMStateStoreService

Record diagnostics for a container

Specified by:

storeContainerDiagnostics in class NMStateStoreService

Parameters:

containerId - the container ID

diagnostics - the container diagnostics

Throws:

IOException

storeContainerLaunched

public void storeContainerLaunched(org.apache.hadoop.yarn.api.records.ContainerId containerId)
                            throws IOException

Description copied from class: NMStateStoreService

Record that a container has been launched

Specified by:

storeContainerLaunched in class NMStateStoreService

Parameters:

containerId - the container ID

Throws:

IOException

storeContainerKilled

public void storeContainerKilled(org.apache.hadoop.yarn.api.records.ContainerId containerId)
                          throws IOException

Description copied from class: NMStateStoreService

Record a request to kill a container

Specified by:

storeContainerKilled in class NMStateStoreService

Parameters:

containerId - the container ID

Throws:

IOException

storeContainerCompleted

public void storeContainerCompleted(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                    int exitCode)
                             throws IOException

Description copied from class: NMStateStoreService

Record that a container has completed

Specified by:

storeContainerCompleted in class NMStateStoreService

Parameters:

containerId - the container ID

exitCode - the exit code from the container

Throws:

IOException

removeContainer

public void removeContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
                     throws IOException

Description copied from class: NMStateStoreService

Remove records corresponding to a container

Specified by:

removeContainer in class NMStateStoreService

Parameters:

containerId - the container ID

Throws:

IOException

loadApplicationsState

public NMStateStoreService.RecoveredApplicationsState loadApplicationsState()
                                                                     throws IOException

Description copied from class: NMStateStoreService

Load the state of applications

Specified by:

loadApplicationsState in class NMStateStoreService

Returns:

recovered state for applications

Throws:

IOException

storeApplication

public void storeApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId,
                             YarnServerNodemanagerRecoveryProtos.ContainerManagerApplicationProto p)
                      throws IOException

Description copied from class: NMStateStoreService

Record the start of an application

Specified by:

storeApplication in class NMStateStoreService

Parameters:

appId - the application ID

p - state to store for the application

Throws:

IOException

removeApplication

public void removeApplication(org.apache.hadoop.yarn.api.records.ApplicationId appId)
                       throws IOException

Description copied from class: NMStateStoreService

Remove records corresponding to an application

Specified by:

removeApplication in class NMStateStoreService

Parameters:

appId - the application ID

Throws:

IOException

loadLocalizationState

public NMStateStoreService.RecoveredLocalizationState loadLocalizationState()
                                                                     throws IOException

Description copied from class: NMStateStoreService

Load the state of localized resources

Specified by:

loadLocalizationState in class NMStateStoreService

Returns:

recovered localized resource state

Throws:

IOException

startResourceLocalization

public void startResourceLocalization(String user,
                                      org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                      org.apache.hadoop.yarn.proto.YarnProtos.LocalResourceProto proto,
                                      org.apache.hadoop.fs.Path localPath)
                               throws IOException

Description copied from class: NMStateStoreService

Record the start of localization for a resource

Specified by:

startResourceLocalization in class NMStateStoreService

Parameters:

user - the username or null if the resource is public

appId - the application ID if the resource is app-specific or null

proto - the resource request

localPath - local filesystem path where the resource will be stored

Throws:

IOException

finishResourceLocalization

public void finishResourceLocalization(String user,
                                       org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                       YarnServerNodemanagerRecoveryProtos.LocalizedResourceProto proto)
                                throws IOException

Description copied from class: NMStateStoreService

Record the completion of a resource localization

Specified by:

finishResourceLocalization in class NMStateStoreService

Parameters:

user - the username or null if the resource is public

appId - the application ID if the resource is app-specific or null

proto - the serialized localized resource

Throws:

IOException

removeLocalizedResource

public void removeLocalizedResource(String user,
                                    org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                    org.apache.hadoop.fs.Path localPath)
                             throws IOException

Description copied from class: NMStateStoreService

Remove records related to a resource localization

Specified by:

removeLocalizedResource in class NMStateStoreService

Parameters:

user - the username or null if the resource is public

appId - the application ID if the resource is app-specific or null

localPath - local filesystem path where the resource will be stored

Throws:

IOException

loadDeletionServiceState

public NMStateStoreService.RecoveredDeletionServiceState loadDeletionServiceState()
                                                                           throws IOException

Description copied from class: NMStateStoreService

Load the state of the deletion service

Specified by:

loadDeletionServiceState in class NMStateStoreService

Returns:

recovered deletion service state

Throws:

IOException

storeDeletionTask

public void storeDeletionTask(int taskId,
                              YarnServerNodemanagerRecoveryProtos.DeletionServiceDeleteTaskProto taskProto)
                       throws IOException

Description copied from class: NMStateStoreService

Record a deletion task

Specified by:

storeDeletionTask in class NMStateStoreService

Parameters:

taskId - the deletion task ID

taskProto - the deletion task protobuf

Throws:

IOException

removeDeletionTask

public void removeDeletionTask(int taskId)
                        throws IOException

Description copied from class: NMStateStoreService

Remove records corresponding to a deletion task

Specified by:

removeDeletionTask in class NMStateStoreService

Parameters:

taskId - the deletion task ID

Throws:

IOException

loadNMTokensState

public NMStateStoreService.RecoveredNMTokensState loadNMTokensState()
                                                             throws IOException

Description copied from class: NMStateStoreService

Load the state of NM tokens

Specified by:

loadNMTokensState in class NMStateStoreService

Returns:

recovered state of NM tokens

Throws:

IOException

storeNMTokenCurrentMasterKey

public void storeNMTokenCurrentMasterKey(org.apache.hadoop.yarn.server.api.records.MasterKey key)
                                  throws IOException

Description copied from class: NMStateStoreService

Record the current NM token master key

Specified by:

storeNMTokenCurrentMasterKey in class NMStateStoreService

Parameters:

key - the master key

Throws:

IOException

storeNMTokenPreviousMasterKey

public void storeNMTokenPreviousMasterKey(org.apache.hadoop.yarn.server.api.records.MasterKey key)
                                   throws IOException

Description copied from class: NMStateStoreService

Record the previous NM token master key

Specified by:

storeNMTokenPreviousMasterKey in class NMStateStoreService

Parameters:

key - the previous master key

Throws:

IOException

storeNMTokenApplicationMasterKey

public void storeNMTokenApplicationMasterKey(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attempt,
                                             org.apache.hadoop.yarn.server.api.records.MasterKey key)
                                      throws IOException

Description copied from class: NMStateStoreService

Record a master key corresponding to an application

Specified by:

storeNMTokenApplicationMasterKey in class NMStateStoreService

Parameters:

attempt - the application attempt ID

key - the master key

Throws:

IOException

removeNMTokenApplicationMasterKey

public void removeNMTokenApplicationMasterKey(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attempt)
                                       throws IOException

Description copied from class: NMStateStoreService

Remove a master key corresponding to an application

Specified by:

removeNMTokenApplicationMasterKey in class NMStateStoreService

Parameters:

attempt - the application attempt ID

Throws:

IOException

loadContainerTokensState

public NMStateStoreService.RecoveredContainerTokensState loadContainerTokensState()
                                                                           throws IOException

Description copied from class: NMStateStoreService

Load the state of container tokens

Specified by:

loadContainerTokensState in class NMStateStoreService

Returns:

recovered state of container tokens

Throws:

IOException

storeContainerTokenCurrentMasterKey

public void storeContainerTokenCurrentMasterKey(org.apache.hadoop.yarn.server.api.records.MasterKey key)
                                         throws IOException

Description copied from class: NMStateStoreService

Record the current container token master key

Specified by:

storeContainerTokenCurrentMasterKey in class NMStateStoreService

Parameters:

key - the master key

Throws:

IOException

storeContainerTokenPreviousMasterKey

public void storeContainerTokenPreviousMasterKey(org.apache.hadoop.yarn.server.api.records.MasterKey key)
                                          throws IOException

Description copied from class: NMStateStoreService

Record the previous container token master key

Specified by:

storeContainerTokenPreviousMasterKey in class NMStateStoreService

Parameters:

key - the previous master key

Throws:

IOException

storeContainerToken

public void storeContainerToken(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                Long expTime)
                         throws IOException

Description copied from class: NMStateStoreService

Record the expiration time for a container token

Specified by:

storeContainerToken in class NMStateStoreService

Parameters:

containerId - the container ID

expTime - the container token expiration time

Throws:

IOException

removeContainerToken

public void removeContainerToken(org.apache.hadoop.yarn.api.records.ContainerId containerId)
                          throws IOException

Description copied from class: NMStateStoreService

Remove records for a container token

Specified by:

removeContainerToken in class NMStateStoreService

Parameters:

containerId - the container ID

Throws:

IOException

loadLogDeleterState

public NMStateStoreService.RecoveredLogDeleterState loadLogDeleterState()
                                                                 throws IOException

Description copied from class: NMStateStoreService

Load the state of log deleters

Specified by:

loadLogDeleterState in class NMStateStoreService

Returns:

recovered log deleter state

Throws:

IOException

storeLogDeleter

public void storeLogDeleter(org.apache.hadoop.yarn.api.records.ApplicationId appId,
                            YarnServerNodemanagerRecoveryProtos.LogDeleterProto proto)
                     throws IOException

Description copied from class: NMStateStoreService

Store the state of a log deleter

Specified by:

storeLogDeleter in class NMStateStoreService

Parameters:

appId - the application ID for the log deleter

proto - the serialized state of the log deleter

Throws:

IOException

removeLogDeleter

public void removeLogDeleter(org.apache.hadoop.yarn.api.records.ApplicationId appId)
                      throws IOException

Description copied from class: NMStateStoreService

Remove the state of a log deleter

Specified by:

removeLogDeleter in class NMStateStoreService

Parameters:

appId - the application ID for the log deleter

Throws:

IOException

initStorage

protected void initStorage(org.apache.hadoop.conf.Configuration conf)
                    throws IOException

Specified by:

initStorage in class NMStateStoreService

Throws:

IOException

openDatabase

protected org.iq80.leveldb.DB openDatabase(org.apache.hadoop.conf.Configuration conf)
                                    throws IOException

Throws:

IOException

checkVersion

protected void checkVersion()
                     throws IOException

1) Versioning scheme: major.minor. For e.g. 1.0, 1.1, 1.2...1.25, 2.0 etc. 2) Any incompatible change of state-store is a major upgrade, and any compatible change of state-store is a minor upgrade. 3) Within a minor upgrade, say 1.1 to 1.2: overwrite the version info and proceed as normal. 4) Within a major upgrade, say 1.2 to 2.0: throw exception and indicate user to use a separate upgrade tool to upgrade NM state or remove incompatible old state.

Throws:

IOException