com.google.cloud.hadoop.gcsio

Class DirectoryListCache

java.lang.Object

com.google.cloud.hadoop.gcsio.DirectoryListCache

Direct Known Subclasses:

FileSystemBackedDirectoryListCache, InMemoryDirectoryListCache

public abstract class DirectoryListCache
extends Object

DirectoryListCache is an abstract class providing supplemental accounting of full paths for directories and files created and deleted in GoogleCloudStorageFileSystem. It serves primarily as a supplemental list index for underlying Storage.list() requests, because the remote listing is updated asynchronously, so that recently-created files may not appear in a subsequent listing, even if the request is made by the same client which created the file.

Depending on the extent to which the underlying data structures accessed by implementations of this class are shared, callers can guarantee varying levels of immediate list consistency between file-creation and a subsequent "list" request.

Implementations of this class must be thread-safe.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DirectoryListCache.Config Container for various cache-configuration parameters used by a DirectoryListCache when managing expiration/retention policies, etc.
static class	DirectoryListCache.Type Known types of DirectoryListCache implementations.

Field Summary

Fields

Modifier and Type	Field and Description
protected DirectoryListCache.Config	cacheConfig
protected com.google.api.client.util.Clock	clock

Constructor Summary

Constructors

Constructor and Description
DirectoryListCache()

Method Summary

Modifier and Type	Method and Description
abstract boolean	containsEntriesForImplicitDirectories() Implementations should indicate whether implicit directories are automatically detected and added as CacheEntry items when putResourceId is called without first adding explicit directory objects for the implied parent directories of the added object.
abstract List<CacheEntry>	getBucketList()
abstract CacheEntry	getCacheEntry(StorageResourceId resourceId) Returns the CacheEntry associated with resourceId, or null if it doesn't exist.
abstract int	getInternalNumBuckets() Gets the internal number of CachedBucket entries, which may not be equal to the size of getBucketList() if there are expired entries.
abstract int	getInternalNumObjects() Gets the internal total count of cached StorageObject entries.
DirectoryListCache.Config	getMutableConfig() Returns the Config instance used by this DirectoryListCache instance to determine expiration ages, etc.
abstract List<CacheEntry>	getObjectList(String bucketName, String objectNamePrefix, String delimiter, Set<String> returnedPrefixes)
abstract List<CacheEntry>	getRawBucketList()
protected boolean	isCacheEntryExpired(CacheEntry entry) Helper for determining whether a CacheEntry is entirely expired and should be removed from the cache.
protected void	maybeInvalidateExpiredInfo(CacheEntry entry) Helper for possibly clearing the GoogleCloudStorageItemInfo stored in entry based on cacheConfig settings.
abstract CacheEntry	putResourceId(StorageResourceId resourceId) Adds the names of the Bucket or StorageObject referenced by resourceId to the cache, with no attached metadata.
abstract void	removeResourceId(StorageResourceId resourceId) Removes CacheEntry associated with resourceId, if it exists.
void	setClock(com.google.api.client.util.Clock clock) Sets the static Clock instance used for calculating expiration times.
abstract boolean	supportsCacheEntryByReference() Implementations should indicate whether or not CacheEntry objects returned from methods are intended to be shared references that are authoritative cache state regarding metadata like cached GoogleCloudStorageItemInfo, etc.
protected static void	validateResourceId(StorageResourceId resourceId) Helper to validate resourceId that may be a Bucket or StorageObject.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

clock

protected com.google.api.client.util.Clock clock

cacheConfig

protected DirectoryListCache.Config cacheConfig

Constructor Detail

DirectoryListCache

public DirectoryListCache()

Method Detail

supportsCacheEntryByReference

public abstract boolean supportsCacheEntryByReference()

Implementations should indicate whether or not CacheEntry objects returned from methods are intended to be shared references that are authoritative cache state regarding metadata like cached GoogleCloudStorageItemInfo, etc. If true, then the implementation implicitly supports caching of GoogleCloudStorageItemInfo, and if false, the implication is that info caching is *not* supported. Likewise, if 'true', CacheEntries are implied to be shared references with their own synchronization, so that multiple entry holders may pass info from one holder to another without re-querying the DirectoryListCache. If 'false', then each CacheEntry logically represents a wrapper around some other source of authoritative information, and thus references should not be assumed to be shared, and cached info should not be expected to persist between CacheEntry instances corresponding to the same StorageResourceId.

containsEntriesForImplicitDirectories

public abstract boolean containsEntriesForImplicitDirectories()

Implementations should indicate whether implicit directories are automatically detected and added as CacheEntry items when putResourceId is called without first adding explicit directory objects for the implied parent directories of the added object.

putResourceId

public abstract CacheEntry putResourceId(StorageResourceId resourceId)
                                  throws IOException

Adds the names of the Bucket or StorageObject referenced by resourceId to the cache, with no attached metadata. If the entry already exists, then nothing is modified. If resourceId is a StorageObject, the parent Bucket name is also added to the cache, if it doesn't already exist. TODO(user): Even if the entry exists, it might be correct to invalidate any existing metadata and force a refresh next time it is fetched.

Returns:

The CacheEntry corresponding to the item added.

Throws:

IOException

getCacheEntry

public abstract CacheEntry getCacheEntry(StorageResourceId resourceId)
                                  throws IOException

Returns the CacheEntry associated with resourceId, or null if it doesn't exist. This returns the real mutable CacheEntry (rather than a copy of the data) so that the caller may efficiently update the info stored in the CacheEntry if necessary.

Throws:

IOException

removeResourceId

public abstract void removeResourceId(StorageResourceId resourceId)
                               throws IOException

Removes CacheEntry associated with resourceId, if it exists. Cached GoogleCloudStorageItemInfo associated with the resourceId is also removed, if it exists. If resourceId denotes a non-empty bucket, then all the cached StorageObject children of that bucket will also be removed from the cache; it is the caller's responsibility to ensure that the bucket should really be removed. Note that normal expiration of a CachedBucket will *not* remove the actual CachedBucket, even though the bucketName will stop appearing in calls to getBucketList().

Throws:

IOException

getBucketList

public abstract List<CacheEntry> getBucketList()
                                        throws IOException

Returns:

List of CacheEntry corresponding to Buckets known by this cache; includes buckets added to the cache automatically when caching a StorageObject contained in the corresponding bucket. Will not return null. Hides/removes expired bucket entries and clears any expired GoogleCloudStorageItemInfo associated with buckets.

Throws:

IOException

getRawBucketList

public abstract List<CacheEntry> getRawBucketList()
                                           throws IOException

Returns:

List of *all* Bucket CacheEntries, including ones that might be expired. Doesn't actively invalidate, set expiration, or delete expired entries. Can be used when the internal bucket list is needed without wanting to cause any mutations in the cache.

Throws:

IOException

getObjectList

public abstract List<CacheEntry> getObjectList(String bucketName,
                                               String objectNamePrefix,
                                               String delimiter,
                                               Set<String> returnedPrefixes)
                                        throws IOException

Parameters:

bucketName - The bucket inside of which to list objects.

objectNamePrefix - The prefix to be used to match object names to return.

delimiter - The character for specifying 'directory' boundaries, or null.

returnedPrefixes - A container to be populated with implied "directory objects" that come from some object that includes the prefix, followed by some string, then followed by the 'delimiter'. This may or may not be a duplicate of one of the actual returned directory objects. For example, if gs://foo/bar/baz.txt exists and we query with parameters ("foo", "ba", "/", {}) then the returnedPrefixes will be populated with the string "bar/" by virtue of existence of the "bar/baz.txt" file. May be null if the caller doesn't desire fetching such returnedPrefixes.

Returns:

List of CacheEntrys for StorageObjects residing in bucket bucketName that match the provided objectNamePrefix and delimiter, or possibly null if no such objects are present. May also return an empty list.

Throws:

IOException

getInternalNumBuckets

public abstract int getInternalNumBuckets()
                                   throws IOException

Gets the internal number of CachedBucket entries, which may not be equal to the size of getBucketList() if there are expired entries. Does not mutate the cache.

Throws:

IOException

getInternalNumObjects

public abstract int getInternalNumObjects()
                                   throws IOException

Gets the internal total count of cached StorageObject entries. Does not mutate the cache.

Throws:

IOException

getMutableConfig

public DirectoryListCache.Config getMutableConfig()

Returns the Config instance used by this DirectoryListCache instance to determine expiration ages, etc. It is the actual mutable Config object, such that modifications in-place made to the returned Config will immediately be reflected in the behavior of the DirectoryListCache.

setClock

public void setClock(com.google.api.client.util.Clock clock)

Sets the static Clock instance used for calculating expiration times.

isCacheEntryExpired

protected boolean isCacheEntryExpired(CacheEntry entry)

Helper for determining whether a CacheEntry is entirely expired and should be removed from the cache.

validateResourceId

protected static void validateResourceId(StorageResourceId resourceId)

Helper to validate resourceId that may be a Bucket or StorageObject.

maybeInvalidateExpiredInfo

protected void maybeInvalidateExpiredInfo(CacheEntry entry)

Helper for possibly clearing the GoogleCloudStorageItemInfo stored in entry based on cacheConfig settings.