org.apache.flink.runtime.operators.hash

Class CompactingHashTable<T>

java.lang.Object

org.apache.flink.runtime.operators.hash.AbstractMutableHashTable<T>

org.apache.flink.runtime.operators.hash.CompactingHashTable<T>

Type Parameters:

T - Record type stored in hash table

public class CompactingHashTable<T>
extends AbstractMutableHashTable<T>

A hash table that uses Flink's managed memory and supports replacement of records or updates to records. For an overview of the general data structure of the hash table, please refer to the description of the MutableHashTable.

The hash table is internally divided into two parts: The hash index, and the partition buffers that store the actual records. When records are inserted or updated, the hash table appends the records to its corresponding partition, and inserts or updates the entry in the hash index. In the case that the hash table runs out of memory, it compacts a partition by walking through the hash index and copying all reachable elements into a fresh partition. After that, it releases the memory of the partition to compact.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	CompactingHashTable.EntryIterator Iterator that traverses the whole hash table once If entries are inserted during iteration they may be overlooked by the iterator
class	CompactingHashTable.HashTableProber<PT>

Field Summary

Fields inherited from class org.apache.flink.runtime.operators.hash.AbstractMutableHashTable

buildSideComparator, buildSideSerializer, closed, stateLock

Constructor Summary

Constructors

Constructor and Description
CompactingHashTable(org.apache.flink.api.common.typeutils.TypeSerializer<T> buildSideSerializer, org.apache.flink.api.common.typeutils.TypeComparator<T> buildSideComparator, List<org.apache.flink.core.memory.MemorySegment> memorySegments)
CompactingHashTable(org.apache.flink.api.common.typeutils.TypeSerializer<T> buildSideSerializer, org.apache.flink.api.common.typeutils.TypeComparator<T> buildSideComparator, List<org.apache.flink.core.memory.MemorySegment> memorySegments, int avgRecordLen)

Method Summary

Modifier and Type	Method and Description
void	abort()
void	buildTableWithUniqueKey(org.apache.flink.util.MutableObjectIterator<T> input)
void	close() Closes the hash table.
org.apache.flink.util.MutableObjectIterator<T>	getEntryIterator()
List<org.apache.flink.core.memory.MemorySegment>	getFreeMemory()
<PT> CompactingHashTable.HashTableProber<PT>	getProber(org.apache.flink.api.common.typeutils.TypeComparator<PT> probeSideComparator, org.apache.flink.api.common.typeutils.TypePairComparator<PT,T> pairComparator)
void	insert(T record)
void	insertOrReplaceRecord(T record) Replaces record in hash table if record already present or append record if not.
void	open() Initialize the hash table

Methods inherited from class org.apache.flink.runtime.operators.hash.AbstractMutableHashTable

getBuildSideComparator, getBuildSideSerializer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CompactingHashTable

public CompactingHashTable(org.apache.flink.api.common.typeutils.TypeSerializer<T> buildSideSerializer,
                           org.apache.flink.api.common.typeutils.TypeComparator<T> buildSideComparator,
                           List<org.apache.flink.core.memory.MemorySegment> memorySegments)

CompactingHashTable

public CompactingHashTable(org.apache.flink.api.common.typeutils.TypeSerializer<T> buildSideSerializer,
                           org.apache.flink.api.common.typeutils.TypeComparator<T> buildSideComparator,
                           List<org.apache.flink.core.memory.MemorySegment> memorySegments,
                           int avgRecordLen)

Method Detail

open

public void open()

Initialize the hash table

Specified by:

open in class AbstractMutableHashTable<T>

close

public void close()

Closes the hash table. This effectively releases all internal structures and closes all open files and removes them. The call to this method is valid both as a cleanup after the complete inputs were properly processed, and as an cancellation call, which cleans up all resources that are currently held by the hash join. If another process still access the hash table after close has been called no operations will be performed.

Specified by:

close in class AbstractMutableHashTable<T>

abort

public void abort()

Specified by:

abort in class AbstractMutableHashTable<T>

getFreeMemory

public List<org.apache.flink.core.memory.MemorySegment> getFreeMemory()

Specified by:

getFreeMemory in class AbstractMutableHashTable<T>

buildTableWithUniqueKey

public void buildTableWithUniqueKey(org.apache.flink.util.MutableObjectIterator<T> input)
                             throws IOException

Throws:

IOException

insert

public final void insert(T record)
                  throws IOException

Specified by:

insert in class AbstractMutableHashTable<T>

Throws:

IOException

insertOrReplaceRecord

public void insertOrReplaceRecord(T record)
                           throws IOException

Replaces record in hash table if record already present or append record if not. May trigger expensive compaction.

Specified by:

insertOrReplaceRecord in class AbstractMutableHashTable<T>

Parameters:

record - record to insert or replace

Throws:

IOException

getProber

public <PT> CompactingHashTable.HashTableProber<PT> getProber(org.apache.flink.api.common.typeutils.TypeComparator<PT> probeSideComparator,
                                                              org.apache.flink.api.common.typeutils.TypePairComparator<PT,T> pairComparator)

Specified by:

getProber in class AbstractMutableHashTable<T>

getEntryIterator

public org.apache.flink.util.MutableObjectIterator<T> getEntryIterator()

Specified by:

getEntryIterator in class AbstractMutableHashTable<T>

Returns:

Iterator over hash table

See Also:

CompactingHashTable.EntryIterator