The Berkeley DB Java Edition Package: BDB JE Library Version 12.1.6.0 (Release 6.2.7) Change Log

Oracle Berkeley DB Java Edition 12c R1 Change Log

Release 6.2.7

Upgrading from JE 6.1 or earlier

In JE 6.2 the on-disk file format moved to 10. The file format change is forward compatible in that JE files created with release 6.1 and earlier can be read when opened with JE 6.2 or later. The change is not backward compatible in that files created with JE 6.2 or later cannot be read by earlier releases. After an existing environment is opened read/write using JE 6.2, the environment can no longer be read by earlier releases.

Upgrading from JE 6.0 or earlier

An API change in JE 6.1.3 [#23330] requires application changes if write operations are performed on a non-replicated database in a replicated environment. A code change is necessary for applications with the following characteristics:

A ReplicatedEnvironment is used.
A non-replicated, transactional Database is accessed (DatabaseConfig.setReplicated(false) and setTransactional(true) are called) in this environment.
When writing to this database, an explicit (non-null) Transaction is specified.

In order to perform write operations in such cases, the application must now call TransactionConfig.setLocalWrite(true) and use this configuration to create a Transaction for performing writes to the non-replicated database.

In addition, it is no longer possible to use a single transaction to write to both replicated and a non-replicated databases. IllegalOperationException will be thrown if this is attempted.

These changes were necessary to prevent corruption when a transaction contains write operations for both replicated and non-replicated databases, and a failover occurs that causes a rollback of this transaction. The probability of corruption is low, but it can occur under the right conditions.

For more information see the javadoc for TransactionConfig.setLocalWrite(true), and the "Non-replicated Databases in a Replicated Environment" section of the ReplicatedEnvironment class javadoc.

Upgrading from JE 5.0 or earlier

In addition to the file format changes, a change was made involving partial Btree and duplicate comparators. Partial comparators are an advanced feature that few applications use. As of JE 6.0, using partial comparators is not recommended. Applications that do use partial comparators must change their comparator classes to implement the new PartialComparator tag interface, before running the application with JE 6. Failure to do so may cause incorrect behavior during transaction aborts. See the PartialComparator javadoc for more information.

Upgrading from JE 4.1 or earlier

There are two important notes about the file format change in JE 5.0.

The file format change enabled significant improvements in operation performance, memory and disk footprint, and concurrency of databases with duplicate keys. Due to these changes, an upgrade utility must be run before opening an environment with this release, if the environment was created using JE 4.1 or earlier. See the Upgrade Procedure below for more information.
An application which uses JE replication may not upgrade directly from JE 4.0 to JE 5.0 or later. Instead, the upgrade must be done from JE 4.0 to JE 4.1 and then to JE 5.0 or later. Applications already at JE 4.1 are not affected. Upgrade guidance can be found in the new chapter, "Upgrading a JE Replication Group", in the "Getting Started with BDB JE High Availability" guide.

Due to the format changes in JE 5, a special utility program must be run for an environment created with JE 4.1 or earlier, prior to opening the environment with JE 5.0 or later. The utility program is part of JE 4.1. JE 4.1.20, or a later version of JE 4.1, must be used.

One of two utility programs must be used, which are available in the release package for JE 4.1.20, or a later release of JE 4.1. If you are currently running a release earlier than JE 4.1.20, then you must download the latest JE 4.1 release package in order to run these utilities.

The steps for upgrading are as follows.

Stop the application using BDB JE.

Run the DbPreUpgrade_4_1 or DbRepPreUpgrade_4_1 utility. If you are using a regular non-replicated Environment:

    java -jar je-4.1.20.jar DbPreUpgrade_4_1 -h <dir>

If you are using a JE ReplicatedEnvironment:

    java -jar je-4.1.20.jar DbRepPreUpgrade_4_1
         -h <dir>
         -groupName <group name>
         -nodeName <node name>
         -nodeHostPort <host:port>

Finally, start the application using the current JE 5.0 (or later) release of BDB JE.

The second step -- running the utility program -- does not perform data conversion. This step simply performs a special checkpoint to prepare the environment for upgrade. It should take no longer than an ordinary startup and shutdown.

During the last step -- when the application opens the JE environment using the current release (JE 5 or later) -- all databases configured for duplicates will automatically be converted before the Environment or ReplicatedEnvironment constructor returns. Note that a database might be explicitly configured for duplicates using DatabaseConfig.setSortedDuplicates(true), or implicitly configured for duplicates by using a DPL MANY_TO_XXX relationship (Relationship.MANY_TO_ONE or Relationship.MANY_TO_MANY).

The duplicate database conversion only rewrites internal nodes in the Btree, not leaf nodes. In a test with a 500 MB cache, conversion of a 10 million record data set (8 byte key and data) took between 1.5 and 6.5 minutes, depending on number of duplicates per key. The high end of this range is when 10 duplicates per key were used; the low end is with 1 million duplicates per key.

To make the duplicate database conversion predictable during deployment, users should measure the conversion time on a non-production system before upgrading a deployed system. When duplicates are converted, the Btree internal nodes are preloaded into the JE cache. A new configuration option, EnvironmentConfig.ENV_DUP_CONVERT_PRELOAD_ALL, can be set to false to optimize this process if the cache is not large enough to hold the internal nodes for all databases. For more information, see the javadoc for this property.

If an application has no databases configured for duplicates, then the last step simply opens the JE environment normally, and no data conversion is performed.

If the user fails to run the DbPreUpgrade_4_1 or DbRepPreUpgrade_4_1 utility program before opening an environment with JE 5 or later for the first time, an exception such as the following will normally be thrown by the Environment or ReplicatedEnvironment constructor:

  com.sleepycat.je.EnvironmentFailureException: (JE 6.0.1) JE 4.1 duplicate DB
  entries were found in the recovery interval. Before upgrading to JE 5.0, the
  following utility must be run using JE 4.1 (4.1.20 or later):
  DbPreUpgrade_4_1.  See the change log.
  UNEXPECTED_STATE: Unexpected internal state, may have side effects.
    at com.sleepycat.je.EnvironmentFailureException.unexpectedState(EnvironmentFailureException.java:376)
    at com.sleepycat.je.recovery.RecoveryManager.checkLogVersion8UpgradeViolations(RecoveryManager.java:2694)
    at com.sleepycat.je.recovery.RecoveryManager.buildTree(RecoveryManager.java:549)
    at com.sleepycat.je.recovery.RecoveryManager.recover(RecoveryManager.java:198)
    at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:610)
    ...

If the user fails to run the DbPreUpgrade_4_1 or DbRepPreUpgrade_4_1 utility program, but no exception is thrown when the environment is opened with JE 5 or later, this is probably because the application performed an Environment.sync before last closing the environment with JE 4.1 or earlier, and nothing else happened to be written (by the application or JE background threads) after the sync operation. In this case, running the upgrade utility is not necessary.

Changes in 6.2.7

A cursor may now be optionally configured to be "non-sticky". This has certain performance advantages:
- Some processing is avoided because the prior position is not maintained.
- The lock on the record at the prior position is released before acquiring the lock on the record at the new position. This can help to prevent deadlocks in certain situations.
For more information, see the javadoc for CursorConfig.setNonSticky.
[#23775] (6.2.0)

Further exploitation of BIN-deltas for CRUD operations.
For backgroud and previous work in this area, see the changelog for the 6.1 release. In this release we have extended the set of CRUD operations that are performed in BIN-deltas, without the need to mutate them to full BINs (and thus saving the disk reads that would be required to fetch the full BINs in memory). Specifically, the following additional operations can now exploit BIN-deltas:
Insertions and updates, when no tree node splits are required and the key of the record to be inserted/updated is found in a BIN-delta.
Blind operations: we say that a record operation (insertion, update, or deletion) is performed "blindly" in a BIN-delta, when the delta does not contain a slot with the operation's key and we don't need to access the full BIN to check whether such a slot exists there or to extract any information from the full-BIN slot, if it exists. The condition that no tree node splits are required applies to blind operations as well. The following operations can be performed blindly: - Replay of insertions at replica nodes. - Insertions during recovery redo. - Updates and deletes during recovery redo, for databases with duplicates.
A new statistic has been added to count the number blind operations performed, including the blind put operations described below. This count can be obtained via the EnvironmentStats.getNBINDeltaBlindOps() method.
[#23680] (6.2.0)

Blind put operations in BIN-deltas.
Normally, blind puts are not possible: we need to know whether the put is actually an update or an insertion, i.e., whether the key exists in the full BIN or not. Furthermore, in case of update we also need to know the location of the previous record version to make the current update abortable. However, it is possible to answer at least the key existence question by adding a small amount of extra information in the deltas. If we do so, puts that are actual insertions can be done blindly.
To answer whether a key exists in a full BIN or not, each BIN-delta stores a bloom filter, which is a very compact, approximate representation of the set of keys in the full BIN. Bloom filters can answer set membership questions with no false negatives and very low probability of false positives. As a result, put operations that are actual insertions can almost always be performed blindly.
To make possible the blind puts optimization in JE databases that use custom BTree and/or duplicates comparators, these comparators must perform "binary equality", that is, they must consider two keys (byte arrays) to be equal if and only if they have the same length and they are equal byte-per-byte. To communicate to the JE engine that a comparator does binary equality, the comparator must implement the new BinaryEqualityComparator tag interface.
[#23768] (6.2.1)

Added LockMode.READ_UNCOMMITTED_ALL. When using this mode, unlike READ_UNCOMMITTED, deleted records will not be skipped by read operations when the deleting transaction is still open (and may later abort, in which case the record will no longer be deleted). See the LockMode javadoc for further details.
[#23660] (6.2.1)

Added two optimizations for secondary DB read operations.
- For secondary DB read operations where the primary record data is not requested (because DatabaseEntry.setPartial is called on the 'data' parameter), a Btree lookup and record lock of the primary record are no longer performed. This change does not impact the meaning of the isolation mode used for such secondary reads, i.e., the semantics are correct without acquiring a lock on the primary record.
- For secondary DB read operations where the primary record data is requested, one less record lock is now acquired. Previously, both the primary and secondary records were locked. Now, only the primary record is locked. This optimization does not apply to the serializable isolation mode. The optimization applies only to the read-committed and repeatable-read isolation modes, and does not impact the meaning of these modes, i.e., the semantics are correct without acquiring a lock on the secondary record.
[#23326] (6.2.2)

Fixed a bug that could cause the following Collections API and DPL methods to incorrectly return an empty result (no records).
- When using the DPL (com.sleepycat.persist) and calling EntityCursor.last() when the cursor was created with an end point (toKey parameter). Also when EntityCursor.prev() or prevNoDup() is called and the cursor is not initialized, since this is equivalent to calling last().
- When using the Collections API (com.sleepycat.collections) and calling SortedSet.last() or SortedMap.lastKey().
The problem occurs when searching for the last record in a key range, while another thread is concurrently inserting records near the end of the key range. An empty result is returned, regardless of the number of records in the key range.
[#23687] (6.2.2)

Fixed bugs in the computation of the nINCompactKey and nINNoTarget stats (EnvironmentStats.getNINCompactKeyIN and getNINNoTarget). Prior to the fixes, these stats would sometimes have negative values. [#23718] (6.2.3)

Fixed a bug that caused a DB to become unusable when it is removed or truncated (by calling Environment.removeDatabase or truncateDatabase) using a read-committed transaction, and the transaction aborts (explicitly, or due to a crash before commit). In this case the DB will not be accessible -- it cannot be opened, truncated or removed. When attempting to open the DB, an exception such as the following is thrown:

Exception in thread "main" com.sleepycat.je.DatabaseNotFoundException: (JE 6.1.5) Attempted to remove non-existent database ...
at com.sleepycat.je.dbi.DbTree.lockNameLN(DbTree.java:869)
at com.sleepycat.je.dbi.DbTree.doRemoveDb(DbTree.java:1130)
at com.sleepycat.je.dbi.DbTree.dbRemove(DbTree.java:1183)
at com.sleepycat.je.Environment$1.runWork(Environment.java:947)
at com.sleepycat.je.Environment$DbNameOperation.runOnce(Environment.java:1172)
at com.sleepycat.je.Environment$DbNameOperation.run(Environment.java:1155)
at com.sleepycat.je.Environment.removeDatabase(Environment.java:941)
...

A workaround for the problem in earlier releases is to avoid using read-committed for a transaction used to perform a DB remove or truncate operation.

[#23821] (6.2.3)

Fixed a bug that caused an exception during log cleaning, although it has been observed very rarely. It could also potentially cause data corruption, but this has never been reported or observed in tests. Examples of the exceptions that have been observed are below.

com.sleepycat.je.EnvironmentFailureException: Environment invalid because of
previous exception: (JE 6.1.0) ...
    at com.sleepycat.je.EnvironmentFailureException.unexpectedException(EnvironmentFailureException.java:315)
    at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:477)
    at com.sleepycat.je.log.LogManager.logItems(LogManager.java:419)
    at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:324)
    at com.sleepycat.je.log.LogManager.log(LogManager.java:272)
    at com.sleepycat.je.log.LogManager.log(LogManager.java:261)
    at com.sleepycat.je.log.LogManager.log(LogManager.java:223)
    at com.sleepycat.je.dbi.EnvironmentImpl.rewriteMapTreeRoot(EnvironmentImpl.java:1285)
    at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:701)
    at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:274)
    at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:137)
    at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:148)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 111
    at com.sleepycat.util.PackedInteger.writeInt(PackedInteger.java:188)
    at com.sleepycat.je.log.LogUtils.writePackedInt(LogUtils.java:155)
    at com.sleepycat.je.cleaner.DbFileSummary.writeToLog(DbFileSummary.java:79)
    at com.sleepycat.je.dbi.DatabaseImpl.writeToLog(DatabaseImpl.java:2410)
    at com.sleepycat.je.dbi.DbTree.writeToLog(DbTree.java:2050)
    at com.sleepycat.je.log.entry.SingleItemEntry.writeEntry(SingleItemEntry.java:114)
    at com.sleepycat.je.log.LogManager.marshallIntoBuffer(LogManager.java:745)
    at com.sleepycat.je.log.LogManager.serialLogWork(LogManager.java:611)
    at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:461)
    ... 11 more

Another instance of the same problem with a slightly different stack trace is below:

java.nio.BufferOverflowException UNEXPECTED_EXCEPTION_FATAL: Unexpected
internal Exception, unable to continue. Environment is invalid and must be
closed.
    at com.sleepycat.je.EnvironmentFailureException.unexpectedException(EnvironmentFailureException.java:315)
    at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:481)
    at com.sleepycat.je.log.LogManager.logItems(LogManager.java:423)
    at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:325)
    at com.sleepycat.je.log.LogManager.log(LogManager.java:273)
    at com.sleepycat.je.tree.LN.logInternal(LN.java:600)
    at com.sleepycat.je.tree.LN.log(LN.java:411)
    at com.sleepycat.je.cleaner.FileProcessor.processFoundLN(FileProcessor.java:1070)
    at com.sleepycat.je.cleaner.FileProcessor.processLN(FileProcessor.java:884)
    at com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:673)
    at com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:278)
    at com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:137)
    at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:148)
Caused by: java.nio.BufferOverflowException
    at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:189)
    at java.nio.ByteBuffer.put(ByteBuffer.java:859)
    at com.sleepycat.je.log.LogUtils.writeBytesNoLength(LogUtils.java:350)
    at com.sleepycat.je.log.entry.LNLogEntry.writeBaseLNEntry(LNLogEntry.java:371)
    at com.sleepycat.je.log.entry.LNLogEntry.writeEntry(LNLogEntry.java:333)
    at com.sleepycat.je.log.entry.BaseReplicableEntry.writeEntry(BaseReplicableEntry.java:48)
    at com.sleepycat.je.log.entry.LNLogEntry.writeEntry(LNLogEntry.java:52)
    at com.sleepycat.je.log.LogManager.marshallIntoBuffer(LogManager.java:751)
    at com.sleepycat.je.log.LogManager.serialLogWork(LogManager.java:617)
    at com.sleepycat.je.log.LogManager.serialLog(LogManager.java:465)

[#23492] (6.2.3)

Fixed a locking bug that causes a deadlock when no real deadlock exists.
The bug shows up with cursors using read-committed isolation. Here is the specific scenario: 1. Cursor C1 in thread T1 reads a record R using Transaction X1. C1 creates a ReadCommittedLocker L1, with X1 as its buddy. L1 locks R. 2. Cursor C2 in thread T2 tries to write-lock R, using another Transaction X2. X2 waits for L1 (T2 waits for T1). 3. Cursor C3 in thread T1 tries to read R using X1. C3 creates a ReadCommittedLocker L3, with X1 as its buddy. L3 tries to lock R. L1 and L3 are not recognized as buddies, so L3 waits for X2 (T1 waits for T2)
[#23821] (6.2.4)

The ant build (build.xml) has been updated so that the JUnit jar file is now downloaded from Maven Central when needed for running tests with the 'test' target. This jar is no longer needed for building a JE jar file with the 'jar' target. See installation.html for an updated description of how to build JE and run the unit tests. [#23669] (6.2.7)