mv cache object warming

Status

• merged to master - January 25, 2016
• code complete - July 30, 2015
• development started - July 24, 2015

History / Context

NOTE: This feature is not fully released to open source community at this time. Portions of the feature are part of the private repository leveldb_ee. See leveldb_ee's wiki for further details.

leveldb's performance is highly dependent upon its caches: file cache, block cache, and operating system block cache. A restart of leveldb flushes at least the file cache and block cache. A server restart flushes all three. An application that depends heavily upon leveldb for fast access is weakened right after a leveldb starts due to two or three of its caches lacking good content.

The file cache is the most performance critical cache. A miss against the file cache requires an .sst table file to be opened, index read, index decompressed, bloom filter read, and bloom filter decompressed. These five steps happens before leveldb attempts to read the requested data. Large datasets can easily open 500 or more .sst table files per second while trying to properly populate the file cache. Each file open is not only slow, but also interferes with other leveldb file operations such as reads for an already open file.

The user application can suffer for minutes during a server restart. With Riak, the performance of an entire server cluster is impacted by the restart of just one server. This makes the problem even more critical since individual servers regularly need software and hardware restarts for software updates and security patches.

This branch adds logic to write the file names of open .sst table files to a temporary file. leveldb then reads the temporary file upon next startup and processes each file into the file cache. This branch does not attempt to preload the block cache or manipulate the operating system block cache.

This branch is enabled / disabled via leveldb::Options structure. It has a new member variable "cache_object_warming". It defaults to true.