-
Notifications
You must be signed in to change notification settings - Fork 1
Home
TinyDB is a course designed to help you quickly familiarize yourself with the underlying storage engine of TiKV Project.
After completing this course, you will have a better understanding of LSM-tree based KV stores with high throughput, high scalability and high space utilization.
TinyDB is forked from open source TinyKV, a key-value storage system with the Raft consensus algorithm. TinyKV focuses on the storage layer of a distributed database system, which uses badger, a Go library to store keys and values, as its storage engine. In order to get closer to the actual implementation of TiKV, TinyDB plans to replace the original storage engine badger with LevelDB/RocksDB wrapped by Golang.
In this course, you need to implement an existing optimization method on LevelDB/RocksDB. We provide several projects in Main Course, which introduce some classic and generally accepted optimization ideas presented in recent famous paper. Please choose one and implement it.
After completing the implementation, you need to test and evaluate your optimization. We provide go-ycsb, which can be used to evaluate database performance. If you successfully implement a project, you will get better performance in reading or writing or some other dimension. Finally, you need to chart your evaluation results and submit a report and source code. The experts will give you an appropriate score depending on your optimization results and report. Go-ycsb usage and report requirements are introduced in Evaluation & Report.
LevelDB/RocksDB is a storage engine for server workloads on various storage media, with the initial focus on fast storage (especially Flash storage). It is a C++ library to store key-value pairs. It supports both point lookups and range scans, and provides different types of ACID guarantees.
RocksDB borrows significant code from the open source leveldb project and does a lot of performance optimization. It performs better than LevelDB under many real workloads and TiKV uses it as storage engine. However, RocksDB has a higher amount of code and is more difficult to learn. As a beginner's course for KV storage, TinyDB requires you to complete the course with LevelDB. Besides, if you already have the knowledge of LSM-tree or enough coding ability, TinyDB also provide code based on the RocksDB.
TinyDB uses LevelDB as the storage engine by default. If you want to change it to RocksDB, just rename folder kv/storage/standalone_storage to kv/storage/standalone_storage_ldb and folder kv/storage/standalone_storage_rdb to kv/storage/standalone_storage.
-
git: The source code of TinyDB is hosted on GitHub as a git repository. To work with git repository, please installgit. -
go: TinyDB is a Go project. To build TinyDB from source, please installgowith version greater or equal to 1.13. -
leveldb: LevelDB is a storage engine of TinyDB, please installleveldbwith version greater or equal to 1.7. -
rocksdb: RocksDB is also a storage engine of TinyDB, please installrocksdbwith version greater or equal to 5.16.
Clone the source code to your development machine.
git clone https://github.com/QingyangZ/tinydbBuild TinyDB from the source code.
cd tinydb
makeBuild go-ycsb and test TinyDB
cd go-ycsb
make./bin/go-ycsb load tinydb -P workloads/workloada./bin/go-ycsb run tinydb -P workloads/workloada