Skip to content

Commit

Permalink
updated readme
Browse files Browse the repository at this point in the history
Signed-off-by: Sienna Satterwhite <[email protected]>
  • Loading branch information
siennathesane committed Dec 13, 2024
1 parent c70243d commit 2f4d274
Showing 1 changed file with 2 additions and 3 deletions.
5 changes: 2 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
[![Build & Test](https://github.com/siennathesane/cesiumdb/actions/workflows/build-and-test.yml/badge.svg)](https://github.com/siennathesane/cesiumdb/actions/workflows/build-and-test.yml)
[![codecov](https://codecov.io/gh/siennathesane/cesiumdb/graph/badge.svg?token=D7RBD3OX2U)](https://codecov.io/gh/siennathesane/cesiumdb)

# CesiumDB
Expand Down Expand Up @@ -35,12 +36,10 @@ CesiumDB does let you bring your own hybrid logical clock implementation. This i

## Unsafety: Or... How To Do Dangerous Things Safely

There is a non-trivial amount of `unsafe` code. Most of it is related to the internal filesystem implementation with `mmap` (which cannot be made safe), and some of it is lifetime elusions. Regarding lifetime elusions, this is primarily because I do not want `async` code in the core implementation. I do not believe a low-level db/storage library should require `async`.
There is a non-trivial amount of `unsafe` code. Most of it is related to the internal filesystem implementation with `mmap` (which cannot be made safe).

Internally, the filesystem I built for CesiumDB is a lock-free, thread-safe portable filesystem since one of my use cases is an embedded system that doesn't have a filesystem, only a device driver. LMDB is a huge inspiration for this project, so I wanted to utilize a lot of the same methodologies around `mmap`, but to make it as safe as possible. The nifty part is that Linux doesn't distinguish between a file and a block device for `mmap`, so I can `mmap` a block device and treat it like a file. The perk is that we get native write speeds for the device, we have a bin-packing filesystem that is portable across devices, and if all else fails, we can just `fallocate` a file and use that. The downside is that writing directly to device memory is dangerous and is inherently "unsafe", so a lot of the optimizations are `unsafe` because of this.

I think the `SegmentWriter` is the most painful example of this. It takes an `Arc<RWLock<FRangeHandle<'a>>>` (a file handle) and wraps it in a `ManuallyDrop`. In `SegmentWriter<'a>`, when we're building it, we kick off an unmanaged background thread that is what actually writes blocks to the filesystem implementation. So when `SegmentWriter::write` is called during a `Memtable` flush, it's actually temporarily writing the data to an `Arc<SegQueue<Block>>` (a lock-free queue) that the background thread is reading from. When `drop(SegmentWriter)` happens, the background thread runs until it sees an `AtomicBool` is set to `true`, empty the queue, sets the `CondVar` and returns, the `drop(SegmentWriter)` sees the `CondVar` is set, and finishes. Notice how we're not actually calling `ManuallyDrop::drop` on the `Arc<RWLock<FRangeHandle<'a>>>`? This ensures that it doesn't get dropped by `SegmentWriter`, and since we're waiting for the thread to finish, our transmutation of lifetimes is safe.

There is :sparkles: __EXTENSIVE__ :sparkles: testing around the `unsafe` code, and I am confident in its correctness. My goal is to keep this project at a high degree of code coverage with tests to help continue to ensure said confidence. However, if you find a bug, please submit an issue or PR.

## Contributing
Expand Down

0 comments on commit 2f4d274

Please sign in to comment.