-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #19 from mxmlkzdh/readme
README modifications
- Loading branch information
Showing
2 changed files
with
123 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,121 @@ | ||
# SnowMint | ||
A blazingly fast unique and roughly sortable IDs generator, based on Twitter's Snowflake | ||
# SnowMint - A Blazingly Fast Unique ID Generator | ||
[![Go Workflow](https://github.com/mxmlkzdh/snowmint/actions/workflows/go.yml/badge.svg)](https://github.com/mxmlkzdh/snowmint/actions) | ||
|
||
**SnowMint** is a high-performance, distributed unique ID generator based on X's Snowflake algorithm. It provides unique, roughly sortable IDs that are generated using a client/server model with a custom protocol. | ||
|
||
## SnowMint IDs | ||
|
||
### ID Generation Algorithm | ||
SnowMint leverages X’s Snowflake algorithm to generate **signed, non-negative 64-bit unique identifiers called SnowMint IDs**. The format of the ID consists of: | ||
|
||
- **Sign**: A 1-bit field. It will always be 0. | ||
- **Timestamp**: A 43-bit field representing the current time in milliseconds since your organization's epoch. This allows for generating 278 years 11 months 2.049 days worth of unique IDs from the epoch. | ||
- **DataCenter ID**: A 5-bit field identifying the data center where the ID was generated. | ||
- **Node ID**: A 5-bit field identifying the machine where the ID was generated. | ||
- **Sequence**: A 10-bit field for a sequence number that resets every millisecond. | ||
|
||
This combination ensures that SnowMint can (theoretically) generate more than 1,000,000 unique IDs per second, even in distributed environments. | ||
|
||
### An Example | ||
The SnowMint ID `817347092935625747` is generated by a SnowMint server with the following (configurable) parameters: | ||
- Epoch: 946684800000 (Saturday, January 01 2000 00:00:00.00 GMT+0000) | ||
- DataCenter ID: 0 | ||
- Node ID: 19 | ||
|
||
The binary representation of this ID is presented below. Note that for this particular ID, the timestamp is `1726167730122 (Thursday, September 12, 2024 19:02:10.122 GMT+0000)` and the sequence number is equal to `19`. | ||
|
||
``` | ||
0 0001011010101111100110011011001101111001010 00000 10011 0000010011 | ||
62 19 14 9 0 | ||
``` | ||
|
||
### Sorting IDs | ||
Since the first significant 43 bits represent the timestamp, SnowMint IDs are naturally sortable by creation time. IDs generated earlier will have a smaller numeric value than those generated later, allowing simple chronological ordering by comparing ID values directly. You can easily retreieve this timestamp by the following formula: | ||
``` | ||
(SnowMintID >> 20) + EPOCH | ||
``` | ||
|
||
### Time Synchronization in Distributed Systems | ||
|
||
In distributed systems, it is **crucial** that all nodes maintain synchronized clocks to ensure the uniqueness of the IDs. Since the SnowMint algorithm heavily relies on the system timestamp (43 bits of the ID represent the time), any drift in a node's clock can lead to the generation of duplicate IDs, which breaks the uniqueness guarantee. | ||
|
||
To avoid this issue: | ||
- **Synchronize Time Across Nodes**: Use tools like **NTP (Network Time Protocol)** or similar to keep system clocks in sync. | ||
- **Monitor Time Drift**: Ensure that the time drift between nodes is kept to a minimum (e.g., within a few milliseconds). | ||
- **Fallback Mechanism**: If a node detects that its clock is out of sync, it should halt ID generation until the clock is corrected to prevent collisions. | ||
|
||
Failing to synchronize time across all nodes may result in non-unique IDs being generated, which could lead to issues in systems where uniqueness is critical. | ||
|
||
## The Protocol | ||
|
||
SnowMint uses a lightweight, highly optimized custom protocol over raw TCP connections. This design focuses on speed and simplicity, ensuring ultra-fast ID generation and retrieval. | ||
|
||
### How it Works: | ||
1. **Connection**: Clients open a TCP connection to the SnowMint server. | ||
2. **Command**: The client sends a single `GET` command to the server. | ||
3. **Response**: The server responds immediately with a 64-bit unique ID. | ||
|
||
This minimalist protocol reduces overhead, delivering unparalleled speed compared to traditional HTTP-based services. | ||
|
||
### Performance Benefits: | ||
- **Raw TCP**: Eliminates HTTP headers and other overhead, reducing the time between a request and response. | ||
- **Low-Latency**: Designed for microsecond-scale latencies, making it ideal for high-throughput systems. | ||
|
||
## Install | ||
|
||
The SnowMint server accepts the following optional commandline flags: | ||
|
||
`--address` The address for the server to bind to (default: localhost) | ||
|
||
`--port` The port for the server to bind to (default: 8080) | ||
|
||
`--datacenter` The server's data center ID between 0 and 31 (default: 0) | ||
|
||
`--node` The server's node ID between 0 and 31 (default: 0) | ||
|
||
`--epoch` Your organization's epoch in milliseconds (default: 0 _Wednesday, December 31, 1969 7:00:00 PM_) | ||
|
||
Note that in distributed systems, it is **crucial** that all instances of SnowMint servers are started with the same `epoch`. | ||
|
||
### Native Deployment | ||
1. Download the latest release from the [SnowMint releases page](#) (or alternatively, you can clone this repository and build the binary yourself with `go build`). | ||
2. Extract the archive and run the binary with your desired flags; e.g.: | ||
```bash | ||
./snowmint --node=<NODE_ID> | ||
``` | ||
|
||
### Docker Deployment | ||
To run SnowMint in a Docker container, simply use the following with your desired flags; e.g.: | ||
```bash | ||
docker pull mxmlkzdh/snowmint:latest | ||
docker run -d --name snowmint -p 8080:8080 mxmlkzdh/snowmint --node=<NODE_ID> | ||
``` | ||
|
||
### Generate Your First SnowMint ID | ||
Once an instance of the SnowMint server is up and running, execute the following command in your favorite terminal emulator and you'll receive a newly minted SnowMint ID! | ||
```bash | ||
echo GET | nc localhost 8080 | ||
``` | ||
## Clients | ||
SnowMint provides easy-to-use SDKs for popular programming languages to integrate with your system and retrieve unique IDs. | ||
| Language | Description | | ||
| ------------- | ------------- | | ||
| [Go SDK](https://github.com/mxmlkzdh/snowmint-go) | The Go client SDK allows seamless integration into Go applications. A simple GET request over TCP fetches the unique ID. | | ||
| [Java SDK](https://github.com/mxmlkzdh/snowmint-java) | The Java SDK offers a similarly efficient way to connect to the SnowMint server, providing support for applications in JVM environments. | | ||
## Benchmarks | ||
SnowMint has been benchmarked to handle thousands of requests per second, with latencies in the microsecond range. Thanks to the custom protocol and raw TCP connections, it outperforms traditional HTTP-based systems by a significant margin. | ||
- **ID Generation Rate**: More than 100,000 IDs per second per node. | ||
- **Latency**: Sub-millisecond, typically under X microseconds. | ||
- In a native deployment, response time for each unique ID is roughly between 5 to 10 microseconds. | ||
- In a Docker container, response time for each unique ID is roughly between 10 to 15 microseconds. | ||
## Sources | ||
- [Snowflake ID](https://en.wikipedia.org/wiki/Snowflake_ID) | ||
## License | ||
The SnowMint project is licensed under the [MIT License](LICENSE). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters