Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

README modifications #19

Merged
merged 4 commits into from
Sep 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 121 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,121 @@
# SnowMint
A blazingly fast unique and roughly sortable IDs generator, based on Twitter's Snowflake
# SnowMint - A Blazingly Fast Unique ID Generator
[![Go Workflow](https://github.com/mxmlkzdh/snowmint/actions/workflows/go.yml/badge.svg)](https://github.com/mxmlkzdh/snowmint/actions)

**SnowMint** is a high-performance, distributed unique ID generator based on X's Snowflake algorithm. It provides unique, roughly sortable IDs that are generated using a client/server model with a custom protocol.

## SnowMint IDs

### ID Generation Algorithm
SnowMint leverages X’s Snowflake algorithm to generate **signed, non-negative 64-bit unique identifiers called SnowMint IDs**. The format of the ID consists of:

- **Sign**: A 1-bit field. It will always be 0.
- **Timestamp**: A 43-bit field representing the current time in milliseconds since your organization's epoch. This allows for generating 278 years 11 months 2.049 days worth of unique IDs from the epoch.
- **DataCenter ID**: A 5-bit field identifying the data center where the ID was generated.
- **Node ID**: A 5-bit field identifying the machine where the ID was generated.
- **Sequence**: A 10-bit field for a sequence number that resets every millisecond.

This combination ensures that SnowMint can (theoretically) generate more than 1,000,000 unique IDs per second, even in distributed environments.

### An Example
The SnowMint ID `817347092935625747` is generated by a SnowMint server with the following (configurable) parameters:
- Epoch: 946684800000 (Saturday, January 01 2000 00:00:00.00 GMT+0000)
- DataCenter ID: 0
- Node ID: 19

The binary representation of this ID is presented below. Note that for this particular ID, the timestamp is `1726167730122 (Thursday, September 12, 2024 19:02:10.122 GMT+0000)` and the sequence number is equal to `19`.

```
0 0001011010101111100110011011001101111001010 00000 10011 0000010011
62 19 14 9 0
```

### Sorting IDs
Since the first significant 43 bits represent the timestamp, SnowMint IDs are naturally sortable by creation time. IDs generated earlier will have a smaller numeric value than those generated later, allowing simple chronological ordering by comparing ID values directly. You can easily retreieve this timestamp by the following formula:
```
(SnowMintID >> 20) + EPOCH
```

### Time Synchronization in Distributed Systems

In distributed systems, it is **crucial** that all nodes maintain synchronized clocks to ensure the uniqueness of the IDs. Since the SnowMint algorithm heavily relies on the system timestamp (43 bits of the ID represent the time), any drift in a node's clock can lead to the generation of duplicate IDs, which breaks the uniqueness guarantee.

To avoid this issue:
- **Synchronize Time Across Nodes**: Use tools like **NTP (Network Time Protocol)** or similar to keep system clocks in sync.
- **Monitor Time Drift**: Ensure that the time drift between nodes is kept to a minimum (e.g., within a few milliseconds).
- **Fallback Mechanism**: If a node detects that its clock is out of sync, it should halt ID generation until the clock is corrected to prevent collisions.

Failing to synchronize time across all nodes may result in non-unique IDs being generated, which could lead to issues in systems where uniqueness is critical.

## The Protocol

SnowMint uses a lightweight, highly optimized custom protocol over raw TCP connections. This design focuses on speed and simplicity, ensuring ultra-fast ID generation and retrieval.

### How it Works:
1. **Connection**: Clients open a TCP connection to the SnowMint server.
2. **Command**: The client sends a single `GET` command to the server.
3. **Response**: The server responds immediately with a 64-bit unique ID.

This minimalist protocol reduces overhead, delivering unparalleled speed compared to traditional HTTP-based services.

### Performance Benefits:
- **Raw TCP**: Eliminates HTTP headers and other overhead, reducing the time between a request and response.
- **Low-Latency**: Designed for microsecond-scale latencies, making it ideal for high-throughput systems.

## Install

The SnowMint server accepts the following optional commandline flags:

`--address` The address for the server to bind to (default: localhost)

`--port` The port for the server to bind to (default: 8080)

`--datacenter` The server's data center ID between 0 and 31 (default: 0)

`--node` The server's node ID between 0 and 31 (default: 0)

`--epoch` Your organization's epoch in milliseconds (default: 0 _Wednesday, December 31, 1969 7:00:00 PM_)

Note that in distributed systems, it is **crucial** that all instances of SnowMint servers are started with the same `epoch`.

### Native Deployment
1. Download the latest release from the [SnowMint releases page](#) (or alternatively, you can clone this repository and build the binary yourself with `go build`).
2. Extract the archive and run the binary with your desired flags; e.g.:
```bash
./snowmint --node=<NODE_ID>
```

### Docker Deployment
To run SnowMint in a Docker container, simply use the following with your desired flags; e.g.:
```bash
docker pull mxmlkzdh/snowmint:latest
docker run -d --name snowmint -p 8080:8080 mxmlkzdh/snowmint --node=<NODE_ID>
```

### Generate Your First SnowMint ID
Once an instance of the SnowMint server is up and running, execute the following command in your favorite terminal emulator and you'll receive a newly minted SnowMint ID!
```bash
echo GET | nc localhost 8080
```

## Clients

SnowMint provides easy-to-use SDKs for popular programming languages to integrate with your system and retrieve unique IDs.

| Language | Description |
| ------------- | ------------- |
| [Go SDK](https://github.com/mxmlkzdh/snowmint-go) | The Go client SDK allows seamless integration into Go applications. A simple GET request over TCP fetches the unique ID. |
| [Java SDK](https://github.com/mxmlkzdh/snowmint-java) | The Java SDK offers a similarly efficient way to connect to the SnowMint server, providing support for applications in JVM environments. |

## Benchmarks
SnowMint has been benchmarked to handle thousands of requests per second, with latencies in the microsecond range. Thanks to the custom protocol and raw TCP connections, it outperforms traditional HTTP-based systems by a significant margin.

- **ID Generation Rate**: More than 100,000 IDs per second per node.
- **Latency**: Sub-millisecond, typically under X microseconds.
- In a native deployment, response time for each unique ID is roughly between 5 to 10 microseconds.
- In a Docker container, response time for each unique ID is roughly between 10 to 15 microseconds.

## Sources
- [Snowflake ID](https://en.wikipedia.org/wiki/Snowflake_ID)

## License
The SnowMint project is licensed under the [MIT License](LICENSE).
4 changes: 2 additions & 2 deletions config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ type Config struct {
func LoadConfig() *Config {
address := flag.String("address", "localhost", "The address to bind to.")
port := flag.Int("port", 8080, "The port to bind to.")
dataCenterID := flag.Int("dataCenterID", 0, "The data center ID.")
nodeID := flag.Int("nodeID", 0, "The node ID.")
dataCenterID := flag.Int("datacenter", 0, "The data center ID.")
nodeID := flag.Int("node", 0, "The node ID.")
epoch := flag.Int("epoch", 0, "The epoch in milliseconds.")
flag.Parse()
return &Config{
Expand Down
Loading