-
Notifications
You must be signed in to change notification settings - Fork 1
License
santeri/pydiststore
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
h1. Diststore *NOTE: this is not really suitable for real use yet.* The system is a asymmetric key/value store where all the data is stored on two master servers. Commonly accessed keys are distributed to more servers than the infrequently accessed keys. The master nodes are selected based in which ones have the most keys. If a master node fails a new is selected automatically and it will sync itself with the old remaining master. [todo: persistent storage ] [todo: key expiration] h2. Installation h3. Using source 1. Install python & python-httplib2 2. Install with setup.py 3. Run the diststored script on all nodes using, for example, supervisord. h3. Using debuild 1. # apt-get install pbuilder 2. # /usr/lib/pbuilder/pbuilder-satisfydepends 3. $ debuild -us -uc 4. install generated debian package. 4. Run diststored on all nodes [todo: daemonize & install init scripts] h2. Test cases Before running the test cases make sure that the ip range used by them is set up properly. The test cases use addresses in the range 10.1.2.1 to 10.1.2.11, which can be added as aliases on lo0. h2. Node services The nodes run a http server for settings and getting keys/values and a thread that listens for multicast request. h3. Post request All key/value pairs are set using http POST, using a form with @key@ and @value@ fields. They are sent to the current master servers, or if there aren't any a master selection is done based on which nodes have the most keys. If a master server goes down it will be noticed on the next post, when a node tries to send the key to the master. If that happens all nodes will redo the master selection process. The new master server will then start fetching all keys from the remaining master. [todo: deleting keys] h3. Post local The same as post, but won't try to send the key/value to masters. h3. Get request If a node receives a get request for a key which it doesn't have locally, a multicast request is sent to all nodes. All other node which have the key will reply to the node using a udp packet and the original node then gets the value using http. The value is sent to the client and the key/value pair is cached locally. [todo: The key/value pairs expire after a set timeout on all nodes except for the master nodes] h3. Get local Get a key locally, without requesting missing keys from the cluster. h3. List keys List all keys the node has, used by the synchronization process which is done when a new nodes becomes master.
About
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published