Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Embed key into dict entry #541

Merged
merged 22 commits into from
Jul 2, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions src/db.c
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,14 @@ robj *lookupKeyWriteOrReply(client *c, robj *key, robj *reply) {
return o;
}

/* Add the key to the DB. It's up to the caller to increment the reference
/* Add the key to the DB.
*
* The kvstore handles `key` based on `dictType` during initialization:
* - If `dictType.embedded-entry` is 1, it clones the `key`.
* - Otherwise, it assumes ownership of the `key`.
* In this case a copy of `key` is made in kvstore, the caller must ensure the `key` is properly freed.
hpatro marked this conversation as resolved.
Show resolved Hide resolved
*
* It's up to the caller to increment the reference
* counter of the value if needed.
*
* If the update_if_existing argument is false, the program is aborted
Expand Down Expand Up @@ -246,8 +253,13 @@ int getKeySlot(sds key) {
* give more control to the caller, nor will signal the key as ready
* since it is not useful in this context.
*
* The function returns 1 if the key was added to the database, making a
* copy of the SDS string, otherwise 0 is returned, The caller should free the SDS string. */
* The function returns 1 if the key was added to the database, otherwise 0 is returned.
*
* The kvstore handles `key` based on `dictType` during initialization:
* - If `dictType.embedded-entry` is 1, it clones the `key`.
* - Otherwise, it assumes ownership of the `key`.
* In this case a copy of `key` is made in kvstore, the caller must ensure the `key` is properly freed.
hpatro marked this conversation as resolved.
Show resolved Hide resolved
*/
int dbAddRDBLoad(serverDb *db, sds key, robj *val) {
hpatro marked this conversation as resolved.
Show resolved Hide resolved
int slot = getKeySlot(key);
dictEntry *de = kvstoreDictAddRaw(db->keys, slot, key, NULL);
Expand Down
4 changes: 4 additions & 0 deletions src/dict.c
Original file line number Diff line number Diff line change
Expand Up @@ -529,6 +529,10 @@ int dictAdd(dict *d, void *key, void *val) {
* with the existing entry if existing is not NULL.
*
* If key was added, the hash entry is returned to be manipulated by the caller.
*
* The dict handles `key` based on `dictType` during initialization:
* - If `dictType.embedded-entry` is 1, it clones the `key`.
* - Otherwise, it assumes ownership of the `key`.
*/
dictEntry *dictAddRaw(dict *d, void *key, dictEntry **existing) {
/* Get the position for the new key or NULL if the key already exists. */
Expand Down
6 changes: 6 additions & 0 deletions src/kvstore.h
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,12 @@ typedef dict *(kvstoreDictLUTDefragFunction)(dict *d);
void kvstoreDictLUTDefrag(kvstore *kvs, kvstoreDictLUTDefragFunction *defragfn);
void *kvstoreDictFetchValue(kvstore *kvs, int didx, const void *key);
dictEntry *kvstoreDictFind(kvstore *kvs, int didx, void *key);
/*
* The kvstore handles `key` based on `dictType` during initialization:
* - If `dictType.embedded-entry` is 1, it clones the `key`.
* - Otherwise, it assumes ownership of the `key`.
* The caller must ensure the `key` is properly freed.
*/
hpatro marked this conversation as resolved.
Show resolved Hide resolved
dictEntry *kvstoreDictAddRaw(kvstore *kvs, int didx, void *key, dictEntry **existing);
hpatro marked this conversation as resolved.
Show resolved Hide resolved
void kvstoreDictSetKey(kvstore *kvs, int didx, dictEntry *de, void *key);
void kvstoreDictSetVal(kvstore *kvs, int didx, dictEntry *de, void *val);
Expand Down
10 changes: 5 additions & 5 deletions src/sds.c
Original file line number Diff line number Diff line change
Expand Up @@ -201,14 +201,14 @@ static inline size_t sdsminlen(sds s) {

/* This method copies the sds `s` into `buf` which is the target character buffer. */
size_t sdscopytobuffer(unsigned char *buf, size_t buf_len, sds s, uint8_t *hdr_size) {
size_t reqd_keylen = sdsminlen(s);
size_t required_keylen = sdsminlen(s);
if (buf == NULL) {
return reqd_keylen;
return required_keylen;
}
assert(buf_len >= reqd_keylen);
memcpy(buf, sdsAllocPtr(s), reqd_keylen);
assert(buf_len >= required_keylen);
memcpy(buf, sdsAllocPtr(s), required_keylen);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do you need this double information about the header size? you can extract it from the "sds", right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An sds s is a pointer to where the string content starts, so it can be used as a C string. It does not point to the start of the allocation. The header is store before the char data in the same allocation, i.e. at s[-1], s[-2], etc. The header is backwards-encoded in some way, so the byte at s[-1] says which kind of header it is and how large the header is.

       sds-----.
               |
  allocation   v
  +-----------+----------------------+
  | hdr       |string contents    \0 |
  +-----------+----------------------+

When we store this embedded, we want to be able to restore the sds pointer, so we store the size of the header size in the first. When we restore the sds pointer, we can find it using S (the offset to the string contents).

+-+-----------+----------------------+
|S| hdr       |string contents    \0 |
+-+-----------+----------------------+

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean the callers of this function can call sdsHdrSize(s[-1]) themselves? I'm not sure if it's public.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, right agree. I am not so supportive of the desire to keep something as sds when it's not, but that's already in my overall comment, maybe I'll elaborate more there. Thanks

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

key is being used throughout the engine as a sds so changing that would be even more touchpoint. We could dynamically build it on the dictGetKey call but that would be additional penalty. Hence, storing it as a sds made the most sense.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have the same question, the low 3bit in sdshdr.flag have told us the header length, and exposing function sdsHdrSize will not lead to coupling between dict.c and sds.c, I don't see any harm in it.

                                                            
 sdshdr               sds                                   
   │                  │                                     
   │                  │                                     
   ▼─────┬─────┬─────┌▼───────────────────────────────┬──┐  
   │ len │ alloc     │                                │  │  
   │     │     │flags│                                │\0│  
   └─────┼─────┼─────└────────────────────────────────┴──┘  
                                                                                                                                                                                            

*hdr_size = sdsHdrSize(s[-1]);
return reqd_keylen;
return required_keylen;
}

/* Free an sds string. No operation is performed if 's' is NULL. */
Expand Down
1 change: 0 additions & 1 deletion src/server.h
Original file line number Diff line number Diff line change
Expand Up @@ -3558,7 +3558,6 @@ int dictSdsKeyCaseCompare(dict *d, const void *key1, const void *key2);
void dictSdsDestructor(dict *d, void *val);
void dictListDestructor(dict *d, void *val);
void *dictSdsDup(dict *d, const void *key);
size_t dictSdsKeyToBytes(unsigned char *buf, const void *key, unsigned char *header_size);

/* Git SHA1 */
char *serverGitSHA1(void);
Expand Down
Loading