[hashset feature] Replace dict with hashset for keys and expire

[zuiderkwast]

This PR is based on #1186. When it is merged, this PR will change target to unstable.

Keys and expire are embedded in robj.

Dict is replaced by hashset for the keys and expire tables. Hashset is a hash table that doesn't have a dict entry. Pointers point directly to robj, where also the expire is stored, if any. Also pubsub channels are changed, since they use kvstore. Kvstore is changed from dict to hashset.

Some db.c functions like dbAdd, setKey and setExpire are changed to return the reallocated object after embedding the key and optional expire. setKey does not increment the reference counter, since it would require duplicating the object.

An robj value with an embedded key can be called valkey (val + key). A valkey typedef is added for robj and it's used when the object has an embedded key, to distinguish it from an object from other usages of robj (such as a string object from the command parser in c->argv).

keys                                      expire
hashset            robj                   hashset
+---+         +-------------------+        +---+
| 1 --------->| type, enc, lru    |<-------- 1 |
| 2 |         | refcount, flags   |        | 2 |
| 3 |         | ptr               |        | 3 |
| . |         | expire?           |        | . |
| . |         | embedded key      |        | . |
| . |         | embedded value?   |        | . |
+---+         +-------------------+        +---+

? = optional

A quick benchmark using valkey-benchmark showed small improvements for all types, but more for get and set than complex types.

Rebarding memory usage, DEBUG POPULATE 10000000 (10M) resulted in memory usage (used_memory_human) of 669M, compared to 872M on unstable. That's a 23% drop in memory usage and 21.28 bytes per key. (DEBUG POPULATE created keys of length 11 and string values of length 13. Both of these fit in an robj allocation of size 32 bytes.)

Fixes #992.

TODO:

• Implement memory prefetching (for io threading, to be ported from the dict implementation).

[SoftlyRaining]

Didn't look at db.c yet

[SoftlyRaining]

Think we'll implement something, or will we end up completely removing prefetch?

[zuiderkwast]

Yes... Help wanted. :)

[zuiderkwast]

@touitou-dan

The prefetch implementation for dict is accessing dict internals to do incremental lookup, one memory access at a time. It's breaking the abstraction and adds duplicated logic. I don't like that part.

For hashset, I have two ideas about how to fetch multiple keys in parallel:

1. A function that looks up multiple keys in parallel. It can return a bitmap with a bit for each key, whether it was found or not.

   unsigned hashsetBatchFind(int count, hashset **sets, const void **keys, void **found);

2. A function that incrementally looks up one key. It needs to be called repeatedly until it's done. The caller provides an (opaque) state that the implementation can use.

   int hashsetIncrementalFind(hashset *s, const void *key, void **found, hashsetIncrementalFindState *state);

   The prefetching code can run multiple such lookups in round-robin.

[codecov]

Codecov Report

Attention: Patch coverage is 93.85965% with 49 lines in your changes missing coverage. Please review.

Project coverage is 70.94%. Comparing base (5129254) to head (159bc70).

Files with missing lines	Patch %	Lines
src/kvstore.c	95.21%	12 Missing ⚠️
src/module.c	0.00%	10 Missing ⚠️
src/object.c	94.44%	7 Missing ⚠️
src/db.c	95.80%	6 Missing ⚠️
src/defrag.c	88.67%	6 Missing ⚠️
src/debug.c	63.63%	4 Missing ⚠️
src/memory_prefetch.c	40.00%	3 Missing ⚠️
src/replication.c	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           hashset    #1178      +/-   ##
===========================================
+ Coverage    70.77%   70.94%   +0.16%     
===========================================
  Files          115      115              
  Lines        64714    63760     -954     
===========================================
- Hits         45804    45233     -571     
+ Misses       18910    18527     -383     

Files with missing lines	Coverage Δ
src/aof.c	80.11% <100.00%> (-0.02%)	⬇️
src/bitops.c	93.76% <100.00%> (-0.01%)	⬇️
src/cluster.c	88.35% <100.00%> (-0.02%)	⬇️
src/cluster_legacy.c	86.30% <100.00%> (ø)
src/evict.c	97.70% <100.00%> (-0.06%)	⬇️
src/expire.c	96.59% <100.00%> (+0.02%)	⬆️
src/geo.c	93.58% <100.00%> (-0.02%)	⬇️
src/hyperloglog.c	91.23% <100.00%> (ø)
src/lazyfree.c	86.11% <100.00%> (ø)
src/pubsub.c	96.27% <100.00%> (+0.04%)	⬆️
... and 19 more

... and 6 files with indirect coverage changes

[hpatro]

This is great progress. Will take a look this week.

[zuiderkwast]

Included in #1186.