useless MaxMemory

[goupdate]

config param MaxMemory is used to dump all data from memory when its exhausted, but it doesnt, just writes limit is reached and do nothing:

2025/01/23 19:35:39 Memory used: 10248, Max Memory: 10240
2025/01/23 19:35:39 Memory used: 10248, Max Memory: 10240
2025/01/23 19:35:39 Memory used: 10248, Max Memory: 10240
2025/01/23 19:35:39 Memory used: 10248, Max Memory: 10240
2025/01/23 19:35:39 Memory used: 10248, Max Memory: 10240
v=Hello, SugarDB!
2025/01/23 19:35:39 Memory used: 10248, Max Memory: 10240
v=Hello, SugarDB!3

instead of this its usefull to make it memory-work-set, all other data should stored in file / snapshot / aof file and be searching in mem and there.

[kelvinmwinuka]

@goupdate what eviction policy have you set? With noeviction, writes are stopped when a hard memory limit is reached. Otherwise, evictions are carried out to make more room for new writes. The store does not overflow to disk, we do not plan to implement this. Snapshots must be deliberately triggered or configured to run on an interval/write threshold.

[goupdate]

what eviction policy have you set?

nothing. default.

With noeviction, writes are stopped when a hard memory limit is reached. Otherwise, evictions are carried out to make more room for new writes.

Thats totally incorrect. "MAX memory" is parameter (by its name) to set barrier for library to set UPPER memory limit. So if your logic based only on evictions, then "MaxMemory" is not "MAX" memory limit usage as LRU and other libs do, it's something other. Without real MaxMemory limit we cant use this library. It should not skip storing data. NoEvictions - maybe in memory?
Or rename MaxMemory or describe it that it is not RAM max memory, its whole physicall drive datasize memory limit - thats another point from it all looks correct, but it's uselesss as we have huge SSD/HDD and can hold all with RAIDs, but all of us have <64GB RAM and it's usefull to have RAM MAX memory limit as it is in other libs (redis, leveldb, sugardb, etc.).

[guycipher]

@goupdate that's not incorrect. Read what @kelvinmwinuka wrote, and read it carefully.

[guycipher]

@goupdate I write LSM tree's what are you talking about max memory? You always set the memory you want to use(for runs to disk). In this has what Kelvin is saying is right. It's safe based on his systems logic. If you want another solution use log structured.

[guycipher]

@goupdate you mention redis and then leveldb. Their completely different mate.

[guycipher]

Don't take anything I'm saying the wrong way, I'm telling you this so you can learn. Have a good day.

[goupdate]

i know guys.
i'm writing about config "max memory limit". it presents there and there.
if you set leveldb such limt - it never outperforms that value of RAM.

But here i created default build-in server, set MaxMemory to 10240 and expect library never runs out of 10KB RAM just storing new values after filling up 10kb, to the drive, but it skips. It should not do so!

[goupdate]

If im wrong, ok, there are ways how to do this in redis (is this alias-clone or no?) or in sugardb or on other forks of redis.
All os them allows to set real MAX memory limit and library keep new values not in RAM, in drive or swaps if LRU swapping occured.

[guycipher]

That's different. That's offloading to disk. I don't know if they have that option. Yes I agree this is more for cache though @goupdate.

[guycipher]

If you want memory and disk you're looking for an LSM tree.

[guycipher]

Are you looking for embedded storage with no redis capabilities? @kelvinmwinuka I think you should add the offloading if you have it, I'm not sure. Redis does have offloading I believe. This is different than log structured approaches though mind you both.

[goupdate]

sure, LSM tree. by description MaxMemory seems to be that, but it doesn't.
you know, leveldb works like LSM tree, but not, there some caching, maybe here too? cause just skipping added values it's too danger and critical, there are no any info here about that - library just skips values if they DISK limit is reached (not even RAM) and MaxMemory is disk limit, not RAM.

I have tested this lib to add to my projects, but hopelessly should skip it, need build-in redis analog like LRU lib https://github.com/MasterDimmy/golang-lruexpire with offloading like LSM tree as you mensioned.

[guycipher]

True, yeah @kelvinmwinuka you might have to implement a write ahead log to turn those blocked writes somewhere temporarily. @goupdate true, yeah I'm not sure. I'm not the Redis guy, lsm guy.

[goupdate]

https://github.com/nmmmnu/HM4

https://github.com/facebook/rocksdb

LSM trees to fix redis lacks of this are going on, but i can't see build-in solutions for golang yet.

[goupdate]

maybe some usefull things from this?

https://github.com/JyotinderSingh/goLSM

it has what this needs.

[kelvinmwinuka]

@goupdate @guycipher offloading to disk is not a planned feature anytime soon.