Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove cached node bytes from merkle nodes #2393

Merged
merged 37 commits into from
Dec 14, 2023
Merged

Conversation

dboehm-avalabs
Copy link
Contributor

We mostly load nodes just before we edit them, so holding on to the bytes just wastes memory.

Copy link

@danlaine danlaine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a benchmark that measures the effect of this change?

x/merkledb/codec.go Outdated Show resolved Hide resolved
x/merkledb/db.go Outdated Show resolved Hide resolved
x/merkledb/codec.go Outdated Show resolved Hide resolved
x/merkledb/codec.go Outdated Show resolved Hide resolved
x/merkledb/codec.go Outdated Show resolved Hide resolved
Base automatically changed from ShrinkNodeStorage to dev November 29, 2023 20:57
Co-authored-by: Dan Laine <[email protected]>
Signed-off-by: David Boehm <[email protected]>
@dboehm-avalabs
Copy link
Contributor Author

Is there a benchmark that measures the effect of this change?

Not explicitly. This is mostly to reduce the size of cache entries since they currently store the node's bytes twice, once as the raw bytes and once in the various fields on the node struct. Removing this extra copy with roughly halve the memory footprint per node, so we should be able to store almost twice the nodes in the cache.

x/merkledb/codec.go Outdated Show resolved Hide resolved
@@ -66,17 +64,7 @@ func (n *node) hasValue() bool {

// Returns the byte representation of this node.
func (n *node) bytes() []byte {
if n.nodeBytes == nil {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing this cached value means we'll have to reserialize the node before writing it. Are we concerned about the additional time that'll take? Seems to be a tradeoff between memory usage and CPU usage.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

anything we are writing is something we edited so this would have always been nil

@dboehm-avalabs dboehm-avalabs added this pull request to the merge queue Dec 14, 2023
Merged via the queue into dev with commit 4909a20 Dec 14, 2023
17 checks passed
@dboehm-avalabs dboehm-avalabs deleted the RemoveNodeBytes branch December 14, 2023 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants