Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Caching system for shard file manager #159

Merged
merged 3 commits into from
Feb 1, 2025
Merged

Caching system for shard file manager #159

merged 3 commits into from
Feb 1, 2025

Conversation

hoytak
Copy link
Collaborator

@hoytak hoytak commented Jan 30, 2025 •
edited
Loading

This PR enables a caching system for the shard file manager by:

  • shard file manager is associated with a single directory. All registered shards must be in that directory.
  • Within-process caching is optionally done by directory.
  • The shard file manager can optionally reference a secondary manager for chunk and file lookup. In our case, the main cache directory is cached and global, but each session directory has its own shard file manager where lookup is attempted first, falling back to the global cache directory manager if that fails.

@hoytak hoytak requested a review from seanses January 30, 2025 01:54
@hoytak hoytak force-pushed the hoytak/250127-referencing-shard-file-manager branch from d64897d to b43d6a9 Compare January 30, 2025 20:13
seanses
seanses reviewed Jan 31, 2025
View reviewed changes
mdb_shard/src/shard_file_handle.rs
Self::load_all(path, false)
}

pub fn load_all(path: impl AsRef<Path>, load_expired: bool) -> Result<Vec<Arc<Self>>> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is load_all(.., true) used anywhere? If not we don't need to make this function pub.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it was used for testing, but then changed that.

mdb_shard/src/shard_file_manager.rs Outdated Show resolved Hide resolved
mdb_shard/src/shard_file_manager.rs Show resolved Hide resolved
data/src/shard_interface.rs Outdated Show resolved Hide resolved
mdb_shard/src/shard_file_manager.rs Show resolved Hide resolved
seanses
seanses approved these changes Jan 31, 2025
View reviewed changes
Copy link
Collaborator

@seanses seanses left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@hoytak hoytak merged commit d3144cd into main Feb 1, 2025
2 checks passed
@hoytak hoytak deleted the hoytak/250127-referencing-shard-file-manager branch February 1, 2025 00:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@seanses seanses seanses approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hoytak @seanses