Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-file plot.bin wrapper #3274

Open
nazar-pc opened this issue Dec 2, 2024 · 0 comments
Open

Multi-file plot.bin wrapper #3274

nazar-pc opened this issue Dec 2, 2024 · 0 comments
Labels
farmer Farming library/app good first issue Good for newcomers improvement it is already working, but can be better performance Related to performance measurement or improvement refactor

Comments

@nazar-pc
Copy link
Member

nazar-pc commented Dec 2, 2024

Right now we have a problem on Windows: due to NTFS file system that triggers full write of the file we disable plot cache on large farms and even for small farms writing of a couple terabytes takes a really long time.

We can do an annoying thing and split plot.bin into multiple files, let's say 100G each and create a wrapper data structure similar to DirectIoFile that will allow to transparently read and write contents as if it was a single file.

This way we constrain writes to just one file worth of space at a time and improve UX for Windows users.

@nazar-pc nazar-pc added refactor farmer Farming library/app improvement it is already working, but can be better performance Related to performance measurement or improvement labels Dec 2, 2024
@nazar-pc nazar-pc added the good first issue Good for newcomers label Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
farmer Farming library/app good first issue Good for newcomers improvement it is already working, but can be better performance Related to performance measurement or improvement refactor
Projects
Development

No branches or pull requests

1 participant