Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove mutex locks from Zip/7z backed archives #1916

Draft
wants to merge 8 commits into
base: master
Choose a base branch
from
Draft

Conversation

lhog
Copy link
Collaborator

@lhog lhog commented Jan 24, 2025

No description provided.

@lhog lhog marked this pull request as draft January 24, 2025 13:11
@lhog lhog changed the title Scan perf fix imp Remove mutex locks from Zip/7z backed archives Jan 24, 2025
Remove CBufferedArchive as part of the previous
Remove mutex lock from GetFile() in CZipArchive and CSevenZipArchive by opening up to ThreadPool::GetThreadNum() instances of archive files in a lazy manner
@p2004a
Copy link
Collaborator

p2004a commented Jan 24, 2025

The BufferedArchive is cache after decompression, OS cache is cache before decompression, aren't there some cases where some files are read many times and constant decompression will have large impact? (I have no data, just asking)

@lhog
Copy link
Collaborator Author

lhog commented Jan 24, 2025

I profiled BufferedArchive some time ago and found that the buffering didn't actually make any sense because most files (if not all) in most cases (if not all) were loaded only once per archive lifetime.

Your comment is well timed, I'll try it once again before slashing it.

@lhog
Copy link
Collaborator Author

lhog commented Jan 24, 2025

Actually I'm wrong, There are some recurring queries, not very substantial, but not negligible either.

@sprunk
Copy link
Collaborator

sprunk commented Jan 24, 2025

Would be nice if whatever method you used to measure could be available as a debug tool (printing a map of file : open count would already be sufficient). For example Lua files being opened twice is grounds for optimisation.

lhog added 6 commits January 24, 2025 16:05
This reverts commit 591df5a.
This reverts commit 3851edf.
… assumptions about the way files are requirested from multiple threads
@lhog
Copy link
Collaborator Author

lhog commented Jan 24, 2025

@p2004a can you have a look why dedicated cannot link ThreadPool functions?

We effectively need to split archive lib into two versions,
one with and one without full threadpool because unitsync and
dedicated do not use threadpool.
@p2004a
Copy link
Collaborator

p2004a commented Feb 2, 2025

I've pushed a commit that I believe makes it work, ptal. You need to check if it works under msvc, I've not tested there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants