Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slowdown when error dataset grows #340

Open
dcesari opened this issue Nov 13, 2024 · 0 comments
Open

Slowdown when error dataset grows #340

dcesari opened this issue Nov 13, 2024 · 0 comments
Assignees

Comments

@dcesari
Copy link
Member

dcesari commented Nov 13, 2024

I noticed that when error dataset grows (e.g. tenths of GB in a daily file), the arki-scan process tends to slow down by many times, while this is not noticed when importing to a regular dataset having daily files of the same size (I cannot say whether the slowdown affects also data that do not end in error, it could be the case, but I am not sure). Could this be related to how the error dataset is managed?
It is not a serious problem, because of course error dataset should not grow so much, but if it has a simple solution it would be welcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants