Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Xapian Sidekiq queue #7922

Merged
merged 1 commit into from
Sep 29, 2023
Merged

Add Xapian Sidekiq queue #7922

merged 1 commit into from
Sep 29, 2023

Conversation

gbp
Copy link
Member

@gbp gbp commented Sep 28, 2023

What does this do?

Queue InfoRequestExpireJob on a Xapian Sidekiq queue which has concurrency disable so only one job is processed at a time.

Why was this needed?

Calls to InfoRequestExpireJob when passed a User would block processing of other jobs such as attachment processing and cause support mail.

Implementation notes

The order of the queues in config/general.yml is important (as they aren't weighted). the worker threads will always prioritise the first queue (default) so attachment processing and other jobs get priority.

Because we dealing with Xapian we also want to ensure only one Xapian related task is happening at any one time (although currently the expire job, isn't actually running Xapian indexing, but having multiple expire job, for the same resource running at the same time, could cause issues too) adding the sidekiq-limit_fetch to configure the limits in config/general.yml allows us to do this.

Queue `InfoRequestExpireJob` on this new queue and disable concurrency
so only one job is processed at a time.
@gbp
Copy link
Member Author

gbp commented Sep 29, 2023

I have tested this on our staging ENV and it seems to be working well. i.e. two queues, all InfoRequestExpireJob jobs are on the xapian queue, getting exec one at a time.

@gbp gbp merged commit 430c98f into develop Sep 29, 2023
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant