Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: Reuse rand for shuffling queues #916

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

l0nax
Copy link
Contributor

@l0nax l0nax commented Sep 12, 2024

Instead of creating a new rand.Rand every time we need the next queues, we create it once and reuse it.

The reason for this change comes from monitoring our application: After a few minutes, the github.com/hibiken/asynq.(*processor).queues method will consume a tremendous amount of CPU time and create a lot of garbage.

Since the queue method is called in its own goroutine, there is no need to add any synchronization primitives.

Here is the profiling flame graph of our application in a time window of 5 minutes:
image

processor.go Outdated Show resolved Hide resolved
Instead of creating a new `rand.Rand` every time we need the next
queues, we create it once and reuse it.
@kamikazechaser
Copy link
Collaborator

@l0nax Checkout my comment on #868.

@pascaldekloe
Copy link

Don't forget to https://go.dev/blog/randv2 for best results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed performance pr-refactor work in progress Assignee is working on the issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants