Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce default concurrency in config.example.yml #775

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

amorken
Copy link

@amorken amorken commented Jan 30, 2025

The default concurrency settings should be conservative and increased when there's a need rather than be high by default and scaled down when they cause problems.

The default concurrency settings should be conservative and increased when there's a need rather than be high by default and scaled down when they cause problems.
Copy link

codecov bot commented Jan 30, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.03%. Comparing base (a02b9cf) to head (7ad9163).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #775      +/-   ##
==========================================
- Coverage   80.05%   80.03%   -0.03%     
==========================================
  Files         134      134              
  Lines       18834    18834              
  Branches     2800     2800              
==========================================
- Hits        15078    15073       -5     
- Misses       2912     2916       +4     
- Partials      844      845       +1     

see 5 files with indirect coverage changes

@einarmo
Copy link
Collaborator

einarmo commented Jan 30, 2025

This won't actually fix the problem, as they are hard-coded elsewhere. We can look into reducing them, I agree they are way too high, but I disagree with setting them to 1. If a user produces enough events to ever use more than a single parallel write, they probably want to have at least a few parallel writes.

It's not always easy to understand that the extractor isn't keeping up, and users do not read documentation for extractors. This is not the same as API docs or most of our other technical documentation, as the level of technical know-how is often way lower with extractors.

I've personally never seen a user correctly change these settings. I've seen them set to 0, and to 1000. If we can set sane defaults that rarely need changing I would much rather do that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants