Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix][e2e] Fix e2e failed due to server overload. #15578

Closed
wants to merge 14 commits into from
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ master:
server-load-protection:
enabled: true
# Master max cpu usage, when the master's cpu usage is smaller then this value, master server can execute workflow.
max-cpu-usage-percentage-thresholds: 0.9
max-cpu-usage-percentage-thresholds: 1.5
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
max-cpu-usage-percentage-thresholds: 1.5
max-cpu-usage-percentage-thresholds: 1.0

The max value of this parameter shouldld be 1.0.

# Master max JVM memory usage , when the master's jvm memory usage is smaller then this value, master server can execute workflow.
max-jvm-memory-usage-percentage-thresholds: 0.9
# Master max System memory usage , when the master's system memory usage is smaller then this value, master server can execute workflow.
Expand All @@ -216,7 +216,7 @@ worker:
server-load-protection:
enabled: true
# Worker max cpu usage, when the worker's cpu usage is smaller then this value, worker server can be dispatched tasks.
max-cpu-usage-percentage-thresholds: 0.9
max-cpu-usage-percentage-thresholds: 1.5
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
max-cpu-usage-percentage-thresholds: 1.5
max-cpu-usage-percentage-thresholds: 1.0

Copy link
Contributor Author

@jackyyyyyssss jackyyyyyssss Feb 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ruanwenjun Hello Sometimes the utilization rate of underlying resources can be greater than 1, so setting it to 1.5 can overload and cause e2e failure if set to 1
image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a bug, we may need to use getTotalCpuUsedPercentage() to compare with maxCpuUsagePercentageThresholds at ServerLoadProtection

# Worker max JVM memory usage , when the worker's jvm memory usage is smaller then this value, worker server can be dispatched tasks.
max-jvm-memory-usage-percentage-thresholds: 0.9
# Worker max System memory usage , when the worker's system memory usage is smaller then this value, worker server can be dispatched tasks.
Expand Down
Loading