Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

connectivity: Check for unexpected packet drops #2151

Merged
merged 3 commits into from
Dec 11, 2023

Conversation

pchaigno
Copy link
Member

@pchaigno pchaigno commented Dec 5, 2023

This pull requests ensures we fail the connectivity tests if we hit any unexpected packet drops. The first commit transforms the existing missed-tail-call drops check into a new check for unexpected packet drops. The second commit defines a new flag to set the list of expected packet drop reasons. The last commit adds a bunch of packet drop reasons.

@brb

This comment was marked as outdated.

@pchaigno

This comment was marked as outdated.

Copy link
Member

@tklauser tklauser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM apart from an open question on the new drop reasons CLI flag.

internal/cli/cmd/connectivity.go Show resolved Hide resolved
Convert the check for missed tail call packet drops into a more general
check for any unexpected packets drops. Current expected packet drop
reasons are Policy denied and Policy denied by denylist only. Others
will be added in subsequent commits.

Contrary to the Missed tail call check who only ran in the context of
the conn-disrupt test, this new check is always run by default.

Signed-off-by: Paul Chaignon <[email protected]>
@brb
Copy link
Member

brb commented Dec 7, 2023

Nice! Have you tried running /test on cilium/cilium after running git grep -l cilium_cli_ci_version: | xargs sed -i "s/cilium_cli_ci_version: .*/cilium_cli_ci_version: $CI_BUILD_VSN/g"?

@pchaigno
Copy link
Member Author

pchaigno commented Dec 7, 2023

Nice! Have you tried running /test on cilium/cilium after running git grep -l cilium_cli_ci_version: | xargs sed -i "s/cilium_cli_ci_version: .*/cilium_cli_ci_version: $CI_BUILD_VSN/g"?

No. But we have a flag to set the list of expected reasons, so we can adjust if specific tests need additional reasons.

@brb
Copy link
Member

brb commented Dec 7, 2023

No. But we have a flag to set the list of expected reasons, so we can adjust if specific tests need additional reasons.

From my personal experience, I'd use it as a last resort. It's difficult to keep a mental map in a head of all options which need to be customized in each job X branches.

Also, if you don't test it now, then it might become someone else's problem once a new CLI version is released, and cilium/cilium's CLI dependency is bumped.

@pchaigno
Copy link
Member Author

pchaigno commented Dec 7, 2023

No. But we have a flag to set the list of expected reasons, so we can adjust if specific tests need additional reasons.

From my personal experience, I'd use it as a last resort. It's difficult to keep a mental map in a head of all options which need to be customized in each job X branches.

Agree. Either it's obvious (SRv6 drop in SRv6 workflow) or I'll update the default list.

Also, if you don't test it now, then it might become someone else's problem once a new CLI version is released, and cilium/cilium's CLI dependency is bumped.

I'm planning to update the CLI myself. I need this quickly.

Copy link
Member

@brb brb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried running the change myself in cilium/cilium#29718, and most of the red builds are due to this check failing.

If we merge it now (or w/o trying it out), then this change will block other changes (which will be released together) from being used in the cilium/cilium CI.

This new flag can be used to customize the set of expected reasons for
packet drops, for the new test that ensure we don't have any unexpected
packet drops.

Signed-off-by: Paul Chaignon <[email protected]>
We currently have many packet drops that are more or less expected. Some
like "Authentication required" are expected for specific features and
others like "Stale or unroutable IP" are known and understood even
though we'd prefer to avoid them.

Signed-off-by: Paul Chaignon <[email protected]>
@pchaigno
Copy link
Member Author

pchaigno commented Dec 8, 2023

I tried running the change myself in cilium/cilium#29718, and most of the red builds are due to this check failing.

I added those drop reasons.

If we merge it now (or w/o trying it out), then this change will block other changes (which will be released together) from being used in the cilium/cilium CI.

That's FUD given we have a flag to add other drop reason exceptions.

@pchaigno pchaigno requested a review from brb December 8, 2023 12:24
@pchaigno pchaigno added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Dec 10, 2023
@michi-covalent
Copy link
Contributor

the kind failure is tracked in #2162. merging 🚀🙏

@michi-covalent michi-covalent merged commit 4d0d9bf into main Dec 11, 2023
18 of 20 checks passed
@michi-covalent michi-covalent deleted the pr/pchaigno/check-unexpected-packet-drops branch December 11, 2023 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready-to-merge This PR has passed all tests and received consensus from code owners to merge.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants