Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataset set postmatch 5576 v10 #11962

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

catenacyber
Copy link
Contributor

Link to ticket: https://redmine.openinfosecfoundation.org/issues/
https://redmine.openinfosecfoundation.org/issues/5576

Describe changes:

  • detect/dataset: delay set operation after signature full match

SV_BRANCH=OISF/suricata-verify#2093

#11834 with avoid rerunning transforms and detection and new SV tests

DRAFT : this shows the POC to have feedback before doing a clean git history.
Feedback is excepted on the design :

  • detect: postmatch can run AppLayerTxMatch callbacks (in its own commit)
  • usage of delay_postmatch: buffers using dataset/set are put at the tail of inspections whatever their progress
  • usage of DETECT_ENGINE_INSPECT_SIG_MATCH_MORE_BUF: dataset may return this new case to tell that ok we match on a multi-buffer, but we still want to try all occurrences of a multi-buffer (instead of returning a simple early match)

@catenacyber
Copy link
Contributor Author

And new bug found in https://redmine.openinfosecfoundation.org/issues/7326

Copy link

codecov bot commented Oct 15, 2024

Codecov Report

Attention: Patch coverage is 80.23952% with 33 lines in your changes missing coverage. Please review.

Project coverage is 77.89%. Comparing base (378f678) to head (c8e68d4).

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #11962      +/-   ##
==========================================
- Coverage   82.76%   77.89%   -4.88%     
==========================================
  Files         910      910              
  Lines      249014   249103      +89     
==========================================
- Hits       206105   194046   -12059     
- Misses      42909    55057   +12148     
Flag Coverage Δ
fuzzcorpus ?
livemode 18.76% <43.11%> (+0.06%) ⬆️
pcap 44.11% <19.76%> (+0.26%) ⬆️
suricata-verify 62.19% <80.23%> (+0.03%) ⬆️
unittests 58.97% <19.16%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@suricata-qa
Copy link

Information: QA ran without warnings.

Pipeline 23107

Will be useful for dataset, when it needs to find a transaction
buffer again.
The set operation of dataset keyword was done even if signature
did not fully match, which is not the expected behavior.
We want dataset to behave like flowbits for instance.

This patch changes the behavior of the dataset keyword to do a
match and a post match for the set operation.
The postmatch retrieves the data, using the list identifier
associated to the buffer for this signature.

This avoids to store the buffer(s), when we do not have a
dedicated storage (per signature and per tx) that can own
and clean arbitrary buffers over multiple packets, in
the case the transaction spans over multiple packets
with different tx progresses for instance.
If detection runs on one packet, the InspectionBuffer are
cached and fast to get.
The most expensive case if for multi buffers, where we
need to run detection again, to see which occurences
match all payload keywords and should be added in the dataset.

Ticket: OISF#5576
@catenacyber
Copy link
Contributor Author

I had forgotten the no-random in SV

@inashivb inashivb self-requested a review October 16, 2024 05:50
@suricata-qa
Copy link

Information: QA ran without warnings.

Pipeline 23119

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants