[Trendmicro Vision One] - Fixed cursor & pagination logic for "detections" data stream #12916
Conversation
ShourieG
added
Integration:trend_micro_vision_one
|
TODO - test this with a live account |
🚀 Benchmarks reportTo see the full report comment with |
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
| # If last pagination cycle was finished successfully | ||
| # we move the startDate pointer forward | ||
| # else we reprocess the last interval from the beginning. | ||
| # If none of the values are in the cursor it means is a fresh start |
There was a problem hiding this comment.
| # If none of the values are in the cursor it means is a fresh start | |
| # If none of the values are in the cursor it means it is a fresh start |
| @@ -27,29 +27,49 @@ request.transforms: | |||
| value: '{{page_size}}' | |||
| - set: | |||
| target: url.params.startDateTime | |||
| value: '[[formatDate ((parseDate .cursor.last_update_at).Add (parseDuration "-{{additional_look_back}}"))]]' | |||
| # If last pagination cycle was finished successfully | |||
There was a problem hiding this comment.
| # If last pagination cycle was finished successfully | |
| # If the last pagination cycle was completed successfully |
| value: '[[formatDate ((parseDate .cursor.last_update_at).Add (parseDuration "-{{additional_look_back}}"))]]' | ||
| # If last pagination cycle was finished successfully | ||
| # we move the startDate pointer forward | ||
| # else we reprocess the last interval from the beginning. |
There was a problem hiding this comment.
| # else we reprocess the last interval from the beginning. | |
| # otherwise we reprocess the last interval from the beginning. |
| # and we pick the default value. | ||
| value: >- | ||
| [[- if eq .cursor.pagination_finished "true" -]] | ||
| # look-back time should only be applied for fresh pagination intervals |
There was a problem hiding this comment.
If you want to include comments in the Go template, you will need to use the template comment syntax.
| # look-back time should only be applied for fresh pagination intervals | |
| [[- /* look-back time should only be applied for fresh pagination intervals */ -]] |
What is the precise definition of "fresh" here? I can see two possible interpretations; one matches the behaviour here (not started but after one that has completed), the other doesn't (never started). The immediately obvious interpretation is the second one, so I think this needs to be clearer.
There was a problem hiding this comment.
By fresh pagination cycle I meant a completely new pagination cycle that occurs on every new interval not between pages during a particular interval, I'll make it a bit clearer
|
@efd6, addressed all the comments |
| @@ -1,4 +1,9 @@ | |||
| # newer versions go on top | |||
| - version: "1.25.2" | |||
| changes: | |||
| - description: Added pagination and cursor fix to mirror the logic for a similar google_workspace scenario in pull - https://github.com/elastic/beats/pull/34274. | |||
There was a problem hiding this comment.
| - description: Added pagination and cursor fix to mirror the logic for a similar google_workspace scenario in pull - https://github.com/elastic/beats/pull/34274. | |
| - description: Fixed pagination and cursor handling. |
(users who want to know about pull requests will know how to find them, presenting them to other people is not really helpful, so instead of putting it here, put it in the commit message or just rely on the discussion in this PR — sorry, I missed this on the first round)
💚 Build Succeeded
History
cc @ShourieG |
|
Type of change
Proposed commit message
Fixed cursor & pagination logic for "detections" data stream to mirror a similar logic in implemented for a google_workspace fix.
Please refer to this issue to see the details.
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots