Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AWS S3] Introduce start timestamp and ignore older timespan to AWS S3 based integrations #12645

Merged
merged 2 commits into from
Mar 4, 2025
Merged

[AWS S3] Introduce start timestamp and ignore older timespan to AWS S3 based integrations #12645

merged 2 commits into from
Mar 4, 2025

Conversation

Kavindu-Dodan
Copy link
Contributor

@Kavindu-Dodan Kavindu-Dodan commented Feb 6, 2025 •
edited
Loading

Proposed commit message

Introduce Ignore Older Timespan and Start Timestamp properties to integrations backed by AWS S3 input,

  • Ignore Older Timespan: Accepts a timespan in which entries are accepted for processing
  • Start Timestamp: Accepts a timestamp from which objects are accepted for processing

Configuring these properties allows S3 input to efficiently manage its internal registry. For example, setting Ignore Older Timespan to 2h makes the S3 input registry only track entries within the last 2 hours. Once entries are beyond the timespan, input can remove them from the registry, thus reducing memory consumption.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

Screenshots

Configuration rendered (Title matching existing format)

image

@Kavindu-Dodan Kavindu-Dodan force-pushed the feat/s3-start-and-ignore-older-timestamps branch 3 times, most recently from 888b269 to 2540178 Compare February 7, 2025 18:02
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Feb 7, 2025 •
edited
Loading

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@Kavindu-Dodan Kavindu-Dodan added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team [elastic/obs-ds-hosted-services] label Feb 7, 2025
@Kavindu-Dodan Kavindu-Dodan marked this pull request as ready for review February 7, 2025 19:38
@Kavindu-Dodan Kavindu-Dodan requested review from a team as code owners February 7, 2025 19:38
@Kavindu-Dodan Kavindu-Dodan changed the title [AWS S3] Introduce start timestamp and ignore older duration to AWS S3 based integrations [AWS S3] Introduce start timestamp and ignore older timespan to AWS S3 based integrations Feb 7, 2025
@MichaelKatsoulis MichaelKatsoulis self-requested a review February 10, 2025 16:01
@agithomas
Copy link
Contributor

agithomas commented Feb 11, 2025 •
edited
Loading

@Kavindu-Dodan , could you please consider backporting the beats changes so that we can bump up the package versions only to 8.16.5 instead of 8.18.0?

cc @zmoog

zmoog
zmoog reviewed Feb 11, 2025
View reviewed changes
Copy link
Contributor

@zmoog zmoog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should consider a backporting the Beats change to keep the min stack version in the 8.16.x series.

@Kavindu-Dodan
Copy link
Contributor Author

Kavindu-Dodan commented Feb 14, 2025 •
edited
Loading

Pending backports to 8.16.x 1 (next - 8.16.5) & 8.17.x 2 (next - 8.17.3) tracks. I will update this PR accordingly

Footnotes

  1. https://github.com/elastic/beats/pull/42716 ↩

  2. https://github.com/elastic/beats/pull/42717 ↩

@Kavindu-Dodan Kavindu-Dodan force-pushed the feat/s3-start-and-ignore-older-timestamps branch from 2540178 to 58cdb90 Compare February 14, 2025 21:23
@Kavindu-Dodan Kavindu-Dodan force-pushed the feat/s3-start-and-ignore-older-timestamps branch 3 times, most recently from 9fc363c to 93ea04f Compare February 19, 2025 21:16
@agithomas
Copy link
Contributor

@Kavindu-Dodan , we have a great illustration of this feature's behaviour as part of the description of this issue.

Do you think that it will be good to include this illustration in AWS documentation? Reference: Azure Integration I think, if we include this illustration, it would avoid misconfiguration of this setting.

@andrewkroh, could you please share your opinion - if it is a good idea to include the illustration as part of the AWS documentation?

The changes look good to me. As discussed, kindly merge the changes after the 8.16.5 is available. Also, let us get the approval from the security-integrations team.

@Kavindu-Dodan
Copy link
Contributor Author

Kavindu-Dodan commented Feb 20, 2025 •
edited
Loading

@agithomas good point on the documentation. I thought of adding some diagrams, but when checking the current AWS integration documentation, I couldn't find a suitable place to add the details. So I went with detailed descriptions for the integration input fields instead.

Anyway, let me know if there's better placement for detailed documentation.

@Kavindu-Dodan Kavindu-Dodan force-pushed the feat/s3-start-and-ignore-older-timestamps branch from 93ea04f to c7c60dc Compare February 20, 2025 17:53
andrewkroh
andrewkroh reviewed Feb 20, 2025
View reviewed changes
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andrewkroh, could you please share your opinion - if it is a good idea to include the illustration as part of the AWS documentation?

Personally, I don't think they are necessary to convey the meaning of the configuration options. If we need to provide additional context, then I would suggest linking to the reference docs like, "See the [input reference documentation] for more details." And add images in the reference docs.

@agithomas
Copy link
Contributor

If we need to provide additional context, then I would suggest linking to the reference docs like, "See the [input reference documentation] for more details." And add images in the reference docs.

Thanks @andrewkroh for your inputs. Yes, i agree that it would be the best approach.

@Kavindu-Dodan , it need not be part of this PR, it could be a separate PR by including the documentation team as reviewers.

@Kavindu-Dodan Kavindu-Dodan force-pushed the feat/s3-start-and-ignore-older-timestamps branch 2 times, most recently from 1b23f2f to 241160f Compare February 21, 2025 15:33
agithomas
agithomas approved these changes Feb 24, 2025
View reviewed changes
Copy link
Contributor

@agithomas agithomas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Kavindu-Dodan Kavindu-Dodan marked this pull request as draft February 24, 2025 16:08
@Kavindu-Dodan
Copy link
Contributor Author

Marking as draft and avoiding merging till the First week of March 2025

@Kavindu-Dodan Kavindu-Dodan force-pushed the feat/s3-start-and-ignore-older-timestamps branch from 50451df to 05fb24d Compare March 3, 2025 23:10
@Kavindu-Dodan
Copy link
Contributor Author

/test

@Kavindu-Dodan Kavindu-Dodan marked this pull request as ready for review March 4, 2025 19:47
@Kavindu-Dodan Kavindu-Dodan force-pushed the feat/s3-start-and-ignore-older-timestamps branch from 05fb24d to 4f790da Compare March 4, 2025 20:02
Kavindu-Dodan
Kavindu-Dodan commented Mar 4, 2025
View reviewed changes
Copy link
Contributor Author

@Kavindu-Dodan Kavindu-Dodan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merging today as 8.16.5 was released and available - https://www.elastic.co/blog/elastic-stack-8-16-5-released

@elasticmachine
Copy link

💚 Build Succeeded

History

@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
94.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@Kavindu-Dodan Kavindu-Dodan merged commit d682f74 into elastic:main Mar 4, 2025
7 checks passed
@elastic-vault-github-plugin-prod
Copy link

Package amazon_security_lake - 2.5.0 containing this change is available at https://epr.elastic.co/package/amazon_security_lake/2.5.0/

@elastic-vault-github-plugin-prod
Copy link

Package aws - 2.42.0 containing this change is available at https://epr.elastic.co/package/aws/2.42.0/

@elastic-vault-github-plugin-prod
Copy link

Package aws_bedrock - 1.2.0 containing this change is available at https://epr.elastic.co/package/aws_bedrock/1.2.0/

@elastic-vault-github-plugin-prod
Copy link

Package aws_logs - 1.8.0 containing this change is available at https://epr.elastic.co/package/aws_logs/1.8.0/

@elastic-vault-github-plugin-prod
Copy link

Package canva - 0.6.0 containing this change is available at https://epr.elastic.co/package/canva/0.6.0/

@elastic-vault-github-plugin-prod
Copy link

Package carbon_black_cloud - 2.9.0 containing this change is available at https://epr.elastic.co/package/carbon_black_cloud/2.9.0/

@elastic-vault-github-plugin-prod
Copy link

Package cisco_umbrella - 1.30.0 containing this change is available at https://epr.elastic.co/package/cisco_umbrella/1.30.0/

@elastic-vault-github-plugin-prod
Copy link

Package cloudflare_logpush - 1.35.0 containing this change is available at https://epr.elastic.co/package/cloudflare_logpush/1.35.0/

@elastic-vault-github-plugin-prod
Copy link

Package f5_bigip - 1.27.0 containing this change is available at https://epr.elastic.co/package/f5_bigip/1.27.0/

@elastic-vault-github-plugin-prod
Copy link

Package imperva_cloud_waf - 1.9.0 containing this change is available at https://epr.elastic.co/package/imperva_cloud_waf/1.9.0/

@elastic-vault-github-plugin-prod
Copy link

Package jamf_protect - 2.11.0 containing this change is available at https://epr.elastic.co/package/jamf_protect/2.11.0/

@elastic-vault-github-plugin-prod
Copy link

Package sentinel_one_cloud_funnel - 1.10.0 containing this change is available at https://epr.elastic.co/package/sentinel_one_cloud_funnel/1.10.0/

@elastic-vault-github-plugin-prod
Copy link

Package servicenow - 0.11.0 containing this change is available at https://epr.elastic.co/package/servicenow/0.11.0/

@elastic-vault-github-plugin-prod
Copy link

Package sublime_security - 1.7.0 containing this change is available at https://epr.elastic.co/package/sublime_security/1.7.0/

@elastic-vault-github-plugin-prod
Copy link

Package symantec_endpoint_security - 1.9.0 containing this change is available at https://epr.elastic.co/package/symantec_endpoint_security/1.9.0/

@elastic-vault-github-plugin-prod
Copy link

Package tanium - 1.16.0 containing this change is available at https://epr.elastic.co/package/tanium/1.16.0/

@elastic-vault-github-plugin-prod
Copy link

Package trellix_edr_cloud - 1.8.0 containing this change is available at https://epr.elastic.co/package/trellix_edr_cloud/1.8.0/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gizas gizas gizas approved these changes

@andrewkroh andrewkroh andrewkroh approved these changes

@agithomas agithomas agithomas approved these changes

@MichaelKatsoulis MichaelKatsoulis Awaiting requested review from MichaelKatsoulis

@zmoog zmoog Awaiting requested review from zmoog

Assignees
No one assigned
Labels
Team:obs-ds-hosted-services Label for the Observability Hosted Services team [elastic/obs-ds-hosted-services]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[AWS] Introduce ignore_older & start_timestamp
6 participants
@Kavindu-Dodan @agithomas @elasticmachine @zmoog @gizas @andrewkroh