diff --git a/teps/0157-retention-policy-results.md b/teps/0157-retention-policy-results.md new file mode 100644 index 000000000..b2aed0445 --- /dev/null +++ b/teps/0157-retention-policy-results.md @@ -0,0 +1,226 @@ +--- +status: proposed +title: Retention Policy for Tekton Results +creation-date: '2024-07-17' +last-updated: '2024-07-17' +authors: +- '@khrm' +collaborators: [] +--- + +# TEP-0157: Tekton Results: Retention Policy for older Results and Records + + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) + - [Use Cases](#use-cases) + - [Requirements](#requirements) +- [Proposal](#proposal) + - [Notes and Caveats](#notes-and-caveats) +- [Design Details](#design-details) +- [Design Evaluation](#design-evaluation) + - [Reusability](#reusability) + - [Simplicity](#simplicity) + - [Flexibility](#flexibility) + - [User Experience](#user-experience) + - [Performance](#performance) + - [Risks and Mitigations](#risks-and-mitigations) + - [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Implementation Plan](#implementation-plan) + - [Test Plan](#test-plan) + - [Infrastructure Needed](#infrastructure-needed) + - [Upgrade and Migration Strategy](#upgrade-and-migration-strategy) + - [Implementation Pull Requests](#implementation-pull-requests) +- [References](#references) + + +## Summary +Tekton Results stores Pipelineruns, TaskRuns, Events and Logs indefinitely. +This proposed adding a retention policy feature for removing older Result and their associated records +alongwith request to delete logs. + +## Motivation +Storing older results and records indefinitely leads to wastage of storage resources +and degradation of DB performance. Sometime we don't require some pipelines to be deleted from +archives. + +### Goals +- Ability to define retention period for the Results at cluster level. All records and results past that period should be deleted. +- Ability to filter PipelineRuns when setting retention policy. +- A way to delete associated logs also from s3 buckets, gcs buckets or PVC. + +### Non-Goals + + + +### Use Cases +- User can specify a global policy for all the results. All records and logs falling under results satisfying pruning condition will be deleted. +- User can filter results based on cel expression and result Summary expression. All the associated records will be deleted. + +### Requirements +- For all results satisfying delete conditions, following things need to be deleted: +* Results +* Records for PipelineRun and TaskRun +* Records for EventLog +* Deletion of associated logs from s3 bucket, gcs bucket or PVC. EventLog Records should also be deleted. + +## Proposal +A pruner will run which will spin up job at specified interval based on configmap `config-results-retention-policy` given ttl and cel expressions. + +### Notes and Caveats + + +## Design Details +Configmap will have following structure: +` +runAt: 5 4 * * 7 # Specify when to run job so that we can bring DB in maintenance mode if needed. +max-retention: 2880h # Max Retention duration of 120 days. +filters: +- expr: summary.status == FAILURE # When there's failure in PipelineRun + ttl: 700h +- expr: parent == dev # When parent/namespace is dev + ttl: 300h +` + + +## Design Evaluation + + +### Reusability + + + +### Simplicity + + + +### Flexibility + + + +### Conformance + + + +### User Experience + + + +### Performance +This improves the peformance of DB by deleting superfluous results and their associated datas. + +### Risks and Mitigations + + + +### Drawbacks + + + +## Alternatives + + +## Implementation Plan + + + + +### Test Plan + +- We will add a Integration tests like we have for Logging in GCS storage and other scenarios. + +### Infrastructure Needed + + + +### Upgrade and Migration Strategy + + + +### Implementation Pull Requests + + +## References + + diff --git a/teps/README.md b/teps/README.md index 6c2381323..fd4743609 100644 --- a/teps/README.md +++ b/teps/README.md @@ -146,3 +146,4 @@ This is the complete list of Tekton TEPs: |[TEP-0154](0154-concise-remote-resolver-syntax.md) | Concise Remote Resolver Syntax | implementable | 2024-03-21 | |[TEP-0155](0155-store-pipeline-events-in-db.md) | Store Pipeline Events in Tekton Results | proposed | 2024-04-19 | |[TEP-0156](0156-whenexpressions-in-step.md) | WhenExpressions in Steps | proposed | 2024-04-15 | +|[TEP-0157](0157-retention-policy-results.md) | Retention Policy for Tekton Results | proposed | 2024-07-17 |