provide an exclusive lock step in a pipeline to avoid concurrent builds

[jstrachan]

Summary

When performing a release via GitOps we don't want to allow concurrent builds; as odd things can happen due to concurrency. We can happily process PRs concurrently - but often for releases we want to mark those as running one at a time for a given repository + branch.

Eventually this may end up being a core part of tekton: tektoncd/pipeline#1305

Until then a workaround could be to write a leadership election step of sorts - a step which blocks until it grabs a lock for being the 1 allowed pipeline for a given string (the git URL + branch)

Steps to reproduce the behavior

Run 2 release pipelines on a staging/production environment and issues can occur

Expected behavior

We force a 2nd pipeline on a given environment release to block until the first one complets

Strawman design

We already expose the pod logs into each pod via /etc/podinfo via the downward API. Though we should maybe be more explicit and expose the current PipelineRun name into pipeline via the $PIPELINE_RUN environment variable.

We could create a new step - say - jx step pipeline lock which does this:

• grab the current pod'sPipelineRun name and the repository owner/repo/branch
• find the SourceRepository for the owner/repo (we could expose this as a label/env var too to simplify the code)
• use an annotation key of jenkins.io./lock/owner/repo/branch on the SourceRepository
• lookup the annotation for the key.
• if empty, set the value to the PipelineRun name
• if the update works and the value of the updated PipelineRun annotation has the current PipelineRun name, the current pod is the leader and the step can terminate.
• if not the value of the key is the leaderPipelineRun resource to watch. The step then watches the leaderPipelineRun until its complete/deleted and all pods which have the label tekton.dev/pipelineRun=leaderPipelineRun complete/fail
• once there is no running leaderPipelineRun or pods then its time to try become the leader again, so try update the SourceRepository again
• repeat until some timeout and then fail?

[msvticket]

After getting the urgent issue (#5281) fixed with my PR I continued to think in a similar direction. The principal difference is that your solution seems to put the lock on the whole pipeline run while I was thinking of having the lock on a particular step (so rather TaskRun). The reasoning beaning that there might be time consuming steps that can be concurrent. Both varieties would be useful though.

[msvticket]

There are also cases where the order of PipelineRuns are of importance so a lock with FIFO behaviour would be preferable. Maybe solve by storing a queue of runs in the SourceRepository? But then you need to take into consideration that the pod first in the queue might die while waiting.

[jenkins-x-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle stale

[jenkins-x-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle rotten

[msvticket]

/remove-lifecycle rotten

[jenkins-x-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle stale

[jenkins-x-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle rotten

[jenkins-x-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://jenkins-x.io/community.
/close

[jenkins-x-bot]

@jenkins-x-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://jenkins-x.io/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository.

[keskad]

That would be nice to actually limit concurrency for specific jobs or even stages to avoid overlapping, especially, when external services are used.

[ankitm123]

I think we want to solve it upstream in tekton, instead of jx (There's a TEP for that: tektoncd/experimental#699). We have something basic in lighthouse using max_concurrency I believe (https://github.com/jenkins-x/lighthouse/blob/main/docs/trigger/github-com-jenkins-x-lighthouse-pkg-config-job.md)