race conditions when having more than one pipeline of the same branch

[gerardcl]

We have detected at least two situations where one can have at least two pipelines at the same time running (never with same start time):

• pushing two consecutive commits with close git push time or while there was already one running:
  

• pushing a commit and afterwards creating a PR:
  

Depending on the timings, errors can happen in any task:

• task ods-start:

step-ods-start
+ ods-start -project=myproject -environment= -version= -git-full-ref=refs/heads/test-multiple-commits-ods-bug -git-ref-spec= -url=https://bb.example.com/scm/myproject/myproject-acomponent.git -pr-key=0 -pr-base= -http-proxy= -https-proxy= -no-proxy= -ssl-verify=true -submodules=true -depth=1 -pipeline-run-name=poc-ast-test-multiple-commits-ods-bug-wctxc
[INFO] Cleaning checkout directory ...
[INFO] Checking out https://bb.example.com/scm/myproject/myproject-acomponent.git@refs/heads/test-multiple-commits-ods-bug into /workspace/source ...
[ERROR] {"level":"info","ts":1641973524.5227394,"caller":"git/git.go:169","msg":"Successfully cloned https://bb.example.com/scm/myproject/myproject-acomponent.git @ f76f8cf66880e6409668953e34c0d175eba1522b (grafted, HEAD, origin/test-multiple-commits-ods-bug) in path /workspace/source"}
{"level":"error","ts":1641973524.5810995,"caller":"git/git.go:54","msg":"Error running git [submodule update --recursive --init --depth=1]: exit status 1\nfatal: not a git repository (or any parent up to mount point /workspace)\nStopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).\nfatal: /usr/libexec/git-core/git-submodule cannot be used without a working tree.\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\t/opt/app-root/src/go/src/github.com/tektoncd/pipeline-0.24.0/pkg/git/git.go:54\ngithub.com/tektoncd/pipeline/pkg/git.submoduleFetch\n\t/opt/app-root/src/go/src/github.com/tektoncd/pipeline-0.24.0/pkg/git/git.go:204\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\t/opt/app-root/src/go/src/github.com/tektoncd/pipeline-0.24.0/pkg/git/git.go:171\nmain.main\n\t/opt/app-root/src/go/src/github.com/tektoncd/pipeline-0.24.0/cmd/git-init/main.go:53\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:225"}
{"level":"fatal","ts":1641973524.581184,"caller":"git-init/main.go:54","msg":"Error fetching git repository: exit status 1","stacktrace":"main.main\n\t/opt/app-root/src/go/src/github.com/tektoncd/pipeline-0.24.0/cmd/git-init/main.go:54\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:225"}

2022/01/12 07:45:24 exit status 1

• task <tech>-build, e.g. sonar:

2022/01/12 08:31:54 scan failed: scanning failed: exit status 1, stderr: ERROR: Error during SonarQube Scanner execution
ERROR: Another SonarQube analysis is already in progress for this project
ERROR: 
ERROR: Re-run SonarQube Scanner using the -X switch to enable full debug logging.

• task deploy-helm:

Upgrading Helm release to acomponent-0.1.0+a71a414ad68ab8f57a92934a8d820e56fad8e129.tgz...
helm --namespace=myproject-dev secrets upgrade --wait --install --values=./chart/secrets.yaml --set=image.tag=a71a414ad68ab8f57a92934a8d820e56fad8e129 acomponent acomponent-0.1.0+a71a414ad68ab8f57a92934a8d820e56fad8e129.tgz
[helm-secrets] Decrypt skipped: ./chart/secrets.yaml
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
Error: plugin "secrets" exited with error

• task ods-finish:

[INFO] Uploading .ods/artifacts/xunit-reports/report.xml to Nexus repository ods-temporary-artifacts, group /myproject/myproject-acomponent/a2a63f6fdc27ba35654679bfc7996366f74b53d2/xunit-reports ...

2022/01/07 15:32:15 cannot upload artifacts of main repository: could not upload component: component not uploaded: 400 Bad Request

So, any race condition can happen when having more than one pipeline running at same time.

For now, usually the workaround is just being aware of this and afterwards re-run the pipeline in openshift.

Solutions:
1- no solution, but just awareness
2- queueing/semaphore mechanisms (PVC checks, file checks,...)
3- investigate what is already possible from tekton and/or openshift pipelines

New proposals, comments, etc., are welcome!

NOTE: this might also happen when not being only in same branch context

[michaelsauter]

Thanks for opening this.

I wrongly assumed that pipelines wouldn't start in parallel because they mount the same PVC. Seems this is not the case.

Solution (1) is the short-term solution but not one I am ready to accept.

Related to (2) and (3) I found:

• Concurrency limiter controller tektoncd/experimental#699
• Idea: Pipeline Mutexes tektoncd/pipeline#2828
• https://medium.com/ibm-garage/using-lease-resources-to-manage-concurrency-in-tekton-builds-344ba84df297