Skip to content

Commit

Permalink
fix: prevent concurrent build to edit the same namespace
Browse files Browse the repository at this point in the history
This PR prevents builds to edit the same namespace at the same time.

Such a behavior could lead to a namespace to be accidentally wiped out.

Only `jx step helm apply` is locked.

A ConfigMap `jx-lock-{namespace}` is used as a lock. No other build can run while this configmap exists. Waiting builds can edit the data of the ConfigMap in order to be the next one to run. If a build sees that the locking or waiting build is "higher", it will fail. When the build finished, the ConfigMap is removed. A waiting build can also remove the ConfigMap if the lokcing pod has finished.

The algorithm is approximately:
```
    Label: CREATE
    try to create the ConfigMap
    if it succeeds:
        return
    Label: READ
    get the ConfigMap and the locking Pod
    if the locking Pod has finished
        remove the ConfigMap
        goto CREATE
    if the ConfigMap references a "higher" build
        fail
    if the ConfigMap references a "lower" build
        update the ConfigMap
    wait for the ConfigMap or the Pod to be updated
        if the ConfigMap is delete
            goto CREATE
        if the ConfigMap references a different build
            goto READ
        if the Pod has finished
            goto CREATE
```

fixes #6167

Signed-off-by: Aurélien Lambert <[email protected]>
  • Loading branch information
aure-olli authored and jenkins-x-bot committed Apr 3, 2020
1 parent 5a162e8 commit 90e4b80
Show file tree
Hide file tree
Showing 3 changed files with 1,394 additions and 0 deletions.
6 changes: 6 additions & 0 deletions pkg/cmd/step/helm/step_helm_apply.go
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,12 @@ func (o *StepHelmApplyOptions) Run() error {
defer os.RemoveAll(rootTmpDir)
}

if release, err := kube.AcquireBuildLock(kubeClient, devNs, ns); err != nil {
return errors.Wrapf(err, "fail to acquire the lock")
} else {
defer release()
}

// lets use the same child dir name as the original as helm is quite particular about the name of the directory it runs from
_, name := filepath.Split(dir)
if name == "" {
Expand Down
Loading

0 comments on commit 90e4b80

Please sign in to comment.