v0.25.0

v0.25.0 (2025-02-28)

Full Changelog

Added

• SUP-3347: Enable Pagination of Scheduled Jobs #515 (@tomowatt)

Internal

• Adds the information on cluster's UUID in a structured form + minor improvements #531 (@karensawrey)

Dependencies

• Bump buildkite-agent to v3.93.1 #529 (@DrJosh9000)

Agent Changelog

v3.93.1 (2025-02-27)

Full Changelog

Added

• Set env when job cancelled for hooks #3213 (@sj26)

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.25.0
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.25.0
Digest: sha256:f897db57354f239fdd7a3f67c7ae8aef56b33af3587529432ccd8bfd5e6dbad4

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.25.0
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.25.0
Digest: sha256:bf1e138c19318522c5db62b0f62f68fc900194cd9b97ae0017cfab3caa48e51e

Agent

Image: ghcr.io/buildkite/agent:3.93.1
Digest: sha256:03a6fb7b4d98bbf479e687317b42886d2e2a51e0296b669b9fa53c7cca5cc0fb

v0.24.1

v0.24.1 (2025-02-26)

Full Changelog

Fixed

• Fix labels for pod eviction metric by @petetomasik in #525
• Check that CheckoutParams.GitCredentialsSecret is not nil by @petetomasik in #527

Dependencies

• Bump github.com/prometheus/client_golang from 1.20.5 to 1.21.0 by @dependabot in #524
• Bump github.com/google/go-cmp from 0.6.0 to 0.7.0 by @dependabot in #523

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.24.1
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.24.1
Digest: sha256:1f888b75050232efa0f173692ba37c018e1d3ed2b83f93876887b202d9d9ae5a

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.24.1
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.24.1
Digest: sha256:d6cea6214d9d8b5d8b4d4509eb06095d0a62841bd2889db1804eb114ef4820e6

Agent

Image: ghcr.io/buildkite/agent:3.92.1
Digest: sha256:2a1be0f1907b4117bd91f3c89ed849248355adcff9f17f52be3b6693c173c97f

v0.24.0

v0.24.0 (2025-02-24)

Full Changelog

Note

The controller will now add .spec.activeDeadlineSeconds to every Kubernetes Job it creates. This value defaults to 21600 seconds (6 hours) and can be overwritten directly in the controller's configuration or on a per-job basis with the kubernetes plugin. This active deadline will ensure that no Kubernetes Job is allowed to run indefinitely.

Added

• SUP-3258 - Implement .spec.job.activeDeadlineSeconds by @petetomasik in #497
• [SUP-3361] Controller to handle queue paused events and prevent processing jobs by @nsuma8989 in #507
• Annotate pods with job priority number by @mikehw in #519
• Add config option to disable command modification check by @DrJosh9000 in #449

Fixed

• Fix unusable image pull policy options by @DrJosh9000 in #508

Changed

• Account for reason DeadlineExceeded in checkFinishedWithoutPod() by @petetomasik in #517

Internal

• Update CODEOWNERS by @karensawrey in #514
• [SUP-3361] Update readme with information about queue pause config by @nsuma8989 in #521

Dependencies

• Bump the k8s group with 4 updates by @dependabot in #509
• Bump github.com/spf13/pflag from 1.0.5 to 1.0.6 by @dependabot in #493
• Bump github.com/Khan/genqlient from 0.7.0 to 0.8.0 by @petetomasik in #516
• Bump github.com/buildkite/agent/v3 from 3.91.0 to 3.92.1 by @dependabot in #510
• Bump github.com/jedib0t/go-pretty/v6 from 6.6.5 to 6.6.6 by @dependabot in #511
• Bump github.com/go-playground/validator/v10 from 10.24.0 to 10.25.0 by @dependabot in #512
• Bump github.com/spf13/cobra from 1.8.1 to 1.9.1 by @dependabot in #513

New Contributors

• @mikehw made their first contribution in #519

Agent Changelog

v3.92.1 (2025-02-13)

Full Changelog

Removed

• Revert "Ensure the log streamer respects forced shutdown of the agent" #3191 (@wolfeidau)
• Revert "Fix data race on exitImmediately" #3190 (@wolfeidau)

Dependabot

• The usual updates: #3188, #3185 (@dependabot[bot])

[!NOTE]
Reverted #3180 and #3187 as this change introduced a bug which resulted in truncated log output. Will re-think this fix and push it out again in another release after we do some more testing.

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.24.0
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.24.0
Digest: sha256:ed9257b4454d7d9278ea70955a0dce0b5aacf9915af310d3f3a4a7a3646feec4

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.24.0
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.24.0
Digest: sha256:1aadf37418c702e5f4755594bb758a6c3613ed5f382aac5d7ebfc1df6c5a5a3a

Agent

Image: ghcr.io/buildkite/agent:3.92.1
Digest: sha256:2a1be0f1907b4117bd91f3c89ed849248355adcff9f17f52be3b6693c173c97f

v0.23.0

v0.23.0 (2025-02-12)

Full Changelog

Added

• SUP-3261: Use Protobuf encoding for API responses by @petetomasik in #501
• Allow configurable QPS, Burst settings with Kubernetes client, increase defaults by @petetomasik in #489
• Add note about DSA keys deprecation, change example to RSA by @petetomasik in #471
• SUP-3283: Enable configuring GraphQL Result Limits by @tomowatt in #499
• enable name overrides by @alexr56k in #490
• SUP-3349: Apply sane resource requests to imagecheck containers by @petetomasik in #504

Fixed

• Fix resources and port parsing for non-strings by @mikekap in #495

Changed

• Update CODEOWNERS - adding Support Engineers as CODEOWNERS by @karensawrey in #494
• Update CODEOWNERS - remove obsolete team by @karensawrey in #496
• SUP-3216 - Remove checkout container from podSpec, if skipped by @petetomasik in #502

Dependencies

• Bump sigs.k8s.io/controller-runtime from 0.20.0 to 0.20.1 in the k8s group by @dependabot in #485
• Bump github.com/buildkite/agent/v3 from 3.90.0 to 3.91.0 by @dependabot in #492

New Contributors

• @petetomasik made their first contribution in #471
• @karensawrey made their first contribution in #494
• @mikekap made their first contribution in #495
• @tomowatt made their first contribution in #499
• @alexr56k made their first contribution in #490

Agent Changelog

v3.91.0 (2025-01-28)

Full Changelog

Changed

• Jitter within ping, status, log loops #3164 (@DrJosh9000)

Fixed

• Roko v1.3.1 #3157 (@moskyb)
• Better plugin checkout logging #3166 (@DrJosh9000)

Internal

• Add /.buildkite dir for Dockerfile updates #3162 (@DrJosh9000)

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.23.0
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.23.0
Digest: sha256:d6194df0439771a929b8bf133a1689a4f956a32a9b0b369694a516de4230e876

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.23.0
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.23.0
Digest: sha256:6e5c269e1925b2ec0db2aaa5f41ea1b5a28ed8167b60be1f900d54d8c327579e

Agent

Image: ghcr.io/buildkite/agent:3.91.0
Digest: sha256:f46a442f89dec73a979d9ecfd0a7287dde82fefe0e3603aeff3c8324e98373ca

v0.22.0

v0.22.0 (2025-01-22)

Full Changelog

Fixed

• Fix negation of no-submodules option #478 (@DrJosh9000)

Added

• Include PR info into release version #384 (@artem-zinnatullin)

Changed

• Bump github.com/buildkite/agent/v3 from 3.88.0 to 3.90.0 #468 (@dependabot[bot])

Dependencies

• Bump the k8s group across 1 directory with 4 updates #474 (@dependabot[bot])
• Bump github.com/buildkite/roko from 1.2.0 to 1.3.1 #473 (@dependabot[bot])
• Bump github.com/go-playground/validator/v10 from 10.23.0 to 10.24.0 #470 (@dependabot[bot])

Agent Changelog

v3.90.0 (2025-01-10)

Full Changelog

Changed

• Use exponential in acquire-job mode when job acquisition fails #3153 (@moskyb)

Fixed

• Fix nil pointer deref in certain Kubernetes environments #3150 (@DrJosh9000)

v3.89.0 (2025-01-06)

Full Changelog

Added

• Adding support for Additional Hooks Paths #3124 (@CerealBoy)

Internal

• Bump the container-images group across 5 directories with 2 updates #3143 (@dependabot[bot])
• Update golang.org/x/net #3140 (@yob)

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.22.0
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.22.0
Digest: sha256:157b1e8b15894ca7d9228b7013084ddfa59d2e7c82571b098623a61d87a2f92f

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.22.0
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.22.0
Digest: sha256:8bc4529b751fc5c5269b7da9134aa339a20e70947511c90344e2aa03e3fa65a0

Agent

Image: ghcr.io/buildkite/agent:3.90.0
Digest: sha256:ad8fa8867f8de3c5d34511245bcdd9645297fdc42fb27cc8f211732448931099

v0.21.0

v0.21.0 (2024-12-19)

Full Changelog

Note

You can now specify a default image pull policy for containers that don't specify one (--default-image-pull-policy and associated config), and a default image pull policy for the init containers the stack adds to pre-check images can be pulled (--default-image-check-pull-policy).
Image pull policies configured on containers are also no longer unconditionally overridden, for instance you can now use imagePullPolicy: Never and the image should not be pulled (the job may still fail if the image is not present).
These changes are unlikely to cause issues when upgrading, but please let us know of any issues you encounter.

Note

You can now use PodSpecPatch to modify the commands and args of some types of containers (command containers, sidecars, and new containers added via patch). For checkout containers, please continue either configuring a checkout hook, or skip the checkout container (checkout: skip: true in the kubernetes plugin, or default-checkout-params: skip: true in the controller config).

Fixed

• Allow patching some container commands/args #461 (@DrJosh9000)
• Allow running command containers as non-root user with EBS backed (and other) workspace volumes #453 (@patrobinson)
• Image check improvements #459 (@DrJosh9000)
• Schedule jobs in priority order #451 (@DrJosh9000)

Changed

• Bump github.com/buildkite/agent/v3 from 3.87.1 to 3.88.0 #462 (@dependabot[bot])
• Add container image to example under PodSpecPatch #454 (@DrJosh9000)

Dependency bumps

• Bump the k8s group with 3 updates #457 (@dependabot[bot])
• Bump github.com/jedib0t/go-pretty/v6 from 6.6.3 to 6.6.5 #458 (@dependabot[bot])

Agent Changelog

v3.88.0 (2024-12-18)

Full Changelog

Changed

• Prefix fatal error message with 'buildkite-agent:' #3135 (@jordandcarter)
• Notify when host and bootstrap agent paths mismatch #3123 (@jordandcarter)

Fixed

• Enable process debug logging #3134 (@patrobinson)
• Ignore empty submodule clone configs #3122 (@DrJosh9000)
• fix: allow for empty files on hook check #3117 (@nzspambot)
• Parse more standalone $ cases as literal $s and not variable expansions:
  • Bump github.com/buildkite/go-pipeline from 0.13.2 to 0.13.3 #3137 (@dependabot[bot])
  • Bump github.com/buildkite/interpolate from 0.1.4 to 0.1.5 #3138 (@dependabot[bot])

Dependabot

• #3136, #3127, #3129, #3128, #3130, #3132, #3131, #3133, #3125, #3119, #3120, #3121, #3116, #3115 (@dependabot[bot])

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.21.0
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.21.0
Digest: sha256:430f2e916225036a81318ad7c6261a78791b10fe4e28fdfee0d3b318628c1903

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.21.0
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.21.0
Digest: sha256:e86704ae40c97885a9a6a6d020f7fcd9d2299bd33d5a36ada7ad5988d4ba4dcc

Agent

Image: ghcr.io/buildkite/agent:3.88.0
Digest: sha256:3099e07fe3e219d44998a887145e7409659e3ec969410b5dd54427c5865a5e1f

v0.20.2

v0.20.2 (2024-12-16)

Full Changelog

This is the first release to AWS Public ECR! All the changes are in the pipeline or release process.
Otherwise, this release is functionally the same as v0.20.1.

Internal

• Fix release.sh #456 (@DrJosh9000)
• Fix deploy.sh #455 (@DrJosh9000)
• Push images to Public ECR #450 (@DrJosh9000)
• Move Go cache hostpaths to /var/cache #446 (@DrJosh9000)

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.20.2
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.20.2
Digest: sha256:2be056f01245acd35ebe4c487a69f24c9333e3af1195b0c8bdcf9a474a822c60

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.20.2
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.20.2
Digest: sha256:4b025a17f4f7907aefd09eb73fbe351ab5e051aa69c05acc12eb78a8e5a38cc1

Agent

Image: ghcr.io/buildkite/agent:3.87.1
Digest: sha256:39ed89f3e3eee32dd6415e27e90654c6969c96a82d90e8ad8b84cbed031b816e

v0.20.1

v0.20.1 (2024-12-10)

Full Changelog

Fixed

• Submodule checkout failure when checkout configuration is provided #444 (@rianmcguire)

Changed

• Bump sigs.k8s.io/controller-runtime from 0.19.2 to 0.19.3 in the k8s group #443 (@dependabot[bot])

Images

Helm chart

Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.20.1
Digest: sha256:57e4ae64cd4d522a3603b37324307bd2b2473c6d814a9b3467cc5e6080f6e519

Controller

Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.20.1
Digest: sha256:e371ddc75d01c071f36f98ac88b0d506501dc578df77d3e39ebde261ad1b9369

Agent

Image: ghcr.io/buildkite/agent:3.87.1
Digest: sha256:39ed89f3e3eee32dd6415e27e90654c6969c96a82d90e8ad8b84cbed031b816e

v0.20.0

v0.20.0 (2024-12-05)

Full Changelog

Added

• Add job watcher #442 (@DrJosh9000)
• Check for init container failures and InvalidImageName #441 (@DrJosh9000)
• Add job UUID to limiter debug logs in informer callbacks #439 (@DrJosh9000)
• Add Prometheus metrics endpoint #419 (@DrJosh9000)
• Catch duplicate jobs in scheduler #438 (@DrJosh9000)

Fixed

• deduper: Track job existence instead of running state #440 (@DrJosh9000)
• Refine informer usage in deduper #436 (@DrJosh9000)

Dependency updates

• Bump github.com/stretchr/testify from 1.9.0 to 1.10.0 #435 (@dependabot[bot])
• Bump the k8s group with 4 updates #434 (@dependabot[bot])
• Bump github.com/buildkite/agent/v3 from 3.87.0 to 3.87.1 #437 (@dependabot[bot])

Images

Helm chart

Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.20.0
Digest: sha256:dc7e4836bd767753437d8d1db57144f86b6b821f95e0c93fc933828dc21d2c1c

Controller

Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.20.0
Digest: sha256:2e2f1afd2fafd81132a2633e20cd5f650df3dca8ade8bf6b76e58bf8ecf25868

Agent

Image: ghcr.io/buildkite/agent:3.87.1
Digest: sha256:39ed89f3e3eee32dd6415e27e90654c6969c96a82d90e8ad8b84cbed031b816e

v0.19.3

v0.19.3 (2024-11-24)

Full Changelog

Fixed

• Fix limiter token tracking (again) #432 (@DrJosh9000)

Changed

• Match tags in pod watcher #431 (@DrJosh9000)

Images

Helm chart

Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.19.3
Digest: sha256:32806855ff384d126de6e2ccd98303994f37e940f34e01637e40e9877c2c5c28

Controller

Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.19.3
Digest: sha256:6e1508c9ef6b6975984b83995f9b20998a5e9ae226e8a6b1dcd30065096c6341

Agent

Image: ghcr.io/buildkite/agent:3.87.0
Digest: sha256:c65b58556ba3798974df83ae33d425b2e39a7adebddd07835804720dbe8945d3