feat: upgrade cpu workers to ubuntu 24.04 generic-worker image #949
FirefoxCI (pull_request)
The task that creates all of the other tasks in the task graph
Details
View task in Taskcluster | View logs in Taskcluster | View task group in Taskcluster
Task Status
Started: 2024-12-17T15:16:40.114Z
Resolved: 2024-12-17T15:17:11.518Z
Task Execution Time: 31 seconds, 404 milliseconds
Task Status: completed
Reason Resolved: completed
RunId: 0
Artifacts
- public/actions.json
- public/docker-contexts/base.tar.gz
- public/docker-contexts/inference.tar.gz
- public/docker-contexts/test.tar.gz
- public/docker-contexts/toolchain-build.tar.gz
- public/docker-contexts/train.tar.gz
- public/fetch-content
- public/full-task-graph.json
- public/label-to-taskid.json
- public/logs/live_backing.log
- public/logs/live.log
- public/optimizations.log
- public/parameters.yml
- public/runnable-jobs.json
- public/run-task
- public/target-tasks.json
- public/task-graph.json
[taskcluster 2024-12-17 15:16:40.188Z] Task ID: Uyh4X8cxQdKwUWnzSOvJHQ
[taskcluster 2024-12-17 15:16:40.188Z] Worker ID: 2782865028234881919
[taskcluster 2024-12-17 15:16:40.188Z] Worker Group: us-central1-f
[taskcluster 2024-12-17 15:16:40.188Z] Worker Node Type: projects/887720501152/machineTypes/n2-standard-2
[taskcluster 2024-12-17 15:16:40.188Z] Worker Pool: translations-1/decision-gcp
[taskcluster 2024-12-17 15:16:40.188Z] Worker Version: 38.0.5
[taskcluster 2024-12-17 15:16:40.188Z] Public IP: 34.136.145.18
[taskcluster 2024-12-17 15:16:40.188Z] Hostname: translations-1-decision-gcp-msupjsosrawznt9eohkyvq
[taskcluster 2024-12-17 15:16:40.188Z] using cache "translations-level-1-checkouts-sparse-v2" -> /builds/worker/checkouts
559df9a599c4 - Started downloading
b57ab501a8f4 - Started downloading
3bb7dd3accd0 - Started downloading
559df9a599c4 - Downloaded in 0.124 seconds
fad8d951cdaf - Started downloading
fad8d951cdaf - Downloaded in 0.001 seconds
b57ab501a8f4 - Downloaded in 0.431 seconds
3bb7dd3accd0 - Downloaded in 1.333 seconds
Digest: sha256:8c57e30214c85625856479812e63212dc2873b1ffcd97467a7c93508cbb3075a
Status: Downloaded newer image for mozillareleases/taskgraph@sha256:8c57e30214c85625856479812e63212dc2873b1ffcd97467a7c93508cbb3075a
[taskcluster 2024-12-17 15:16:52.620Z] === Task Starting ===
[setup 2024-12-17T15:16:53.439Z] run-task started in /
[setup 2024-12-17T15:16:53.439Z] Invoked by command: --translations-checkout=/builds/worker/checkouts/src -- bash -cx cd /builds/worker/checkouts/src && ln -s /builds/worker/artifacts artifacts && taskgraph decision --pushlog-id='0' --pushdate='0' --project='translations' --owner='[email protected]' --level='1' --repository-type=git --tasks-for='github-pull-request' --base-repository='https://github.com/mozilla/translations' --base-ref='main' --base-rev='051db33d26fe6c2ca1e0e7940343295b9fbb004f' --head-repository='https://github.com/bhearsum/firefox-translations-training' --head-ref='cpu-only-upgrade' --head-rev='c6a36ed189c2103d3754975d4e8bea7abcb34a7f'
[setup 2024-12-17T15:16:53.440Z] Python version: 3.11.2
[vcs 2024-12-17T15:16:53.442Z] WARNING: vcs checkout path (/builds/worker/checkouts/src) not in cache or volume; performance will likely suffer
[setup 2024-12-17T15:16:53.442Z] running as worker:worker
[vcs 2024-12-17T15:16:53.442Z] executing ['git', 'config', '--global', '--add', 'safe.directory', '/builds/worker/checkouts/src']
[vcs 2024-12-17T15:16:53.444Z] executing ['git', 'clone', 'https://github.com/mozilla/translations', '/builds/worker/checkouts/src']
[vcs 2024-12-17T15:16:53.446Z] Cloning into '/builds/worker/checkouts/src'...
[vcs 2024-12-17T15:16:54.803Z] executing ['git', 'fetch', 'origin', 'main']
[vcs 2024-12-17T15:16:55.028Z] From https://github.com/mozilla/translations
[vcs 2024-12-17T15:16:55.029Z] * branch main -> FETCH_HEAD
[vcs 2024-12-17T15:16:55.036Z] executing ['git', 'checkout', 'main']
[vcs 2024-12-17T15:16:55.097Z] Already on 'main'
[vcs 2024-12-17T15:16:55.098Z] Your branch is up to date with 'origin/main'.
[vcs 2024-12-17T15:16:55.098Z] executing ['git', 'fetch', 'origin', '051db33d26fe6c2ca1e0e7940343295b9fbb004f']
[vcs 2024-12-17T15:16:55.306Z] From https://github.com/mozilla/translations
[vcs 2024-12-17T15:16:55.308Z] * branch 051db33d26fe6c2ca1e0e7940343295b9fbb004f -> FETCH_HEAD
[vcs 2024-12-17T15:16:55.315Z] executing ['git', 'fetch', '--no-tags', 'https://github.com/bhearsum/firefox-translations-training', 'cpu-only-upgrade']
[vcs 2024-12-17T15:16:55.604Z] From https://github.com/bhearsum/firefox-translations-training
[vcs 2024-12-17T15:16:55.604Z] * branch cpu-only-upgrade -> FETCH_HEAD
[vcs 2024-12-17T15:16:55.607Z] executing ['git', 'checkout', '-f', '-B', 'cpu-only-upgrade', 'c6a36ed189c2103d3754975d4e8bea7abcb34a7f']
[vcs 2024-12-17T15:16:55.614Z] Switched to a new branch 'cpu-only-upgrade'
[vcs 2024-12-17T15:16:55.615Z] executing ['git', 'submodule', 'init']
[vcs 2024-12-17T15:16:55.636Z] Submodule '3rd_party/browsermt-marian-dev' (https://github.com/browsermt/marian-dev) registered for path '3rd_party/browsermt-marian-dev'
[vcs 2024-12-17T15:16:55.637Z] Submodule 'extract-lex' (https://github.com/marian-nmt/extract-lex) registered for path '3rd_party/extract-lex'
[vcs 2024-12-17T15:16:55.638Z] Submodule 'fast_align' (https://github.com/clab/fast_align) registered for path '3rd_party/fast_align'
[vcs 2024-12-17T15:16:55.639Z] Submodule '3rd_party/kenlm' (https://github.com/kpu/kenlm) registered for path '3rd_party/kenlm'
[vcs 2024-12-17T15:16:55.639Z] Submodule '3rd_party/marian-dev' (https://github.com/marian-nmt/marian-dev) registered for path '3rd_party/marian-dev'
[vcs 2024-12-17T15:16:55.640Z] Submodule '3rd_party/preprocess' (https://github.com/kpu/preprocess.git) registered for path '3rd_party/preprocess'
[vcs 2024-12-17T15:16:55.641Z] Submodule 'inference/3rd_party/browsermt-marian-dev' (https://github.com/browsermt/marian-dev) registered for path 'inference/3rd_party/browsermt-marian-dev'
[vcs 2024-12-17T15:16:55.642Z] Submodule 'inference/3rd_party/emsdk' (https://github.com/emscripten-core/emsdk.git) registered for path 'inference/3rd_party/emsdk'
[vcs 2024-12-17T15:16:55.642Z] Submodule 'inference/3rd_party/ssplit-cpp' (https://github.com/browsermt/ssplit-cpp) registered for path 'inference/3rd_party/ssplit-cpp'
[vcs 2024-12-17T15:16:55.643Z] executing ['git', 'submodule', 'update', '--force']
[vcs 2024-12-17T15:16:55.667Z] Cloning into '/builds/worker/checkouts/src/3rd_party/browsermt-marian-dev'...
[vcs 2024-12-17T15:16:58.025Z] Cloning into '/builds/worker/checkouts/src/3rd_party/extract-lex'...
[vcs 2024-12-17T15:16:58.302Z] Cloning into '/builds/worker/checkouts/src/3rd_party/fast_align'...
[vcs 2024-12-17T15:16:58.683Z] Cloning into '/builds/worker/checkouts/src/3rd_party/kenlm'...
[vcs 2024-12-17T15:16:59.789Z] Cloning into '/builds/worker/checkouts/src/3rd_party/marian-dev'...
[vcs 2024-12-17T15:17:02.610Z] Cloning into '/builds/worker/checkouts/src/3rd_party/preprocess'...
[vcs 2024-12-17T15:17:03.211Z] Cloning into '/builds/worker/checkouts/src/inference/3rd_party/browsermt-marian-dev'...
[vcs 2024-12-17T15:17:05.562Z] Cloning into '/builds/worker/checkouts/src/inference/3rd_party/emsdk'...
[vcs 2024-12-17T15:17:06.290Z] Cloning into '/builds/worker/checkouts/src/inference/3rd_party/ssplit-cpp'...
[vcs 2024-12-17T15:17:06.725Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2024-12-17T15:17:06.733Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2024-12-17T15:17:06.740Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2024-12-17T15:17:06.765Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2024-12-17T15:17:06.881Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2024-12-17T15:17:06.898Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2024-12-17T15:17:06.971Z] Submodule path 'inference/3rd_party/browsermt-marian-dev': checked out '2781d735d4a10dca876d61be587afdab2726293c'
[vcs 2024-12-17T15:17:06.987Z] Submodule path 'inference/3rd_party/emsdk': checked out '2346baa7bb44a4a0571cc75f1986ab9aaa35aa03'
[vcs 2024-12-17T15:17:06.998Z] Submodule path 'inference/3rd_party/ssplit-cpp': checked out 'a311f9865ade34db1e8e080e6cc146f55dafb067'
[vcs 2024-12-17T15:17:06.999Z] cleaning git checkout...
[vcs 2024-12-17T15:17:06.999Z] executing ['git', 'clean', '-nxdff']
[vcs 2024-12-17T15:17:07.003Z] removing []
[vcs 2024-12-17T15:17:07.003Z] successfully cleaned git checkout!
[vcs 2024-12-17T15:17:07.005Z] TinderboxPrint:<a href='https://github.com/bhearsum/firefox-translations-training/commit/c6a36ed189c2103d3754975d4e8bea7abcb34a7f' title='Built from firefox-translations-training commit c6a36ed189c2103d3754975d4e8bea7abcb34a7f'>c6a36ed189c2103d3754975d4e8bea7abcb34a7f</a>
[task 2024-12-17T15:17:07.006Z] executing ['bash', '-cx', "cd /builds/worker/checkouts/src && ln -s /builds/worker/artifacts artifacts && taskgraph decision --pushlog-id='0' --pushdate='0' --project='translations' --owner='[email protected]' --level='1' --repository-type=git --tasks-for='github-pull-request' --base-repository='https://github.com/mozilla/translations' --base-ref='main' --base-rev='051db33d26fe6c2ca1e0e7940343295b9fbb004f' --head-repository='https://github.com/bhearsum/firefox-translations-training' --head-ref='cpu-only-upgrade' --head-rev='c6a36ed189c2103d3754975d4e8bea7abcb34a7f' \n"]
[task 2024-12-17T15:17:07.008Z] + cd /builds/worker/checkouts/src
[task 2024-12-17T15:17:07.008Z] + ln -s /builds/worker/artifacts artifacts
[task 2024-12-17T15:17:07.010Z] + taskgraph decision --pushlog-id=0 --pushdate=0 --project=translations [email protected] --level=1 --repository-type=git --tasks-for=github-pull-request --base-repository=https://github.com/mozilla/translations --base-ref=main --base-rev=051db33d26fe6c2ca1e0e7940343295b9fbb004f --head-repository=https://github.com/bhearsum/firefox-translations-training --head-ref=cpu-only-upgrade --head-rev=c6a36ed189c2103d3754975d4e8bea7abcb34a7f
[task 2024-12-17T15:17:07.280Z] 2024-12-17 15:17:07,280 - INFO - Loading graph configuration.
[task 2024-12-17T15:17:07.302Z] 2024-12-17 15:17:07,302 - WARNING - using default project parameters; add translations to PER_PROJECT_PARAMETERS in /usr/local/lib/python3.11/dist-packages/taskgraph/decision.py to customize behavior for this project
[task 2024-12-17T15:17:07.309Z] 2024-12-17 15:17:07,309 - INFO - Using Parameters(id=70748995d276) (from defaults)
[task 2024-12-17T15:17:07.311Z] 2024-12-17 15:17:07,311 - INFO - writing artifact file `parameters.yml`
[task 2024-12-17T15:17:07.319Z] 2024-12-17 15:17:07,319 - INFO - writing artifact file `actions.json`
[task 2024-12-17T15:17:07.321Z] 2024-12-17 15:17:07,321 - INFO - Loading kinds
[task 2024-12-17T15:17:07.355Z] 2024-12-17 15:17:07,355 - INFO - Generating full task set
[task 2024-12-17T15:17:07.385Z] 2024-12-17 15:17:07,385 - INFO - Generated 12 tasks for kind dataset
[task 2024-12-17T15:17:07.387Z] 2024-12-17 15:17:07,386 - INFO - Writing docker-contexts/base.tar.gz for docker image base
[task 2024-12-17T15:17:07.403Z] 2024-12-17 15:17:07,403 - INFO - Writing docker-contexts/inference.tar.gz for docker image inference
[task 2024-12-17T15:17:07.404Z] 2024-12-17 15:17:07,404 - INFO - Writing docker-contexts/test.tar.gz for docker image test
[task 2024-12-17T15:17:07.405Z] 2024-12-17 15:17:07,405 - INFO - Writing docker-contexts/train.tar.gz for docker image train
[task 2024-12-17T15:17:07.406Z] 2024-12-17 15:17:07,406 - INFO - Writing docker-contexts/toolchain-build.tar.gz for docker image toolchain-build
[task 2024-12-17T15:17:07.424Z] 2024-12-17 15:17:07,424 - INFO - Generated 5 tasks for kind docker-image
[task 2024-12-17T15:17:07.434Z] 2024-12-17 15:17:07,434 - INFO - Generated 9 tasks for kind fetch
[task 2024-12-17T15:17:07.530Z] 2024-12-17 15:17:07,529 - INFO - Generated 10 tasks for kind toolchain
[task 2024-12-17T15:17:07.535Z] 2024-12-17 15:17:07,534 - INFO - Generated 4 tasks for kind analyze-mono
[task 2024-12-17T15:17:07.539Z] 2024-12-17 15:17:07,539 - INFO - Generated 4 tasks for kind analyze-corpus
[task 2024-12-17T15:17:07.545Z] 2024-12-17 15:17:07,545 - INFO - Generated 4 tasks for kind clean-corpus
[task 2024-12-17T15:17:07.548Z] 2024-12-17 15:17:07,548 - INFO - Generated 1 tasks for kind bicleaner-model
[task 2024-12-17T15:17:07.554Z] 2024-12-17 15:17:07,554 - INFO - Generated 4 tasks for kind bicleaner
[task 2024-12-17T15:17:07.559Z] 2024-12-17 15:17:07,559 - INFO - Generated 4 tasks for kind clean-mono
[task 2024-12-17T15:17:07.564Z] 2024-12-17 15:17:07,564 - INFO - Generated 1 tasks for kind merge-corpus
[task 2024-12-17T15:17:07.568Z] 2024-12-17 15:17:07,568 - INFO - Generated 2 tasks for kind merge-mono
[task 2024-12-17T15:17:07.570Z] 2024-12-17 15:17:07,570 - INFO - Generated 1 tasks for kind train-vocab
[task 2024-12-17T15:17:07.573Z] 2024-12-17 15:17:07,573 - INFO - Generated 1 tasks for kind split-corpus
[task 2024-12-17T15:17:07.575Z] 2024-12-17 15:17:07,575 - INFO - Generated 1 tasks for kind merge-devset
[task 2024-12-17T15:17:07.580Z] 2024-12-17 15:17:07,580 - INFO - Generated 4 tasks for kind inference
[task 2024-12-17T15:17:07.584Z] 2024-12-17 15:17:07,584 - INFO - Generated 1 tasks for kind train-backwards
[task 2024-12-17T15:17:07.587Z] 2024-12-17 15:17:07,587 - INFO - Generated 1 tasks for kind split-mono-src
[task 2024-12-17T15:17:07.589Z] 2024-12-17 15:17:07,589 - INFO - Generated 1 tasks for kind split-mono-trg
[task 2024-12-17T15:17:07.594Z] 2024-12-17 15:17:07,594 - INFO - Generated 7 tasks for kind tests
[task 2024-12-17T15:17:07.599Z] 2024-12-17 15:17:07,599 - INFO - Generated 2 tasks for kind translate-mono-trg
[task 2024-12-17T15:17:07.602Z] 2024-12-17 15:17:07,602 - INFO - Generated 1 tasks for kind collect-mono-trg
[task 2024-12-17T15:17:07.605Z] 2024-12-17 15:17:07,605 - INFO - Generated 1 tasks for kind alignments-original
[task 2024-12-17T15:17:07.608Z] 2024-12-17 15:17:07,608 - INFO - Generated 1 tasks for kind alignments-backtranslated
[task 2024-12-17T15:17:07.611Z] 2024-12-17 15:17:07,611 - INFO - Generated 1 tasks for kind train-teacher
[task 2024-12-17T15:17:07.614Z] 2024-12-17 15:17:07,614 - INFO - Generated 1 tasks for kind evaluate-teacher-ensemble
[task 2024-12-17T15:17:07.618Z] 2024-12-17 15:17:07,618 - INFO - Generated 2 tasks for kind translate-corpus
[task 2024-12-17T15:17:07.622Z] 2024-12-17 15:17:07,622 - INFO - Generated 2 tasks for kind extract-best
[task 2024-12-17T15:17:07.624Z] 2024-12-17 15:17:07,624 - INFO - Generated 1 tasks for kind collect-corpus
[task 2024-12-17T15:17:07.628Z] 2024-12-17 15:17:07,628 - INFO - Generated 2 tasks for kind translate-mono-src
[task 2024-12-17T15:17:07.631Z] 2024-12-17 15:17:07,631 - INFO - Generated 1 tasks for kind collect-mono-src
[task 2024-12-17T15:17:07.633Z] 2024-12-17 15:17:07,633 - INFO - Generated 1 tasks for kind merge-translated
[task 2024-12-17T15:17:07.636Z] 2024-12-17 15:17:07,636 - INFO - Generated 1 tasks for kind score
[task 2024-12-17T15:17:07.639Z] 2024-12-17 15:17:07,639 - INFO - Generated 1 tasks for kind cefilter
[task 2024-12-17T15:17:07.642Z] 2024-12-17 15:17:07,642 - INFO - Generated 1 tasks for kind shortlist
[task 2024-12-17T15:17:07.644Z] 2024-12-17 15:17:07,644 - INFO - Generated 1 tasks for kind alignments-student
[task 2024-12-17T15:17:07.647Z] 2024-12-17 15:17:07,647 - INFO - Generated 1 tasks for kind train-student
[task 2024-12-17T15:17:07.650Z] 2024-12-17 15:17:07,650 - INFO - Generated 1 tasks for kind finetune-student
[task 2024-12-17T15:17:07.652Z] 2024-12-17 15:17:07,652 - INFO - Generated 1 tasks for kind quantize
[task 2024-12-17T15:17:07.656Z] 2024-12-17 15:17:07,655 - INFO - Generated 1 tasks for kind evaluate-quantized
[task 2024-12-17T15:17:07.658Z] 2024-12-17 15:17:07,658 - INFO - Generated 1 tasks for kind export
[task 2024-12-17T15:17:07.664Z] 2024-12-17 15:17:07,664 - INFO - Generated 4 tasks for kind evaluate
[task 2024-12-17T15:17:07.666Z] 2024-12-17 15:17:07,666 - INFO - Generated 1 tasks for kind all
[task 2024-12-17T15:17:07.667Z] 2024-12-17 15:17:07,667 - INFO - Generated 1 tasks for kind all-pr-pipeline
[task 2024-12-17T15:17:07.668Z] 2024-12-17 15:17:07,667 - INFO - Generating full task graph
[task 2024-12-17T15:17:07.668Z] 2024-12-17 15:17:07,668 - INFO - Full task graph contains 108 tasks and 301 dependencies
[task 2024-12-17T15:17:07.673Z] 2024-12-17 15:17:07,672 - INFO - writing artifact file `full-task-graph.json`
[task 2024-12-17T15:17:07.697Z] 2024-12-17 15:17:07,696 - INFO - writing artifact file `runnable-jobs.json`
[task 2024-12-17T15:17:07.698Z] 2024-12-17 15:17:07,697 - INFO - Generating target task set
[task 2024-12-17T15:17:07.698Z] 2024-12-17 15:17:07,698 - INFO - Filter filter_target_tasks pruned 86 tasks (22 remain)
[task 2024-12-17T15:17:07.699Z] 2024-12-17 15:17:07,698 - INFO - writing artifact file `target-tasks.json`
[task 2024-12-17T15:17:07.699Z] 2024-12-17 15:17:07,698 - INFO - Generating target task graph
[task 2024-12-17T15:17:07.699Z] 2024-12-17 15:17:07,699 - INFO - Adding 5 tasks with `always_target` attribute
[task 2024-12-17T15:17:07.700Z] 2024-12-17 15:17:07,700 - INFO - Generating optimized task graph
[task 2024-12-17T15:17:07.709Z] 2024-12-17 15:17:07,709 - INFO - Removed 70 tasks by dependents optimized, 5 tasks by skip-unless-changed (skip-unless-changed) during optimization.
[task 2024-12-17T15:17:08.625Z] 2024-12-17 15:17:08,625 - INFO - Replaced 94 tasks by index-search (index-search) during optimization.
[task 2024-12-17T15:17:08.627Z] 2024-12-17 15:17:08,627 - INFO - writing artifact file `task-graph.json`
[task 2024-12-17T15:17:08.628Z] 2024-12-17 15:17:08,628 - INFO - writing artifact file `label-to-taskid.json`
[task 2024-12-17T15:17:08.630Z] 2024-12-17 15:17:08,630 - INFO - Creating task with taskId MpDVE80lSymxW-PqFW7bqQ for tests-taskgraph-diff
[task 2024-12-17T15:17:08.631Z] 2024-12-17 15:17:08,630 - INFO - Creating task with taskId V8pqaajzSeygJBpU83c04w for tests-lint-pyright
[task 2024-12-17T15:17:08.633Z] 2024-12-17 15:17:08,633 - INFO - Creating task with taskId b7e-G3gkTdinN5UX4OSPpg for tests-lint-ruff
[task 2024-12-17T15:17:08.635Z] 2024-12-17 15:17:08,635 - INFO - Creating task with taskId Uh7vr3p0Sta-AOeCx8Fr9Q for tests-test
[task 2024-12-17T15:17:08.640Z] 2024-12-17 15:17:08,639 - INFO - Creating task with taskId PvKSJEpKQnyLOp3M-32f8Q for tests-lint-black
[task 2024-12-17T15:17:08.642Z] 2024-12-17 15:17:08,642 - INFO - Creating task with taskId R8P2KaP6QgWeGSmAFGmfZQ for tests-lint-eslint
[task 2024-12-17T15:17:08.652Z] 2024-12-17 15:17:08,652 - INFO - Creating task with taskId AK4e4FIvRkKERoefDWLBmA for tests-taskgraph-definition
[taskcluster 2024-12-17 15:17:09.162Z] === Task Finished ===
[taskcluster 2024-12-17 15:17:11.054Z] Successful task run with exit code: 0 completed in 30.868 seconds