Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: translations integration tests #210

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

remove `REV` from integration test task definitions

1bcb59e
Select commit
Loading
Failed to load commit list.
Open

feat: translations integration tests #210

remove `REV` from integration test task definitions
1bcb59e
Select commit
Loading
Failed to load commit list.
stage-taskcluster / translations-dataset-opus-tldr-pages_v2023-08-29-en succeeded Jan 27, 2025 in 9m 41s

Stage-TC (issue_comment)

Fetch a monolingual opus dataset for {trg_locale}

Details

View task in Taskcluster | View logs in Taskcluster | View task group in Taskcluster

Task Status

Started: 2025-01-27T12:11:37.000Z
Resolved: 2025-01-27T12:14:53.736Z
Task Execution Time: 3 minutes, 16 seconds, 736 milliseconds
Task Status: completed
Reason Resolved: completed
RunId: 0

Artifacts

- public/build/tldr-pages_v2023-08-29.en.zst
- public/logs/live_backing.log
- public/logs/live.log


[taskcluster 2025-01-27 12:11:37.096Z] Task ID: MpMjzbh7RwyhdJXNOSQR0Q
[taskcluster 2025-01-27 12:11:37.096Z] Worker ID: 6177806456876353906
[taskcluster 2025-01-27 12:11:37.096Z] Worker Group: us-west1-b
[taskcluster 2025-01-27 12:11:37.096Z] Worker Node Type: projects/474530013412/machineTypes/n2-highmem-32
[taskcluster 2025-01-27 12:11:37.096Z] Worker Pool: translations-1/b-linux-large-gcp
[taskcluster 2025-01-27 12:11:37.096Z] Worker Version: 38.0.5
[taskcluster 2025-01-27 12:11:37.096Z] Public IP: 34.168.114.12
[taskcluster 2025-01-27 12:11:37.096Z] Hostname: translations-1-b-linux-large-gcp-eefvi9nvsh-sf7cg9qaapq
[taskcluster 2025-01-27 12:11:37.096Z] using cache "translations-level-1-checkouts-v3-7afeb851dd97df8f3607-LaTmU8ILTKmjq-SGho18mA" -> /builds/worker/checkouts

[taskcluster 2025-01-27 12:11:46.286Z] Downloading artifact "public/image.tar.zst" from task ID: DUNjdZeFTIScNP4MGcgXGQ.
[taskcluster 2025-01-27 12:11:51.288Z] Download Progress: 43.54%
[taskcluster 2025-01-27 12:11:56.288Z] Download Progress: 93.97%
[taskcluster 2025-01-27 12:11:56.698Z] Downloaded artifact successfully.
[taskcluster 2025-01-27 12:11:56.698Z] Downloaded 761.177 mb
[taskcluster 2025-01-27 12:11:56.699Z] Decompressing downloaded image
[taskcluster 2025-01-27 12:12:03.439Z] Loading docker image from downloaded archive.
[taskcluster 2025-01-27 12:12:42.236Z] Image 'public/image.tar.zst' from task 'DUNjdZeFTIScNP4MGcgXGQ' loaded.  Using image ID sha256:a6cdc387f7c79fe4f5247374a546bc03afc39e06ae273fa8d16a68cb6a594d1f.
[taskcluster 2025-01-27 12:12:43.639Z] === Task Starting ===
[setup 2025-01-27T12:12:49.835Z] run-task started in /builds/worker

...(146 lines hidden)...

[task 2025-01-27T12:13:11.106Z]   Downloading scikit_learn-1.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)
[task 2025-01-27T12:13:11.225Z] Collecting scipy==1.14.0 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 110))
[task 2025-01-27T12:13:11.236Z]   Downloading scipy-1.14.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
[task 2025-01-27T12:13:11.298Z] Collecting sentencepiece==0.1.99 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 114))
[task 2025-01-27T12:13:11.309Z]   Downloading sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.7 kB)
[task 2025-01-27T12:13:11.328Z] Collecting simalign==0.4 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 116))
[task 2025-01-27T12:13:11.338Z]   Downloading simalign-0.4-py3-none-any.whl.metadata (6.2 kB)
[task 2025-01-27T12:13:11.359Z] Collecting six==1.16.0 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 118))
[task 2025-01-27T12:13:11.368Z]   Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
[task 2025-01-27T12:13:11.392Z] Collecting sympy==1.12.1 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 122))
[task 2025-01-27T12:13:11.402Z]   Downloading sympy-1.12.1-py3-none-any.whl.metadata (12 kB)
[task 2025-01-27T12:13:11.422Z] Collecting tabulate==0.9.0 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 124))
[task 2025-01-27T12:13:11.431Z]   Downloading tabulate-0.9.0-py3-none-any.whl.metadata (34 kB)
[task 2025-01-27T12:13:11.452Z] Collecting threadpoolctl==3.5.0 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 126))
[task 2025-01-27T12:13:11.461Z]   Downloading threadpoolctl-3.5.0-py3-none-any.whl.metadata (13 kB)
[task 2025-01-27T12:13:11.626Z] Collecting tokenizers==0.19.1 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 128))
[task 2025-01-27T12:13:11.636Z]   Downloading tokenizers-0.19.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)
[task 2025-01-27T12:13:11.687Z] Collecting torch==2.2.2 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 130))
[task 2025-01-27T12:13:11.697Z]   Downloading torch-2.2.2-cp310-cp310-manylinux1_x86_64.whl.metadata (26 kB)
[task 2025-01-27T12:13:11.744Z] Collecting tqdm==4.66.4 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 132))
[task 2025-01-27T12:13:11.754Z]   Downloading tqdm-4.66.4-py3-none-any.whl.metadata (57 kB)
[task 2025-01-27T12:13:11.802Z] Collecting transformers==4.42.3 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 137))
[task 2025-01-27T12:13:11.813Z]   Downloading transformers-4.42.3-py3-none-any.whl.metadata (43 kB)
[task 2025-01-27T12:13:12.089Z] Collecting typing-extensions==4.12.2 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 139))
[task 2025-01-27T12:13:12.098Z]   Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
[task 2025-01-27T12:13:12.115Z] Collecting typo==0.1.5 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 143))
[task 2025-01-27T12:13:12.124Z]   Downloading typo-0.1.5.tar.gz (7.0 kB)
[task 2025-01-27T12:13:12.133Z]   Preparing metadata (setup.py): started
[task 2025-01-27T12:13:12.311Z]   Preparing metadata (setup.py): finished with status 'done'
[task 2025-01-27T12:13:12.343Z] Collecting urllib3==2.2.2 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 145))
[task 2025-01-27T12:13:12.352Z]   Downloading urllib3-2.2.2-py3-none-any.whl.metadata (6.4 kB)
[task 2025-01-27T12:13:12.374Z] Collecting wcwidth==0.2.13 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 147))
[task 2025-01-27T12:13:12.383Z]   Downloading wcwidth-0.2.13-py2.py3-none-any.whl.metadata (14 kB)
[task 2025-01-27T12:13:12.403Z] Collecting zhon==2.0.2 (from -r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 149))
[task 2025-01-27T12:13:12.412Z]   Downloading zhon-2.0.2-py3-none-any.whl.metadata (3.4 kB)
[task 2025-01-27T12:13:12.629Z] Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch==2.2.2->-r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 130))
[task 2025-01-27T12:13:12.638Z]   Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
[task 2025-01-27T12:13:12.658Z] Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch==2.2.2->-r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 130))
[task 2025-01-27T12:13:12.667Z]   Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
[task 2025-01-27T12:13:12.689Z] Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch==2.2.2->-r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 130))
[task 2025-01-27T12:13:12.698Z]   Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
[task 2025-01-27T12:13:12.718Z] Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch==2.2.2->-r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 130))
[task 2025-01-27T12:13:12.727Z]   Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
[task 2025-01-27T12:13:12.748Z] Collecting nvidia-cublas-cu12==12.1.3.1 (from torch==2.2.2->-r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 130))
[task 2025-01-27T12:13:12.757Z]   Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
[task 2025-01-27T12:13:12.776Z] Collecting nvidia-cufft-cu12==11.0.2.54 (from torch==2.2.2->-r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 130))
[task 2025-01-27T12:13:12.785Z]   Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
[task 2025-01-27T12:13:12.806Z] Collecting nvidia-curand-cu12==10.3.2.106 (from torch==2.2.2->-r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 130))
[task 2025-01-27T12:13:12.815Z]   Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
[task 2025-01-27T12:13:12.835Z] Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch==2.2.2->-r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 130))
[task 2025-01-27T12:13:12.844Z]   Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
[task 2025-01-27T12:13:12.864Z] Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch==2.2.2->-r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 130))
[task 2025-01-27T12:13:12.872Z]   Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
[task 2025-01-27T12:13:12.892Z] Collecting nvidia-nccl-cu12==2.19.3 (from torch==2.2.2->-r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 130))
[task 2025-01-27T12:13:12.901Z]   Downloading nvidia_nccl_cu12-2.19.3-py3-none-manylinux1_x86_64.whl.metadata (1.8 kB)
[task 2025-01-27T12:13:12.921Z] Collecting nvidia-nvtx-cu12==12.1.105 (from torch==2.2.2->-r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 130))
[task 2025-01-27T12:13:12.931Z]   Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)
[task 2025-01-27T12:13:12.951Z] Collecting triton==2.2.0 (from torch==2.2.2->-r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 130))
[task 2025-01-27T12:13:12.961Z]   Downloading triton-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.4 kB)
[task 2025-01-27T12:13:13.071Z] Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch==2.2.2->-r /builds/worker/checkouts/vcs/pipeline/data/requirements/data.txt (line 130))
[task 2025-01-27T12:13:13.079Z]   Downloading nvidia_nvjitlink_cu12-12.8.61-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl.metadata (1.7 kB)
[task 2025-01-27T12:13:13.116Z] Downloading blessed-1.20.0-py2.py3-none-any.whl (58 kB)
[task 2025-01-27T12:13:13.127Z] Downloading certifi-2024.6.2-py3-none-any.whl (164 kB)
[task 2025-01-27T12:13:13.145Z] Downloading charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB)
[task 2025-01-27T12:13:13.156Z] Downloading click-8.1.7-py3-none-any.whl (97 kB)
[task 2025-01-27T12:13:13.167Z] Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)
[task 2025-01-27T12:13:13.179Z] Downloading enlighten-1.10.1-py2.py3-none-any.whl (53 kB)
[task 2025-01-27T12:13:13.190Z] Downloading filelock-3.15.4-py3-none-any.whl (16 kB)
[task 2025-01-27T12:13:13.201Z] Downloading fsspec-2024.6.1-py3-none-any.whl (177 kB)
[task 2025-01-27T12:13:13.214Z] Downloading hanzidentifier-1.2.0-py3-none-any.whl (4.8 kB)
[task 2025-01-27T12:13:13.226Z] Downloading huggingface_hub-0.23.4-py3-none-any.whl (402 kB)
[task 2025-01-27T12:13:13.239Z] Downloading idna-3.7-py3-none-any.whl (66 kB)
[task 2025-01-27T12:13:13.251Z] Downloading jinja2-3.1.4-py3-none-any.whl (133 kB)
[task 2025-01-27T12:13:13.262Z] Downloading joblib-1.4.2-py3-none-any.whl (301 kB)
[task 2025-01-27T12:13:13.275Z] Downloading latexcodec-3.0.0-py3-none-any.whl (18 kB)
[task 2025-01-27T12:13:13.286Z] Downloading lxml-5.3.0-cp310-cp310-manylinux_2_28_x86_64.whl (5.0 MB)
[task 2025-01-27T12:13:13.316Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.0/5.0 MB 180.7 MB/s eta 0:00:00
[task 2025-01-27T12:13:13.325Z] Downloading MarkupSafe-2.1.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
[task 2025-01-27T12:13:13.336Z] Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
[task 2025-01-27T12:13:13.343Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 79.9 MB/s eta 0:00:00
[task 2025-01-27T12:13:13.355Z] Downloading mtdata-0.4.1-py3-none-any.whl (819 kB)
[task 2025-01-27T12:13:13.363Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 819.3/819.3 kB 129.1 MB/s eta 0:00:00
[task 2025-01-27T12:13:13.372Z] Downloading networkx-3.3-py3-none-any.whl (1.7 MB)
[task 2025-01-27T12:13:13.382Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 204.0 MB/s eta 0:00:00
[task 2025-01-27T12:13:13.391Z] Downloading numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
[task 2025-01-27T12:13:13.466Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 250.6 MB/s eta 0:00:00
[task 2025-01-27T12:13:13.479Z] Downloading OpenCC-1.1.9-cp310-cp310-manylinux2014_x86_64.whl (1.7 MB)
[task 2025-01-27T12:13:13.489Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 193.1 MB/s eta 0:00:00
[task 2025-01-27T12:13:13.498Z] Downloading packaging-24.1-py3-none-any.whl (53 kB)
[task 2025-01-27T12:13:13.511Z] Downloading portalocker-2.3.0-py2.py3-none-any.whl (15 kB)
[task 2025-01-27T12:13:13.524Z] Downloading prefixed-0.7.1-py2.py3-none-any.whl (13 kB)
[task 2025-01-27T12:13:13.535Z] Downloading psutil-6.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (290 kB)
[task 2025-01-27T12:13:13.548Z] Downloading pybtex-0.24.0-py2.py3-none-any.whl (561 kB)
[task 2025-01-27T12:13:13.557Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 561.4/561.4 kB 65.0 MB/s eta 0:00:00
[task 2025-01-27T12:13:13.566Z] Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB)
[task 2025-01-27T12:13:13.574Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 705.5/705.5 kB 94.4 MB/s eta 0:00:00
[task 2025-01-27T12:13:13.585Z] Downloading regex-2024.5.15-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (775 kB)
[task 2025-01-27T12:13:13.593Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 775.1/775.1 kB 119.1 MB/s eta 0:00:00
[task 2025-01-27T12:13:13.602Z] Downloading requests-2.31.0-py3-none-any.whl (62 kB)
[task 2025-01-27T12:13:13.613Z] Downloading ruamel.yaml-0.18.6-py3-none-any.whl (117 kB)
[task 2025-01-27T12:13:13.625Z] Downloading ruamel.yaml.clib-0.2.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (526 kB)
[task 2025-01-27T12:13:13.632Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 526.7/526.7 kB 69.2 MB/s eta 0:00:00
[task 2025-01-27T12:13:13.642Z] Downloading sacrebleu-2.4.2-py3-none-any.whl (106 kB)
[task 2025-01-27T12:13:13.657Z] Downloading sacremoses-0.1.1-py3-none-any.whl (897 kB)
[task 2025-01-27T12:13:13.665Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 897.5/897.5 kB 121.8 MB/s eta 0:00:00
[task 2025-01-27T12:13:13.677Z] Downloading safetensors-0.4.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
[task 2025-01-27T12:13:13.689Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 111.1 MB/s eta 0:00:00
[task 2025-01-27T12:13:13.698Z] Downloading scikit_learn-1.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.3 MB)
[task 2025-01-27T12:13:13.753Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.3/13.3 MB 252.5 MB/s eta 0:00:00
[task 2025-01-27T12:13:13.766Z] Downloading scipy-1.14.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (41.1 MB)
[task 2025-01-27T12:13:13.968Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.1/41.1 MB 206.2 MB/s eta 0:00:00
[task 2025-01-27T12:13:13.979Z] Downloading sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
[task 2025-01-27T12:13:14.036Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 19.7 MB/s eta 0:00:00
[task 2025-01-27T12:13:14.045Z] Downloading simalign-0.4-py3-none-any.whl (8.1 kB)
[task 2025-01-27T12:13:14.056Z] Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
[task 2025-01-27T12:13:14.069Z] Downloading sympy-1.12.1-py3-none-any.whl (5.7 MB)
[task 2025-01-27T12:13:14.097Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 222.0 MB/s eta 0:00:00
[task 2025-01-27T12:13:14.106Z] Downloading tabulate-0.9.0-py3-none-any.whl (35 kB)
[task 2025-01-27T12:13:14.118Z] Downloading threadpoolctl-3.5.0-py3-none-any.whl (18 kB)
[task 2025-01-27T12:13:14.136Z] Downloading tokenizers-0.19.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
[task 2025-01-27T12:13:14.155Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 209.6 MB/s eta 0:00:00
[task 2025-01-27T12:13:14.167Z] Downloading torch-2.2.2-cp310-cp310-manylinux1_x86_64.whl (755.5 MB)
[task 2025-01-27T12:13:19.260Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 755.5/755.5 MB 80.7 MB/s eta 0:00:00
[task 2025-01-27T12:13:19.269Z] Downloading tqdm-4.66.4-py3-none-any.whl (78 kB)
[task 2025-01-27T12:13:19.319Z] Downloading transformers-4.42.3-py3-none-any.whl (9.3 MB)
[task 2025-01-27T12:13:19.377Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.3/9.3 MB 165.8 MB/s eta 0:00:00
[task 2025-01-27T12:13:19.386Z] Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB)
[task 2025-01-27T12:13:19.397Z] Downloading urllib3-2.2.2-py3-none-any.whl (121 kB)
[task 2025-01-27T12:13:19.409Z] Downloading wcwidth-0.2.13-py2.py3-none-any.whl (34 kB)
[task 2025-01-27T12:13:19.420Z] Downloading zhon-2.0.2-py3-none-any.whl (83 kB)
[task 2025-01-27T12:13:19.435Z] Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)
[task 2025-01-27T12:13:22.314Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 410.6/410.6 MB 113.5 MB/s eta 0:00:00
[task 2025-01-27T12:13:22.325Z] Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)
[task 2025-01-27T12:13:22.393Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.1/14.1 MB 216.2 MB/s eta 0:00:00
[task 2025-01-27T12:13:22.404Z] Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)
[task 2025-01-27T12:13:22.524Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.7/23.7 MB 199.3 MB/s eta 0:00:00
[task 2025-01-27T12:13:22.536Z] Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)
[task 2025-01-27T12:13:22.565Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 823.6/823.6 kB 21.7 MB/s eta 0:00:00
[task 2025-01-27T12:13:22.577Z] Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)
[task 2025-01-27T12:13:27.697Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 731.7/731.7 MB 77.2 MB/s eta 0:00:00
[task 2025-01-27T12:13:27.709Z] Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)
[task 2025-01-27T12:13:28.447Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.6/121.6 MB 165.3 MB/s eta 0:00:00
[task 2025-01-27T12:13:28.459Z] Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)
[task 2025-01-27T12:13:28.743Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.5/56.5 MB 200.5 MB/s eta 0:00:00
[task 2025-01-27T12:13:28.759Z] Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)
[task 2025-01-27T12:13:29.625Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.2/124.2 MB 143.7 MB/s eta 0:00:00
[task 2025-01-27T12:13:29.639Z] Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)
[task 2025-01-27T12:13:30.847Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 196.0/196.0 MB 162.5 MB/s eta 0:00:00
[task 2025-01-27T12:13:30.858Z] Downloading nvidia_nccl_cu12-2.19.3-py3-none-manylinux1_x86_64.whl (166.0 MB)
[task 2025-01-27T12:13:32.008Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 166.0/166.0 MB 144.6 MB/s eta 0:00:00
[task 2025-01-27T12:13:32.017Z] Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)
[task 2025-01-27T12:13:32.034Z] Downloading triton-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (167.9 MB)
[task 2025-01-27T12:13:33.115Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 167.9/167.9 MB 155.6 MB/s eta 0:00:00
[task 2025-01-27T12:13:33.127Z] Downloading nvidia_nvjitlink_cu12-12.8.61-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl (39.2 MB)
[task 2025-01-27T12:13:33.326Z]    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 39.2/39.2 MB 199.0 MB/s eta 0:00:00
[task 2025-01-27T12:13:40.746Z] Building wheels for collected packages: typo, opustrainer
[task 2025-01-27T12:13:40.747Z]   Building wheel for typo (setup.py): started
[task 2025-01-27T12:13:40.961Z]   Building wheel for typo (setup.py): finished with status 'done'
[task 2025-01-27T12:13:40.962Z]   Created wheel for typo: filename=typo-0.1.5-py3-none-any.whl size=6878 sha256=4c340de47e1b8ff40b93d2ffc0579b708dbc507ba2b17c74359200fb5a5c1703
[task 2025-01-27T12:13:40.962Z]   Stored in directory: /builds/worker/.cache/pip/wheels/2e/2f/73/60e0ce42d1375a386b9171a37cd5536e173ad950a98e7dc6b1
[task 2025-01-27T12:13:40.969Z]   Building wheel for opustrainer (pyproject.toml): started
[task 2025-01-27T12:13:41.231Z]   Building wheel for opustrainer (pyproject.toml): finished with status 'done'
[task 2025-01-27T12:13:41.232Z]   Created wheel for opustrainer: filename=opustrainer-0.3-py3-none-any.whl size=45114 sha256=5ec39bb273da3437957daea51af3d96512a66ef0d1450a18396bc6224b0511d8
[task 2025-01-27T12:13:41.232Z]   Stored in directory: /builds/worker/.cache/pip/wheels/09/01/1c/d82c1698dc71385b0bf6eae4f433e85a02f89f99eec2e21072
[task 2025-01-27T12:13:41.235Z] Successfully built typo opustrainer
[task 2025-01-27T12:13:41.862Z] Installing collected packages: wcwidth, typo, sentencepiece, prefixed, portalocker, opencc, mpmath, zhon, urllib3, typing-extensions, tqdm, threadpoolctl, tabulate, sympy, six, safetensors, ruamel-yaml-clib, regex, pyyaml, psutil, packaging, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, numpy, networkx, markupsafe, lxml, latexcodec, joblib, idna, fsspec, filelock, colorama, click, charset-normalizer, certifi, triton, scipy, sacremoses, sacrebleu, ruamel-yaml, requests, pybtex, nvidia-cusparse-cu12, nvidia-cudnn-cu12, jinja2, hanzidentifier, blessed, scikit-learn, opustrainer, nvidia-cusolver-cu12, huggingface-hub, enlighten, torch, tokenizers, mtdata, transformers, simalign
[task 2025-01-27T12:14:42.736Z] Successfully installed blessed-1.20.0 certifi-2024.6.2 charset-normalizer-3.3.2 click-8.1.7 colorama-0.4.6 enlighten-1.10.1 filelock-3.15.4 fsspec-2024.6.1 hanzidentifier-1.2.0 huggingface-hub-0.23.4 idna-3.7 jinja2-3.1.4 joblib-1.4.2 latexcodec-3.0.0 lxml-5.3.0 markupsafe-2.1.5 mpmath-1.3.0 mtdata-0.4.1 networkx-3.3 numpy-1.26.4 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.19.3 nvidia-nvjitlink-cu12-12.8.61 nvidia-nvtx-cu12-12.1.105 opencc-1.1.9 opustrainer-0.3 packaging-24.1 portalocker-2.3.0 prefixed-0.7.1 psutil-6.0.0 pybtex-0.24.0 pyyaml-6.0.1 regex-2024.5.15 requests-2.31.0 ruamel-yaml-0.18.6 ruamel-yaml-clib-0.2.8 sacrebleu-2.4.2 sacremoses-0.1.1 safetensors-0.4.3 scikit-learn-1.5.0 scipy-1.14.0 sentencepiece-0.1.99 simalign-0.4 six-1.16.0 sympy-1.12.1 tabulate-0.9.0 threadpoolctl-3.5.0 tokenizers-0.19.1 torch-2.2.2 tqdm-4.66.4 transformers-4.42.3 triton-2.2.0 typing-extensions-4.12.2 typo-0.1.5 urllib3-2.2.2 wcwidth-0.2.13 zhon-2.0.2
[task 2025-01-27T12:14:43.628Z] [download-mono] Dataset: opus_tldr-pages/v2023-08-29
[task 2025-01-27T12:14:43.628Z] [download-mono] Language: en
[task 2025-01-27T12:14:43.628Z] [download-mono] Max Sentences: 500
[task 2025-01-27T12:14:43.628Z] [download-mono] Mininmum Fluency Threshold: 0.8
[task 2025-01-27T12:14:43.628Z] [download-mono] Artifacts: /builds/worker/artifacts
[task 2025-01-27T12:14:43.628Z] [download-mono] File Destination: /builds/worker/artifacts/tldr-pages_v2023-08-29.en.zst
[task 2025-01-27T12:14:43.628Z] [download-mono] Downloading OPUS monolingual data
[task 2025-01-27T12:14:43.628Z] [download-mono] https://object.pouta.csc.fi/OPUS-tldr-pages/v2023-08-29/mono/en.txt.gz
[task 2025-01-27T12:14:43.628Z] [download-mono] URL: https://object.pouta.csc.fi/OPUS-tldr-pages/v2023-08-29/mono/en.txt.gz
[task 2025-01-27T12:14:45.760Z] [downloads] Download size: 446,937 bytes
[task 2025-01-27T12:14:45.815Z] [downloads] 5% downloaded (24576/446937 bytes)
[task 2025-01-27T12:14:45.975Z] [downloads] 11% downloaded (49152/446937 bytes)
[task 2025-01-27T12:14:45.978Z] [downloads] 16% downloaded (73728/446937 bytes)
[task 2025-01-27T12:14:45.979Z] [downloads] 20% downloaded (90112/446937 bytes)
[task 2025-01-27T12:14:46.137Z] [downloads] 26% downloaded (114688/446937 bytes)
[task 2025-01-27T12:14:46.139Z] [downloads] 31% downloaded (139264/446937 bytes)
[task 2025-01-27T12:14:46.141Z] [downloads] 37% downloaded (163840/446937 bytes)
[task 2025-01-27T12:14:46.143Z] [downloads] 40% downloaded (180224/446937 bytes)
[task 2025-01-27T12:14:46.245Z] [downloads] 46% downloaded (204800/446937 bytes)
[task 2025-01-27T12:14:46.247Z] [downloads] 51% downloaded (229376/446937 bytes)
[task 2025-01-27T12:14:46.299Z] [downloads] 57% downloaded (253952/446937 bytes)
[task 2025-01-27T12:14:46.301Z] [downloads] 60% downloaded (270336/446937 bytes)
[task 2025-01-27T12:14:46.303Z] [downloads] 66% downloaded (294912/446937 bytes)
[task 2025-01-27T12:14:46.305Z] [downloads] 71% downloaded (319488/446937 bytes)
[task 2025-01-27T12:14:46.308Z] [downloads] 75% downloaded (335872/446937 bytes)
[task 2025-01-27T12:14:46.313Z] [downloads] 81% downloaded (360448/446937 bytes)
[task 2025-01-27T12:14:46.317Z] [downloads] 86% downloaded (385024/446937 bytes)
[task 2025-01-27T12:14:46.320Z] [downloads] 92% downloaded (409600/446937 bytes)
[task 2025-01-27T12:14:46.406Z] [downloads] 95% downloaded (425984/446937 bytes)
[task 2025-01-27T12:14:46.409Z] [downloads] 100% downloaded - Download finished.
[taskcluster 2025-01-27 12:14:53.185Z] === Task Finished ===
[taskcluster 2025-01-27 12:14:53.298Z] Successful task run with exit code: 0 completed in 196.203 seconds