Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: translations integration tests #210

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

remove `REV` from integration test task definitions

1bcb59e
Select commit
Loading
Failed to load commit list.
Open

feat: translations integration tests #210

remove `REV` from integration test task definitions
1bcb59e
Select commit
Loading
Failed to load commit list.
stage-taskcluster / translations-train-teacher-ru-en-1 succeeded Jan 27, 2025 in 1h 29m 2s

Stage-TC (issue_comment)

train teacher for ru-en 1

Details

View task in Taskcluster | View logs in Taskcluster | View task group in Taskcluster

Task Status

Started: 2025-01-27T13:20:45.291Z
Resolved: 2025-01-27T13:34:18.359Z
Task Execution Time: 13 minutes, 33 seconds, 68 milliseconds
Task Status: completed
Reason Resolved: completed
RunId: 0

Artifacts

- public/build/config.opustrainer.yml
- public/build/config.opustrainer.yml.state
- public/build/devset.out
- public/build/final.model.npz.best-chrf.npz
- public/build/final.model.npz.best-chrf.npz.decoder.yml
- public/build/model.npz
- public/build/model.npz.best-bleu-detok.npz
- public/build/model.npz.best-bleu-detok.npz.decoder.yml
- public/build/model.npz.best-ce-mean-words.npz
- public/build/model.npz.best-ce-mean-words.npz.decoder.yml
- public/build/model.npz.best-chrf.npz
- public/build/model.npz.best-chrf.npz.decoder.yml
- public/build/model.npz.decoder.yml
- public/build/model.npz.optimizer.npz
- public/build/model.npz.progress.yml
- public/build/model.npz.yml
- public/build/opustrainer.log
- public/build/train.log
- public/build/valid.log
- public/build/vocab.spm
- public/logs/live_backing.log
- public/logs/live.log


[taskcluster 2025-01-27T13:20:45.396Z] Worker Type (translations-1/b-linux-v100-gpu-4) settings:
[taskcluster 2025-01-27T13:20:45.396Z]   {
[taskcluster 2025-01-27T13:20:45.396Z]     "config": {
[taskcluster 2025-01-27T13:20:45.396Z]       "deploymentId": ""
[taskcluster 2025-01-27T13:20:45.396Z]     },
[taskcluster 2025-01-27T13:20:45.396Z]     "generic-worker": {
[taskcluster 2025-01-27T13:20:45.396Z]       "engine": "insecure",
[taskcluster 2025-01-27T13:20:45.396Z]       "go-arch": "amd64",
[taskcluster 2025-01-27T13:20:45.396Z]       "go-os": "linux",
[taskcluster 2025-01-27T13:20:45.396Z]       "go-version": "go1.22.2",
[taskcluster 2025-01-27T13:20:45.396Z]       "release": "https://github.com/taskcluster/taskcluster/releases/tag/v64.2.6",
[taskcluster 2025-01-27T13:20:45.396Z]       "revision": "edab196d7d030a5d625b77335109cd9060ab7e1f",
[taskcluster 2025-01-27T13:20:45.396Z]       "source": "https://github.com/taskcluster/taskcluster/commits/edab196d7d030a5d625b77335109cd9060ab7e1f",
[taskcluster 2025-01-27T13:20:45.396Z]       "version": "64.2.6"
[taskcluster 2025-01-27T13:20:45.396Z]     },
[taskcluster 2025-01-27T13:20:45.396Z]     "image": "projects/taskcluster-imaging/global/images/gw-translations-gcp-googlecompute-2024-04-22t18-22-42z",
[taskcluster 2025-01-27T13:20:45.396Z]     "instance-id": "1102761843089881987",
[taskcluster 2025-01-27T13:20:45.396Z]     "instance-type": "projects/474530013412/machineTypes/n1-highmem-8",
[taskcluster 2025-01-27T13:20:45.396Z]     "local-ipv4": "10.142.0.22",
[taskcluster 2025-01-27T13:20:45.396Z]     "project-id": "fxci-staging-level1-workers",

...(2811 lines hidden)...

[task 2025-01-27T13:33:09.656Z] [2025-01-27 13:33:09] [Trainer] [INFO] Reading original for epoch 1845
[task 2025-01-27T13:33:09.923Z] [2025-01-27 13:33:09] [Trainer] [INFO] Reading original for epoch 1846
[task 2025-01-27T13:33:10.196Z] [2025-01-27 13:33:10] [Trainer] [INFO] Reading original for epoch 1847
[task 2025-01-27T13:33:10.454Z] [2025-01-27 13:33:10] [Trainer] [INFO] Reading original for epoch 1848
[task 2025-01-27T13:33:10.710Z] [2025-01-27 13:33:10] [Trainer] [INFO] Reading original for epoch 1849
[task 2025-01-27T13:33:10.974Z] [2025-01-27 13:33:10] [Trainer] [INFO] Reading original for epoch 1850
[task 2025-01-27T13:33:11.256Z] [2025-01-27 13:33:11] [Trainer] [INFO] Reading original for epoch 1851
[task 2025-01-27T13:33:11.532Z] [2025-01-27 13:33:11] [Trainer] [INFO] Reading original for epoch 1852
[task 2025-01-27T13:33:11.820Z] [2025-01-27 13:33:11] [Trainer] [INFO] Reading original for epoch 1853
[task 2025-01-27T13:33:12.094Z] [2025-01-27 13:33:12] [Trainer] [INFO] Reading original for epoch 1854
[task 2025-01-27T13:33:12.375Z] [2025-01-27 13:33:12] [Trainer] [INFO] Reading original for epoch 1855
[task 2025-01-27T13:33:12.640Z] [2025-01-27 13:33:12] [Trainer] [INFO] Reading original for epoch 1856
[task 2025-01-27T13:33:12.912Z] [2025-01-27 13:33:12] [Trainer] [INFO] Reading original for epoch 1857
[task 2025-01-27T13:33:13.192Z] [2025-01-27 13:33:13] [Trainer] [INFO] Reading original for epoch 1858
[task 2025-01-27T13:33:13.461Z] [2025-01-27 13:33:13] [Trainer] [INFO] Reading original for epoch 1859
[task 2025-01-27T13:33:13.739Z] [2025-01-27 13:33:13] [Trainer] [INFO] Reading original for epoch 1860
[task 2025-01-27T13:33:14.015Z] [2025-01-27 13:33:14] [Trainer] [INFO] Reading original for epoch 1861
[task 2025-01-27T13:33:14.287Z] [2025-01-27 13:33:14] [Trainer] [INFO] Reading original for epoch 1862
[task 2025-01-27T13:33:14.553Z] [2025-01-27 13:33:14] [Trainer] [INFO] Reading original for epoch 1863
[task 2025-01-27T13:33:14.839Z] [2025-01-27 13:33:14] [Trainer] [INFO] Reading original for epoch 1864
[task 2025-01-27T13:33:15.112Z] [2025-01-27 13:33:15] [Trainer] [INFO] Reading original for epoch 1865
[task 2025-01-27T13:33:15.380Z] [2025-01-27 13:33:15] [Trainer] [INFO] Reading original for epoch 1866
[task 2025-01-27T13:33:15.649Z] [2025-01-27 13:33:15] [Trainer] [INFO] Reading original for epoch 1867
[task 2025-01-27T13:33:15.916Z] [2025-01-27 13:33:15] [Trainer] [INFO] Reading original for epoch 1868
[task 2025-01-27T13:33:16.009Z] [2025-01-27 13:33:16] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:16.015Z] [2025-01-27 13:33:16] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:16.188Z] [2025-01-27 13:33:16] [Trainer] [INFO] Reading original for epoch 1869
[task 2025-01-27T13:33:16.467Z] [2025-01-27 13:33:16] [Trainer] [INFO] Reading original for epoch 1870
[task 2025-01-27T13:33:16.757Z] [2025-01-27 13:33:16] [Trainer] [INFO] Reading original for epoch 1871
[task 2025-01-27T13:33:17.037Z] [2025-01-27 13:33:17] [Trainer] [INFO] Reading original for epoch 1872
[task 2025-01-27T13:33:17.320Z] [2025-01-27 13:33:17] [Trainer] [INFO] Reading original for epoch 1873
[task 2025-01-27T13:33:17.360Z] [2025-01-27 13:33:17] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:17.365Z] [2025-01-27 13:33:17] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:17.607Z] [2025-01-27 13:33:17] [Trainer] [INFO] Reading original for epoch 1874
[task 2025-01-27T13:33:17.869Z] [2025-01-27 13:33:17] [Trainer] [INFO] Reading original for epoch 1875
[task 2025-01-27T13:33:18.143Z] [2025-01-27 13:33:18] [Trainer] [INFO] Reading original for epoch 1876
[task 2025-01-27T13:33:18.411Z] [2025-01-27 13:33:18] [Trainer] [INFO] Reading original for epoch 1877
[task 2025-01-27T13:33:18.690Z] [2025-01-27 13:33:18] [Trainer] [INFO] Reading original for epoch 1878
[task 2025-01-27T13:33:18.968Z] [2025-01-27 13:33:18] [Trainer] [INFO] Reading original for epoch 1879
[task 2025-01-27T13:33:19.231Z] [2025-01-27 13:33:19] [Trainer] [INFO] Reading original for epoch 1880
[task 2025-01-27T13:33:19.502Z] [2025-01-27 13:33:19] [Trainer] [INFO] Reading original for epoch 1881
[task 2025-01-27T13:33:19.778Z] [2025-01-27 13:33:19] [Trainer] [INFO] Reading original for epoch 1882
[task 2025-01-27T13:33:20.057Z] [2025-01-27 13:33:20] [Trainer] [INFO] Reading original for epoch 1883
[task 2025-01-27T13:33:20.331Z] [2025-01-27 13:33:20] [Trainer] [INFO] Reading original for epoch 1884
[task 2025-01-27T13:33:20.593Z] [2025-01-27 13:33:20] [Trainer] [INFO] Reading original for epoch 1885
[task 2025-01-27T13:33:20.869Z] [2025-01-27 13:33:20] [Trainer] [INFO] Reading original for epoch 1886
[task 2025-01-27T13:33:21.143Z] [2025-01-27 13:33:21] [Trainer] [INFO] Reading original for epoch 1887
[task 2025-01-27T13:33:21.417Z] [2025-01-27 13:33:21] [Trainer] [INFO] Reading original for epoch 1888
[task 2025-01-27T13:33:21.691Z] [2025-01-27 13:33:21] [Trainer] [INFO] Reading original for epoch 1889
[task 2025-01-27T13:33:21.981Z] [2025-01-27 13:33:21] [Trainer] [INFO] Reading original for epoch 1890
[task 2025-01-27T13:33:22.268Z] [2025-01-27 13:33:22] [Trainer] [INFO] Reading original for epoch 1891
[task 2025-01-27T13:33:22.535Z] [2025-01-27 13:33:22] [Trainer] [INFO] Reading original for epoch 1892
[task 2025-01-27T13:33:22.794Z] [2025-01-27 13:33:22] [Trainer] [INFO] Reading original for epoch 1893
[task 2025-01-27T13:33:23.067Z] [2025-01-27 13:33:23] [Trainer] [INFO] Reading original for epoch 1894
[task 2025-01-27T13:33:23.330Z] [2025-01-27 13:33:23] [Trainer] [INFO] Reading original for epoch 1895
[task 2025-01-27T13:33:23.603Z] [2025-01-27 13:33:23] [Trainer] [INFO] Reading original for epoch 1896
[task 2025-01-27T13:33:23.880Z] [2025-01-27 13:33:23] [Trainer] [INFO] Reading original for epoch 1897
[task 2025-01-27T13:33:24.142Z] [2025-01-27 13:33:24] [Trainer] [INFO] Reading original for epoch 1898
[task 2025-01-27T13:33:24.415Z] [2025-01-27 13:33:24] [Trainer] [INFO] Reading original for epoch 1899
[task 2025-01-27T13:33:24.696Z] [2025-01-27 13:33:24] [Trainer] [INFO] Reading original for epoch 1900
[task 2025-01-27T13:33:24.970Z] [2025-01-27 13:33:24] [Trainer] [INFO] Reading original for epoch 1901
[task 2025-01-27T13:33:25.254Z] [2025-01-27 13:33:25] [Trainer] [INFO] Reading original for epoch 1902
[task 2025-01-27T13:33:25.523Z] [2025-01-27 13:33:25] [Trainer] [INFO] Reading original for epoch 1903
[task 2025-01-27T13:33:25.794Z] [2025-01-27 13:33:25] [Trainer] [INFO] Reading original for epoch 1904
[task 2025-01-27T13:33:26.064Z] [2025-01-27 13:33:26] [Trainer] [INFO] Reading original for epoch 1905
[task 2025-01-27T13:33:26.342Z] [2025-01-27 13:33:26] [Trainer] [INFO] Reading original for epoch 1906
[task 2025-01-27T13:33:26.627Z] [2025-01-27 13:33:26] [Trainer] [INFO] Reading original for epoch 1907
[task 2025-01-27T13:33:26.907Z] [2025-01-27 13:33:26] [Trainer] [INFO] Reading original for epoch 1908
[task 2025-01-27T13:33:27.195Z] [2025-01-27 13:33:27] [Trainer] [INFO] Reading original for epoch 1909
[task 2025-01-27T13:33:27.458Z] [2025-01-27 13:33:27] [Trainer] [INFO] Reading original for epoch 1910
[task 2025-01-27T13:33:27.737Z] [2025-01-27 13:33:27] [Trainer] [INFO] Reading original for epoch 1911
[task 2025-01-27T13:33:28.007Z] [2025-01-27 13:33:28] [Trainer] [INFO] Reading original for epoch 1912
[task 2025-01-27T13:33:28.297Z] [2025-01-27 13:33:28] [Trainer] [INFO] Reading original for epoch 1913
[task 2025-01-27T13:33:28.574Z] [2025-01-27 13:33:28] [Trainer] [INFO] Reading original for epoch 1914
[task 2025-01-27T13:33:28.845Z] [2025-01-27 13:33:28] [Trainer] [INFO] Reading original for epoch 1915
[task 2025-01-27T13:33:29.116Z] [2025-01-27 13:33:29] [Trainer] [INFO] Reading original for epoch 1916
[task 2025-01-27T13:33:29.397Z] [2025-01-27 13:33:29] [Trainer] [INFO] Reading original for epoch 1917
[task 2025-01-27T13:33:29.666Z] [2025-01-27 13:33:29] [Trainer] [INFO] Reading original for epoch 1918
[task 2025-01-27T13:33:29.953Z] [2025-01-27 13:33:29] [Trainer] [INFO] Reading original for epoch 1919
[task 2025-01-27T13:33:30.226Z] [2025-01-27 13:33:30] [Trainer] [INFO] Reading original for epoch 1920
[task 2025-01-27T13:33:30.509Z] [2025-01-27 13:33:30] [Trainer] [INFO] Reading original for epoch 1921
[task 2025-01-27T13:33:30.796Z] [2025-01-27 13:33:30] [Trainer] [INFO] Reading original for epoch 1922
[task 2025-01-27T13:33:31.071Z] [2025-01-27 13:33:31] [Trainer] [INFO] Reading original for epoch 1923
[task 2025-01-27T13:33:31.347Z] [2025-01-27 13:33:31] [Trainer] [INFO] Reading original for epoch 1924
[task 2025-01-27T13:33:31.625Z] [2025-01-27 13:33:31] [Trainer] [INFO] Reading original for epoch 1925
[task 2025-01-27T13:33:31.893Z] [2025-01-27 13:33:31] [Trainer] [INFO] Reading original for epoch 1926
[task 2025-01-27T13:33:32.161Z] [2025-01-27 13:33:32] [Trainer] [INFO] Reading original for epoch 1927
[task 2025-01-27T13:33:32.203Z] [2025-01-27 13:33:32] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:32.204Z] [2025-01-27 13:33:32] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:32.424Z] [2025-01-27 13:33:32] [Trainer] [INFO] Reading original for epoch 1928
[task 2025-01-27T13:33:32.690Z] [2025-01-27 13:33:32] [Trainer] [INFO] Reading original for epoch 1929
[task 2025-01-27T13:33:32.968Z] [2025-01-27 13:33:32] [Trainer] [INFO] Reading original for epoch 1930
[task 2025-01-27T13:33:33.270Z] [2025-01-27 13:33:33] [Trainer] [INFO] Reading original for epoch 1931
[task 2025-01-27T13:33:33.548Z] [2025-01-27 13:33:33] [Trainer] [INFO] Reading original for epoch 1932
[task 2025-01-27T13:33:33.820Z] [2025-01-27 13:33:33] [Trainer] [INFO] Reading original for epoch 1933
[task 2025-01-27T13:33:34.090Z] [2025-01-27 13:33:34] [Trainer] [INFO] Reading original for epoch 1934
[task 2025-01-27T13:33:34.360Z] [2025-01-27 13:33:34] [Trainer] [INFO] Reading original for epoch 1935
[task 2025-01-27T13:33:34.629Z] [2025-01-27 13:33:34] [Trainer] [INFO] Reading original for epoch 1936
[task 2025-01-27T13:33:34.901Z] [2025-01-27 13:33:34] [Trainer] [INFO] Reading original for epoch 1937
[task 2025-01-27T13:33:35.176Z] [2025-01-27 13:33:35] [Trainer] [INFO] Reading original for epoch 1938
[task 2025-01-27T13:33:35.450Z] [2025-01-27 13:33:35] [Trainer] [INFO] Reading original for epoch 1939
[task 2025-01-27T13:33:35.723Z] [2025-01-27 13:33:35] [Trainer] [INFO] Reading original for epoch 1940
[task 2025-01-27T13:33:36.003Z] [2025-01-27 13:33:36] [Trainer] [INFO] Reading original for epoch 1941
[task 2025-01-27T13:33:36.280Z] [2025-01-27 13:33:36] [Trainer] [INFO] Reading original for epoch 1942
[task 2025-01-27T13:33:36.565Z] [2025-01-27 13:33:36] [Trainer] [INFO] Reading original for epoch 1943
[task 2025-01-27T13:33:36.858Z] [2025-01-27 13:33:36] [Trainer] [INFO] Reading original for epoch 1944
[task 2025-01-27T13:33:37.147Z] [2025-01-27 13:33:37] [Trainer] [INFO] Reading original for epoch 1945
[task 2025-01-27T13:33:37.425Z] [2025-01-27 13:33:37] [Trainer] [INFO] Reading original for epoch 1946
[task 2025-01-27T13:33:37.704Z] [2025-01-27 13:33:37] [Trainer] [INFO] Reading original for epoch 1947
[task 2025-01-27T13:33:37.985Z] [2025-01-27 13:33:37] [Trainer] [INFO] Reading original for epoch 1948
[task 2025-01-27T13:33:38.262Z] [2025-01-27 13:33:38] [Trainer] [INFO] Reading original for epoch 1949
[task 2025-01-27T13:33:38.536Z] [2025-01-27 13:33:38] [Trainer] [INFO] Reading original for epoch 1950
[task 2025-01-27T13:33:38.818Z] [2025-01-27 13:33:38] [Trainer] [INFO] Reading original for epoch 1951
[task 2025-01-27T13:33:39.088Z] [2025-01-27 13:33:39] [Trainer] [INFO] Reading original for epoch 1952
[task 2025-01-27T13:33:39.368Z] [2025-01-27 13:33:39] [Trainer] [INFO] Reading original for epoch 1953
[task 2025-01-27T13:33:39.641Z] [2025-01-27 13:33:39] [Trainer] [INFO] Reading original for epoch 1954
[task 2025-01-27T13:33:39.897Z] [2025-01-27 13:33:39] [Trainer] [INFO] Reading original for epoch 1955
[task 2025-01-27T13:33:40.174Z] [2025-01-27 13:33:40] [Trainer] [INFO] Reading original for epoch 1956
[task 2025-01-27T13:33:40.443Z] [2025-01-27 13:33:40] [Trainer] [INFO] Reading original for epoch 1957
[task 2025-01-27T13:33:40.727Z] [2025-01-27 13:33:40] [Trainer] [INFO] Reading original for epoch 1958
[task 2025-01-27T13:33:41.009Z] [2025-01-27 13:33:41] [Trainer] [INFO] Reading original for epoch 1959
[task 2025-01-27T13:33:41.337Z] [2025-01-27 13:33:41] [Trainer] [INFO] Reading original for epoch 1960
[task 2025-01-27T13:33:41.610Z] [2025-01-27 13:33:41] [Trainer] [INFO] Reading original for epoch 1961
[task 2025-01-27T13:33:41.896Z] [2025-01-27 13:33:41] [Trainer] [INFO] Reading original for epoch 1962
[task 2025-01-27T13:33:42.162Z] [2025-01-27 13:33:42] [Trainer] [INFO] Reading original for epoch 1963
[task 2025-01-27T13:33:42.437Z] [2025-01-27 13:33:42] [Trainer] [INFO] Reading original for epoch 1964
[task 2025-01-27T13:33:42.711Z] [2025-01-27 13:33:42] [Trainer] [INFO] Reading original for epoch 1965
[task 2025-01-27T13:33:42.983Z] [2025-01-27 13:33:42] [Trainer] [INFO] Reading original for epoch 1966
[task 2025-01-27T13:33:43.252Z] [2025-01-27 13:33:43] [Trainer] [INFO] Reading original for epoch 1967
[task 2025-01-27T13:33:43.524Z] [2025-01-27 13:33:43] [Trainer] [INFO] Reading original for epoch 1968
[task 2025-01-27T13:33:43.800Z] [2025-01-27 13:33:43] [Trainer] [INFO] Reading original for epoch 1969
[task 2025-01-27T13:33:44.063Z] [2025-01-27 13:33:44] [Trainer] [INFO] Reading original for epoch 1970
[task 2025-01-27T13:33:44.336Z] [2025-01-27 13:33:44] [Trainer] [INFO] Reading original for epoch 1971
[task 2025-01-27T13:33:44.619Z] [2025-01-27 13:33:44] [Trainer] [INFO] Reading original for epoch 1972
[task 2025-01-27T13:33:44.888Z] [2025-01-27 13:33:44] [Trainer] [INFO] Reading original for epoch 1973
[task 2025-01-27T13:33:45.169Z] [2025-01-27 13:33:45] [Trainer] [INFO] Reading original for epoch 1974
[task 2025-01-27T13:33:45.444Z] [2025-01-27 13:33:45] [Trainer] [INFO] Reading original for epoch 1975
[task 2025-01-27T13:33:45.716Z] [2025-01-27 13:33:45] [Trainer] [INFO] Reading original for epoch 1976
[task 2025-01-27T13:33:46.001Z] [2025-01-27 13:33:46] [Trainer] [INFO] Reading original for epoch 1977
[task 2025-01-27T13:33:46.283Z] [2025-01-27 13:33:46] [Trainer] [INFO] Reading original for epoch 1978
[task 2025-01-27T13:33:46.589Z] [2025-01-27 13:33:46] [Trainer] [INFO] Reading original for epoch 1979
[task 2025-01-27T13:33:46.892Z] [2025-01-27 13:33:46] [Trainer] [INFO] Reading original for epoch 1980
[task 2025-01-27T13:33:47.168Z] [2025-01-27 13:33:47] [Trainer] [INFO] Reading original for epoch 1981
[task 2025-01-27T13:33:47.441Z] [2025-01-27 13:33:47] [Trainer] [INFO] Reading original for epoch 1982
[task 2025-01-27T13:33:47.692Z] [2025-01-27 13:33:47] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:47.698Z] [2025-01-27 13:33:47] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:47.707Z] [2025-01-27 13:33:47] [Trainer] [INFO] Reading original for epoch 1983
[task 2025-01-27T13:33:47.984Z] [2025-01-27 13:33:47] [Trainer] [INFO] Reading original for epoch 1984
[task 2025-01-27T13:33:48.254Z] [2025-01-27 13:33:48] [Trainer] [INFO] Reading original for epoch 1985
[task 2025-01-27T13:33:48.527Z] [2025-01-27 13:33:48] [Trainer] [INFO] Reading original for epoch 1986
[task 2025-01-27T13:33:48.823Z] [2025-01-27 13:33:48] [Trainer] [INFO] Reading original for epoch 1987
[task 2025-01-27T13:33:49.102Z] [2025-01-27 13:33:49] [Trainer] [INFO] Reading original for epoch 1988
[task 2025-01-27T13:33:49.383Z] [2025-01-27 13:33:49] [Trainer] [INFO] Reading original for epoch 1989
[task 2025-01-27T13:33:49.654Z] [2025-01-27 13:33:49] [Trainer] [INFO] Reading original for epoch 1990
[task 2025-01-27T13:33:49.926Z] [2025-01-27 13:33:49] [Trainer] [INFO] Reading original for epoch 1991
[task 2025-01-27T13:33:50.200Z] [2025-01-27 13:33:50] [Trainer] [INFO] Reading original for epoch 1992
[task 2025-01-27T13:33:50.466Z] [2025-01-27 13:33:50] [Trainer] [INFO] Reading original for epoch 1993
[task 2025-01-27T13:33:50.745Z] [2025-01-27 13:33:50] [Trainer] [INFO] Reading original for epoch 1994
[task 2025-01-27T13:33:51.014Z] [2025-01-27 13:33:51] [Trainer] [INFO] Reading original for epoch 1995
[task 2025-01-27T13:33:51.018Z] [2025-01-27 13:33:51] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:51.024Z] [2025-01-27 13:33:51] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:51.272Z] [2025-01-27 13:33:51] [Trainer] [INFO] Reading original for epoch 1996
[task 2025-01-27T13:33:51.530Z] [2025-01-27 13:33:51] [Trainer] [INFO] Reading original for epoch 1997
[task 2025-01-27T13:33:51.798Z] [2025-01-27 13:33:51] [Trainer] [INFO] Reading original for epoch 1998
[task 2025-01-27T13:33:58.378Z] [2025-01-27 13:33:58] [Trainer] [INFO] trainer stopped reading input
[task 2025-01-27T13:33:58.478Z] [tracking INFO] Successfully parsed 2526 lines
[task 2025-01-27T13:33:58.478Z] [tracking INFO] Found 50 training entries
[task 2025-01-27T13:33:58.478Z] [tracking INFO] Found 1 validation entries
[fetches 2025-01-27T13:33:58.780Z] removing /home/ubuntu/tasks/task_173798376721600/fetches
[fetches 2025-01-27T13:33:59.134Z] finished
[taskcluster 2025-01-27T13:33:59.145Z]    Exit Code: 0
[taskcluster 2025-01-27T13:33:59.145Z]    User Time: 1h2m46.859518s
[taskcluster 2025-01-27T13:33:59.145Z]  Kernel Time: 1m28.661655s
[taskcluster 2025-01-27T13:33:59.145Z]    Wall Time: 13m13.252199309s
[taskcluster 2025-01-27T13:33:59.145Z]       Result: SUCCEEDED
[taskcluster 2025-01-27T13:33:59.146Z] === Task Finished ===
[taskcluster 2025-01-27T13:33:59.146Z] Task Duration: 13m13.254189288s
[taskcluster 2025-01-27T13:33:59.361Z] Uploading artifact public/build/config.opustrainer.yml.state from file /home/ubuntu/tasks/task_173798376721600/artifacts/config.opustrainer.yml.state with content encoding "gzip", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.389Z] Uploading artifact public/build/final.model.npz.best-chrf.npz from file /home/ubuntu/tasks/task_173798376721600/artifacts/final.model.npz.best-chrf.npz with content encoding "identity", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.406Z] Uploading artifact public/build/devset.out from file /home/ubuntu/tasks/task_173798376721600/artifacts/devset.out with content encoding "gzip", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.422Z] Uploading artifact public/build/vocab.spm from file /home/ubuntu/tasks/task_173798376721600/artifacts/vocab.spm with content encoding "gzip", mime type "application/x-source-rpm" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.433Z] Uploading artifact public/build/final.model.npz.best-chrf.npz.decoder.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/final.model.npz.best-chrf.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.437Z] Uploading artifact public/build/model.npz.best-chrf.npz.decoder.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.best-chrf.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.450Z] Uploading artifact public/build/model.npz.best-bleu-detok.npz from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.best-bleu-detok.npz with content encoding "identity", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.492Z] Uploading artifact public/build/model.npz.best-ce-mean-words.npz.decoder.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.best-ce-mean-words.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.500Z] Uploading artifact public/build/model.npz.decoder.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.504Z] Uploading artifact public/build/model.npz.best-bleu-detok.npz.decoder.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.best-bleu-detok.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.508Z] Uploading artifact public/build/model.npz from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz with content encoding "identity", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.524Z] Uploading artifact public/build/model.npz.best-chrf.npz from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.best-chrf.npz with content encoding "identity", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.524Z] Uploading artifact public/build/model.npz.optimizer.npz from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.optimizer.npz with content encoding "identity", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.643Z] Uploading artifact public/build/model.npz.best-ce-mean-words.npz from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.best-ce-mean-words.npz with content encoding "identity", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.644Z] Uploading artifact public/build/model.npz.progress.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.progress.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.644Z] Uploading artifact public/build/opustrainer.log from file /home/ubuntu/tasks/task_173798376721600/artifacts/opustrainer.log with content encoding "gzip", mime type "text/plain" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.644Z] Uploading artifact public/build/model.npz.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.645Z] Uploading artifact public/build/train.log from file /home/ubuntu/tasks/task_173798376721600/artifacts/train.log with content encoding "gzip", mime type "text/plain" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.644Z] Uploading artifact public/build/config.opustrainer.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/config.opustrainer.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.644Z] Uploading artifact public/build/valid.log from file /home/ubuntu/tasks/task_173798376721600/artifacts/valid.log with content encoding "gzip", mime type "text/plain" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:34:17.831Z] [mounts] Preserving cache: Moving "/home/ubuntu/tasks/task_173798376721600/checkouts" to "/home/ubuntu/caches/I3AP_eWSSS6gKC11ZNTBeg"
[taskcluster 2025-01-27T13:34:17.997Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2025-02-26T12:05:14.661Z