feat: translations integration tests #210
Stage-TC (issue_comment)
train teacher for ru-en 1
Details
View task in Taskcluster | View logs in Taskcluster | View task group in Taskcluster
Task Status
Started: 2025-01-27T13:20:45.291Z
Resolved: 2025-01-27T13:34:18.359Z
Task Execution Time: 13 minutes, 33 seconds, 68 milliseconds
Task Status: completed
Reason Resolved: completed
RunId: 0
Artifacts
- public/build/config.opustrainer.yml
- public/build/config.opustrainer.yml.state
- public/build/devset.out
- public/build/final.model.npz.best-chrf.npz
- public/build/final.model.npz.best-chrf.npz.decoder.yml
- public/build/model.npz
- public/build/model.npz.best-bleu-detok.npz
- public/build/model.npz.best-bleu-detok.npz.decoder.yml
- public/build/model.npz.best-ce-mean-words.npz
- public/build/model.npz.best-ce-mean-words.npz.decoder.yml
- public/build/model.npz.best-chrf.npz
- public/build/model.npz.best-chrf.npz.decoder.yml
- public/build/model.npz.decoder.yml
- public/build/model.npz.optimizer.npz
- public/build/model.npz.progress.yml
- public/build/model.npz.yml
- public/build/opustrainer.log
- public/build/train.log
- public/build/valid.log
- public/build/vocab.spm
- public/logs/live_backing.log
- public/logs/live.log
[taskcluster 2025-01-27T13:20:45.396Z] Worker Type (translations-1/b-linux-v100-gpu-4) settings:
[taskcluster 2025-01-27T13:20:45.396Z] {
[taskcluster 2025-01-27T13:20:45.396Z] "config": {
[taskcluster 2025-01-27T13:20:45.396Z] "deploymentId": ""
[taskcluster 2025-01-27T13:20:45.396Z] },
[taskcluster 2025-01-27T13:20:45.396Z] "generic-worker": {
[taskcluster 2025-01-27T13:20:45.396Z] "engine": "insecure",
[taskcluster 2025-01-27T13:20:45.396Z] "go-arch": "amd64",
[taskcluster 2025-01-27T13:20:45.396Z] "go-os": "linux",
[taskcluster 2025-01-27T13:20:45.396Z] "go-version": "go1.22.2",
[taskcluster 2025-01-27T13:20:45.396Z] "release": "https://github.com/taskcluster/taskcluster/releases/tag/v64.2.6",
[taskcluster 2025-01-27T13:20:45.396Z] "revision": "edab196d7d030a5d625b77335109cd9060ab7e1f",
[taskcluster 2025-01-27T13:20:45.396Z] "source": "https://github.com/taskcluster/taskcluster/commits/edab196d7d030a5d625b77335109cd9060ab7e1f",
[taskcluster 2025-01-27T13:20:45.396Z] "version": "64.2.6"
[taskcluster 2025-01-27T13:20:45.396Z] },
[taskcluster 2025-01-27T13:20:45.396Z] "image": "projects/taskcluster-imaging/global/images/gw-translations-gcp-googlecompute-2024-04-22t18-22-42z",
[taskcluster 2025-01-27T13:20:45.396Z] "instance-id": "1102761843089881987",
[taskcluster 2025-01-27T13:20:45.396Z] "instance-type": "projects/474530013412/machineTypes/n1-highmem-8",
[taskcluster 2025-01-27T13:20:45.396Z] "local-ipv4": "10.142.0.22",
[taskcluster 2025-01-27T13:20:45.396Z] "project-id": "fxci-staging-level1-workers",
...(2811 lines hidden)...
[task 2025-01-27T13:33:09.656Z] [2025-01-27 13:33:09] [Trainer] [INFO] Reading original for epoch 1845
[task 2025-01-27T13:33:09.923Z] [2025-01-27 13:33:09] [Trainer] [INFO] Reading original for epoch 1846
[task 2025-01-27T13:33:10.196Z] [2025-01-27 13:33:10] [Trainer] [INFO] Reading original for epoch 1847
[task 2025-01-27T13:33:10.454Z] [2025-01-27 13:33:10] [Trainer] [INFO] Reading original for epoch 1848
[task 2025-01-27T13:33:10.710Z] [2025-01-27 13:33:10] [Trainer] [INFO] Reading original for epoch 1849
[task 2025-01-27T13:33:10.974Z] [2025-01-27 13:33:10] [Trainer] [INFO] Reading original for epoch 1850
[task 2025-01-27T13:33:11.256Z] [2025-01-27 13:33:11] [Trainer] [INFO] Reading original for epoch 1851
[task 2025-01-27T13:33:11.532Z] [2025-01-27 13:33:11] [Trainer] [INFO] Reading original for epoch 1852
[task 2025-01-27T13:33:11.820Z] [2025-01-27 13:33:11] [Trainer] [INFO] Reading original for epoch 1853
[task 2025-01-27T13:33:12.094Z] [2025-01-27 13:33:12] [Trainer] [INFO] Reading original for epoch 1854
[task 2025-01-27T13:33:12.375Z] [2025-01-27 13:33:12] [Trainer] [INFO] Reading original for epoch 1855
[task 2025-01-27T13:33:12.640Z] [2025-01-27 13:33:12] [Trainer] [INFO] Reading original for epoch 1856
[task 2025-01-27T13:33:12.912Z] [2025-01-27 13:33:12] [Trainer] [INFO] Reading original for epoch 1857
[task 2025-01-27T13:33:13.192Z] [2025-01-27 13:33:13] [Trainer] [INFO] Reading original for epoch 1858
[task 2025-01-27T13:33:13.461Z] [2025-01-27 13:33:13] [Trainer] [INFO] Reading original for epoch 1859
[task 2025-01-27T13:33:13.739Z] [2025-01-27 13:33:13] [Trainer] [INFO] Reading original for epoch 1860
[task 2025-01-27T13:33:14.015Z] [2025-01-27 13:33:14] [Trainer] [INFO] Reading original for epoch 1861
[task 2025-01-27T13:33:14.287Z] [2025-01-27 13:33:14] [Trainer] [INFO] Reading original for epoch 1862
[task 2025-01-27T13:33:14.553Z] [2025-01-27 13:33:14] [Trainer] [INFO] Reading original for epoch 1863
[task 2025-01-27T13:33:14.839Z] [2025-01-27 13:33:14] [Trainer] [INFO] Reading original for epoch 1864
[task 2025-01-27T13:33:15.112Z] [2025-01-27 13:33:15] [Trainer] [INFO] Reading original for epoch 1865
[task 2025-01-27T13:33:15.380Z] [2025-01-27 13:33:15] [Trainer] [INFO] Reading original for epoch 1866
[task 2025-01-27T13:33:15.649Z] [2025-01-27 13:33:15] [Trainer] [INFO] Reading original for epoch 1867
[task 2025-01-27T13:33:15.916Z] [2025-01-27 13:33:15] [Trainer] [INFO] Reading original for epoch 1868
[task 2025-01-27T13:33:16.009Z] [2025-01-27 13:33:16] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:16.015Z] [2025-01-27 13:33:16] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:16.188Z] [2025-01-27 13:33:16] [Trainer] [INFO] Reading original for epoch 1869
[task 2025-01-27T13:33:16.467Z] [2025-01-27 13:33:16] [Trainer] [INFO] Reading original for epoch 1870
[task 2025-01-27T13:33:16.757Z] [2025-01-27 13:33:16] [Trainer] [INFO] Reading original for epoch 1871
[task 2025-01-27T13:33:17.037Z] [2025-01-27 13:33:17] [Trainer] [INFO] Reading original for epoch 1872
[task 2025-01-27T13:33:17.320Z] [2025-01-27 13:33:17] [Trainer] [INFO] Reading original for epoch 1873
[task 2025-01-27T13:33:17.360Z] [2025-01-27 13:33:17] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:17.365Z] [2025-01-27 13:33:17] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:17.607Z] [2025-01-27 13:33:17] [Trainer] [INFO] Reading original for epoch 1874
[task 2025-01-27T13:33:17.869Z] [2025-01-27 13:33:17] [Trainer] [INFO] Reading original for epoch 1875
[task 2025-01-27T13:33:18.143Z] [2025-01-27 13:33:18] [Trainer] [INFO] Reading original for epoch 1876
[task 2025-01-27T13:33:18.411Z] [2025-01-27 13:33:18] [Trainer] [INFO] Reading original for epoch 1877
[task 2025-01-27T13:33:18.690Z] [2025-01-27 13:33:18] [Trainer] [INFO] Reading original for epoch 1878
[task 2025-01-27T13:33:18.968Z] [2025-01-27 13:33:18] [Trainer] [INFO] Reading original for epoch 1879
[task 2025-01-27T13:33:19.231Z] [2025-01-27 13:33:19] [Trainer] [INFO] Reading original for epoch 1880
[task 2025-01-27T13:33:19.502Z] [2025-01-27 13:33:19] [Trainer] [INFO] Reading original for epoch 1881
[task 2025-01-27T13:33:19.778Z] [2025-01-27 13:33:19] [Trainer] [INFO] Reading original for epoch 1882
[task 2025-01-27T13:33:20.057Z] [2025-01-27 13:33:20] [Trainer] [INFO] Reading original for epoch 1883
[task 2025-01-27T13:33:20.331Z] [2025-01-27 13:33:20] [Trainer] [INFO] Reading original for epoch 1884
[task 2025-01-27T13:33:20.593Z] [2025-01-27 13:33:20] [Trainer] [INFO] Reading original for epoch 1885
[task 2025-01-27T13:33:20.869Z] [2025-01-27 13:33:20] [Trainer] [INFO] Reading original for epoch 1886
[task 2025-01-27T13:33:21.143Z] [2025-01-27 13:33:21] [Trainer] [INFO] Reading original for epoch 1887
[task 2025-01-27T13:33:21.417Z] [2025-01-27 13:33:21] [Trainer] [INFO] Reading original for epoch 1888
[task 2025-01-27T13:33:21.691Z] [2025-01-27 13:33:21] [Trainer] [INFO] Reading original for epoch 1889
[task 2025-01-27T13:33:21.981Z] [2025-01-27 13:33:21] [Trainer] [INFO] Reading original for epoch 1890
[task 2025-01-27T13:33:22.268Z] [2025-01-27 13:33:22] [Trainer] [INFO] Reading original for epoch 1891
[task 2025-01-27T13:33:22.535Z] [2025-01-27 13:33:22] [Trainer] [INFO] Reading original for epoch 1892
[task 2025-01-27T13:33:22.794Z] [2025-01-27 13:33:22] [Trainer] [INFO] Reading original for epoch 1893
[task 2025-01-27T13:33:23.067Z] [2025-01-27 13:33:23] [Trainer] [INFO] Reading original for epoch 1894
[task 2025-01-27T13:33:23.330Z] [2025-01-27 13:33:23] [Trainer] [INFO] Reading original for epoch 1895
[task 2025-01-27T13:33:23.603Z] [2025-01-27 13:33:23] [Trainer] [INFO] Reading original for epoch 1896
[task 2025-01-27T13:33:23.880Z] [2025-01-27 13:33:23] [Trainer] [INFO] Reading original for epoch 1897
[task 2025-01-27T13:33:24.142Z] [2025-01-27 13:33:24] [Trainer] [INFO] Reading original for epoch 1898
[task 2025-01-27T13:33:24.415Z] [2025-01-27 13:33:24] [Trainer] [INFO] Reading original for epoch 1899
[task 2025-01-27T13:33:24.696Z] [2025-01-27 13:33:24] [Trainer] [INFO] Reading original for epoch 1900
[task 2025-01-27T13:33:24.970Z] [2025-01-27 13:33:24] [Trainer] [INFO] Reading original for epoch 1901
[task 2025-01-27T13:33:25.254Z] [2025-01-27 13:33:25] [Trainer] [INFO] Reading original for epoch 1902
[task 2025-01-27T13:33:25.523Z] [2025-01-27 13:33:25] [Trainer] [INFO] Reading original for epoch 1903
[task 2025-01-27T13:33:25.794Z] [2025-01-27 13:33:25] [Trainer] [INFO] Reading original for epoch 1904
[task 2025-01-27T13:33:26.064Z] [2025-01-27 13:33:26] [Trainer] [INFO] Reading original for epoch 1905
[task 2025-01-27T13:33:26.342Z] [2025-01-27 13:33:26] [Trainer] [INFO] Reading original for epoch 1906
[task 2025-01-27T13:33:26.627Z] [2025-01-27 13:33:26] [Trainer] [INFO] Reading original for epoch 1907
[task 2025-01-27T13:33:26.907Z] [2025-01-27 13:33:26] [Trainer] [INFO] Reading original for epoch 1908
[task 2025-01-27T13:33:27.195Z] [2025-01-27 13:33:27] [Trainer] [INFO] Reading original for epoch 1909
[task 2025-01-27T13:33:27.458Z] [2025-01-27 13:33:27] [Trainer] [INFO] Reading original for epoch 1910
[task 2025-01-27T13:33:27.737Z] [2025-01-27 13:33:27] [Trainer] [INFO] Reading original for epoch 1911
[task 2025-01-27T13:33:28.007Z] [2025-01-27 13:33:28] [Trainer] [INFO] Reading original for epoch 1912
[task 2025-01-27T13:33:28.297Z] [2025-01-27 13:33:28] [Trainer] [INFO] Reading original for epoch 1913
[task 2025-01-27T13:33:28.574Z] [2025-01-27 13:33:28] [Trainer] [INFO] Reading original for epoch 1914
[task 2025-01-27T13:33:28.845Z] [2025-01-27 13:33:28] [Trainer] [INFO] Reading original for epoch 1915
[task 2025-01-27T13:33:29.116Z] [2025-01-27 13:33:29] [Trainer] [INFO] Reading original for epoch 1916
[task 2025-01-27T13:33:29.397Z] [2025-01-27 13:33:29] [Trainer] [INFO] Reading original for epoch 1917
[task 2025-01-27T13:33:29.666Z] [2025-01-27 13:33:29] [Trainer] [INFO] Reading original for epoch 1918
[task 2025-01-27T13:33:29.953Z] [2025-01-27 13:33:29] [Trainer] [INFO] Reading original for epoch 1919
[task 2025-01-27T13:33:30.226Z] [2025-01-27 13:33:30] [Trainer] [INFO] Reading original for epoch 1920
[task 2025-01-27T13:33:30.509Z] [2025-01-27 13:33:30] [Trainer] [INFO] Reading original for epoch 1921
[task 2025-01-27T13:33:30.796Z] [2025-01-27 13:33:30] [Trainer] [INFO] Reading original for epoch 1922
[task 2025-01-27T13:33:31.071Z] [2025-01-27 13:33:31] [Trainer] [INFO] Reading original for epoch 1923
[task 2025-01-27T13:33:31.347Z] [2025-01-27 13:33:31] [Trainer] [INFO] Reading original for epoch 1924
[task 2025-01-27T13:33:31.625Z] [2025-01-27 13:33:31] [Trainer] [INFO] Reading original for epoch 1925
[task 2025-01-27T13:33:31.893Z] [2025-01-27 13:33:31] [Trainer] [INFO] Reading original for epoch 1926
[task 2025-01-27T13:33:32.161Z] [2025-01-27 13:33:32] [Trainer] [INFO] Reading original for epoch 1927
[task 2025-01-27T13:33:32.203Z] [2025-01-27 13:33:32] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:32.204Z] [2025-01-27 13:33:32] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:32.424Z] [2025-01-27 13:33:32] [Trainer] [INFO] Reading original for epoch 1928
[task 2025-01-27T13:33:32.690Z] [2025-01-27 13:33:32] [Trainer] [INFO] Reading original for epoch 1929
[task 2025-01-27T13:33:32.968Z] [2025-01-27 13:33:32] [Trainer] [INFO] Reading original for epoch 1930
[task 2025-01-27T13:33:33.270Z] [2025-01-27 13:33:33] [Trainer] [INFO] Reading original for epoch 1931
[task 2025-01-27T13:33:33.548Z] [2025-01-27 13:33:33] [Trainer] [INFO] Reading original for epoch 1932
[task 2025-01-27T13:33:33.820Z] [2025-01-27 13:33:33] [Trainer] [INFO] Reading original for epoch 1933
[task 2025-01-27T13:33:34.090Z] [2025-01-27 13:33:34] [Trainer] [INFO] Reading original for epoch 1934
[task 2025-01-27T13:33:34.360Z] [2025-01-27 13:33:34] [Trainer] [INFO] Reading original for epoch 1935
[task 2025-01-27T13:33:34.629Z] [2025-01-27 13:33:34] [Trainer] [INFO] Reading original for epoch 1936
[task 2025-01-27T13:33:34.901Z] [2025-01-27 13:33:34] [Trainer] [INFO] Reading original for epoch 1937
[task 2025-01-27T13:33:35.176Z] [2025-01-27 13:33:35] [Trainer] [INFO] Reading original for epoch 1938
[task 2025-01-27T13:33:35.450Z] [2025-01-27 13:33:35] [Trainer] [INFO] Reading original for epoch 1939
[task 2025-01-27T13:33:35.723Z] [2025-01-27 13:33:35] [Trainer] [INFO] Reading original for epoch 1940
[task 2025-01-27T13:33:36.003Z] [2025-01-27 13:33:36] [Trainer] [INFO] Reading original for epoch 1941
[task 2025-01-27T13:33:36.280Z] [2025-01-27 13:33:36] [Trainer] [INFO] Reading original for epoch 1942
[task 2025-01-27T13:33:36.565Z] [2025-01-27 13:33:36] [Trainer] [INFO] Reading original for epoch 1943
[task 2025-01-27T13:33:36.858Z] [2025-01-27 13:33:36] [Trainer] [INFO] Reading original for epoch 1944
[task 2025-01-27T13:33:37.147Z] [2025-01-27 13:33:37] [Trainer] [INFO] Reading original for epoch 1945
[task 2025-01-27T13:33:37.425Z] [2025-01-27 13:33:37] [Trainer] [INFO] Reading original for epoch 1946
[task 2025-01-27T13:33:37.704Z] [2025-01-27 13:33:37] [Trainer] [INFO] Reading original for epoch 1947
[task 2025-01-27T13:33:37.985Z] [2025-01-27 13:33:37] [Trainer] [INFO] Reading original for epoch 1948
[task 2025-01-27T13:33:38.262Z] [2025-01-27 13:33:38] [Trainer] [INFO] Reading original for epoch 1949
[task 2025-01-27T13:33:38.536Z] [2025-01-27 13:33:38] [Trainer] [INFO] Reading original for epoch 1950
[task 2025-01-27T13:33:38.818Z] [2025-01-27 13:33:38] [Trainer] [INFO] Reading original for epoch 1951
[task 2025-01-27T13:33:39.088Z] [2025-01-27 13:33:39] [Trainer] [INFO] Reading original for epoch 1952
[task 2025-01-27T13:33:39.368Z] [2025-01-27 13:33:39] [Trainer] [INFO] Reading original for epoch 1953
[task 2025-01-27T13:33:39.641Z] [2025-01-27 13:33:39] [Trainer] [INFO] Reading original for epoch 1954
[task 2025-01-27T13:33:39.897Z] [2025-01-27 13:33:39] [Trainer] [INFO] Reading original for epoch 1955
[task 2025-01-27T13:33:40.174Z] [2025-01-27 13:33:40] [Trainer] [INFO] Reading original for epoch 1956
[task 2025-01-27T13:33:40.443Z] [2025-01-27 13:33:40] [Trainer] [INFO] Reading original for epoch 1957
[task 2025-01-27T13:33:40.727Z] [2025-01-27 13:33:40] [Trainer] [INFO] Reading original for epoch 1958
[task 2025-01-27T13:33:41.009Z] [2025-01-27 13:33:41] [Trainer] [INFO] Reading original for epoch 1959
[task 2025-01-27T13:33:41.337Z] [2025-01-27 13:33:41] [Trainer] [INFO] Reading original for epoch 1960
[task 2025-01-27T13:33:41.610Z] [2025-01-27 13:33:41] [Trainer] [INFO] Reading original for epoch 1961
[task 2025-01-27T13:33:41.896Z] [2025-01-27 13:33:41] [Trainer] [INFO] Reading original for epoch 1962
[task 2025-01-27T13:33:42.162Z] [2025-01-27 13:33:42] [Trainer] [INFO] Reading original for epoch 1963
[task 2025-01-27T13:33:42.437Z] [2025-01-27 13:33:42] [Trainer] [INFO] Reading original for epoch 1964
[task 2025-01-27T13:33:42.711Z] [2025-01-27 13:33:42] [Trainer] [INFO] Reading original for epoch 1965
[task 2025-01-27T13:33:42.983Z] [2025-01-27 13:33:42] [Trainer] [INFO] Reading original for epoch 1966
[task 2025-01-27T13:33:43.252Z] [2025-01-27 13:33:43] [Trainer] [INFO] Reading original for epoch 1967
[task 2025-01-27T13:33:43.524Z] [2025-01-27 13:33:43] [Trainer] [INFO] Reading original for epoch 1968
[task 2025-01-27T13:33:43.800Z] [2025-01-27 13:33:43] [Trainer] [INFO] Reading original for epoch 1969
[task 2025-01-27T13:33:44.063Z] [2025-01-27 13:33:44] [Trainer] [INFO] Reading original for epoch 1970
[task 2025-01-27T13:33:44.336Z] [2025-01-27 13:33:44] [Trainer] [INFO] Reading original for epoch 1971
[task 2025-01-27T13:33:44.619Z] [2025-01-27 13:33:44] [Trainer] [INFO] Reading original for epoch 1972
[task 2025-01-27T13:33:44.888Z] [2025-01-27 13:33:44] [Trainer] [INFO] Reading original for epoch 1973
[task 2025-01-27T13:33:45.169Z] [2025-01-27 13:33:45] [Trainer] [INFO] Reading original for epoch 1974
[task 2025-01-27T13:33:45.444Z] [2025-01-27 13:33:45] [Trainer] [INFO] Reading original for epoch 1975
[task 2025-01-27T13:33:45.716Z] [2025-01-27 13:33:45] [Trainer] [INFO] Reading original for epoch 1976
[task 2025-01-27T13:33:46.001Z] [2025-01-27 13:33:46] [Trainer] [INFO] Reading original for epoch 1977
[task 2025-01-27T13:33:46.283Z] [2025-01-27 13:33:46] [Trainer] [INFO] Reading original for epoch 1978
[task 2025-01-27T13:33:46.589Z] [2025-01-27 13:33:46] [Trainer] [INFO] Reading original for epoch 1979
[task 2025-01-27T13:33:46.892Z] [2025-01-27 13:33:46] [Trainer] [INFO] Reading original for epoch 1980
[task 2025-01-27T13:33:47.168Z] [2025-01-27 13:33:47] [Trainer] [INFO] Reading original for epoch 1981
[task 2025-01-27T13:33:47.441Z] [2025-01-27 13:33:47] [Trainer] [INFO] Reading original for epoch 1982
[task 2025-01-27T13:33:47.692Z] [2025-01-27 13:33:47] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:47.698Z] [2025-01-27 13:33:47] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:47.707Z] [2025-01-27 13:33:47] [Trainer] [INFO] Reading original for epoch 1983
[task 2025-01-27T13:33:47.984Z] [2025-01-27 13:33:47] [Trainer] [INFO] Reading original for epoch 1984
[task 2025-01-27T13:33:48.254Z] [2025-01-27 13:33:48] [Trainer] [INFO] Reading original for epoch 1985
[task 2025-01-27T13:33:48.527Z] [2025-01-27 13:33:48] [Trainer] [INFO] Reading original for epoch 1986
[task 2025-01-27T13:33:48.823Z] [2025-01-27 13:33:48] [Trainer] [INFO] Reading original for epoch 1987
[task 2025-01-27T13:33:49.102Z] [2025-01-27 13:33:49] [Trainer] [INFO] Reading original for epoch 1988
[task 2025-01-27T13:33:49.383Z] [2025-01-27 13:33:49] [Trainer] [INFO] Reading original for epoch 1989
[task 2025-01-27T13:33:49.654Z] [2025-01-27 13:33:49] [Trainer] [INFO] Reading original for epoch 1990
[task 2025-01-27T13:33:49.926Z] [2025-01-27 13:33:49] [Trainer] [INFO] Reading original for epoch 1991
[task 2025-01-27T13:33:50.200Z] [2025-01-27 13:33:50] [Trainer] [INFO] Reading original for epoch 1992
[task 2025-01-27T13:33:50.466Z] [2025-01-27 13:33:50] [Trainer] [INFO] Reading original for epoch 1993
[task 2025-01-27T13:33:50.745Z] [2025-01-27 13:33:50] [Trainer] [INFO] Reading original for epoch 1994
[task 2025-01-27T13:33:51.014Z] [2025-01-27 13:33:51] [Trainer] [INFO] Reading original for epoch 1995
[task 2025-01-27T13:33:51.018Z] [2025-01-27 13:33:51] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:51.024Z] [2025-01-27 13:33:51] [Trainer] [WARNING] Skipping line because of exception: ValueError('Out-of-bound alignment pairs')
[task 2025-01-27T13:33:51.272Z] [2025-01-27 13:33:51] [Trainer] [INFO] Reading original for epoch 1996
[task 2025-01-27T13:33:51.530Z] [2025-01-27 13:33:51] [Trainer] [INFO] Reading original for epoch 1997
[task 2025-01-27T13:33:51.798Z] [2025-01-27 13:33:51] [Trainer] [INFO] Reading original for epoch 1998
[task 2025-01-27T13:33:58.378Z] [2025-01-27 13:33:58] [Trainer] [INFO] trainer stopped reading input
[task 2025-01-27T13:33:58.478Z] [tracking INFO] Successfully parsed 2526 lines
[task 2025-01-27T13:33:58.478Z] [tracking INFO] Found 50 training entries
[task 2025-01-27T13:33:58.478Z] [tracking INFO] Found 1 validation entries
[fetches 2025-01-27T13:33:58.780Z] removing /home/ubuntu/tasks/task_173798376721600/fetches
[fetches 2025-01-27T13:33:59.134Z] finished
[taskcluster 2025-01-27T13:33:59.145Z] Exit Code: 0
[taskcluster 2025-01-27T13:33:59.145Z] User Time: 1h2m46.859518s
[taskcluster 2025-01-27T13:33:59.145Z] Kernel Time: 1m28.661655s
[taskcluster 2025-01-27T13:33:59.145Z] Wall Time: 13m13.252199309s
[taskcluster 2025-01-27T13:33:59.145Z] Result: SUCCEEDED
[taskcluster 2025-01-27T13:33:59.146Z] === Task Finished ===
[taskcluster 2025-01-27T13:33:59.146Z] Task Duration: 13m13.254189288s
[taskcluster 2025-01-27T13:33:59.361Z] Uploading artifact public/build/config.opustrainer.yml.state from file /home/ubuntu/tasks/task_173798376721600/artifacts/config.opustrainer.yml.state with content encoding "gzip", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.389Z] Uploading artifact public/build/final.model.npz.best-chrf.npz from file /home/ubuntu/tasks/task_173798376721600/artifacts/final.model.npz.best-chrf.npz with content encoding "identity", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.406Z] Uploading artifact public/build/devset.out from file /home/ubuntu/tasks/task_173798376721600/artifacts/devset.out with content encoding "gzip", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.422Z] Uploading artifact public/build/vocab.spm from file /home/ubuntu/tasks/task_173798376721600/artifacts/vocab.spm with content encoding "gzip", mime type "application/x-source-rpm" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.433Z] Uploading artifact public/build/final.model.npz.best-chrf.npz.decoder.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/final.model.npz.best-chrf.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.437Z] Uploading artifact public/build/model.npz.best-chrf.npz.decoder.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.best-chrf.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.450Z] Uploading artifact public/build/model.npz.best-bleu-detok.npz from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.best-bleu-detok.npz with content encoding "identity", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.492Z] Uploading artifact public/build/model.npz.best-ce-mean-words.npz.decoder.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.best-ce-mean-words.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.500Z] Uploading artifact public/build/model.npz.decoder.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.504Z] Uploading artifact public/build/model.npz.best-bleu-detok.npz.decoder.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.best-bleu-detok.npz.decoder.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.508Z] Uploading artifact public/build/model.npz from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz with content encoding "identity", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.524Z] Uploading artifact public/build/model.npz.best-chrf.npz from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.best-chrf.npz with content encoding "identity", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.524Z] Uploading artifact public/build/model.npz.optimizer.npz from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.optimizer.npz with content encoding "identity", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.643Z] Uploading artifact public/build/model.npz.best-ce-mean-words.npz from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.best-ce-mean-words.npz with content encoding "identity", mime type "application/octet-stream" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.644Z] Uploading artifact public/build/model.npz.progress.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.progress.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.644Z] Uploading artifact public/build/opustrainer.log from file /home/ubuntu/tasks/task_173798376721600/artifacts/opustrainer.log with content encoding "gzip", mime type "text/plain" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.644Z] Uploading artifact public/build/model.npz.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/model.npz.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.645Z] Uploading artifact public/build/train.log from file /home/ubuntu/tasks/task_173798376721600/artifacts/train.log with content encoding "gzip", mime type "text/plain" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.644Z] Uploading artifact public/build/config.opustrainer.yml from file /home/ubuntu/tasks/task_173798376721600/artifacts/config.opustrainer.yml with content encoding "gzip", mime type "application/x-yaml" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:33:59.644Z] Uploading artifact public/build/valid.log from file /home/ubuntu/tasks/task_173798376721600/artifacts/valid.log with content encoding "gzip", mime type "text/plain" and expiry 2025-02-26T12:05:14.661Z
[taskcluster 2025-01-27T13:34:17.831Z] [mounts] Preserving cache: Moving "/home/ubuntu/tasks/task_173798376721600/checkouts" to "/home/ubuntu/caches/I3AP_eWSSS6gKC11ZNTBeg"
[taskcluster 2025-01-27T13:34:17.997Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2025-02-26T12:05:14.661Z