Contribute with trained packages or training checkpoints for a language

[mmokhi]

Hi,
I was wondering if you would be interested with training-checkpoints or trained models for languages?
I see it as a way to crowdsource/share GPU power that we all might have 🙂

My understanding is that you have an index-file with links to you GC storage, but also some are kept on Google-Drive and GitHub-raw-content too.

I appreciate comments/clarifications, and I hope the idea helps 👍

[PJ-Finlay]

Help with training is definitely appreciated! If anyone's interested in helping the first step would be to get the training script running with the auto downloaded data.

[PJ-Finlay]

Oh nice. Let's coordinate training on this thread then, post a language you're working on so we don't duplicate. I'm currently doing Polish and Turkish and planning on Indonesian.

I also added some documentation on packages for the index also posted here:

Making an Argospm package

• Download data from Opus
  • Use as many as you can. For many languages there's a big dropoff in size after the first few datasets so I generally only bother with the large ones.
  • Cite each dataset in MODEL_README.md
  • Only use freely licensed datasets. Some of the datasets have non commercial restrictions.
• Download Wikimedia data
  • Download pre extraced dictionaries from: https://github.com/tatuylonen/wiktextract
  • https://kaikki.org/dictionary/
  • source config.sh && ./generate_wiktionary_data.py kaikki.org-dictionary-Irish.json kaikki.org-dictionary-English.json
• Combine datasets
  • cd raw_data && cat *.zh >> source.zh && cat *.en >> source.zh

[mmokhi]

Great!
Thanks for the hints 🙌 😃
I keep that in mind 👍

So far, I've been working on Persian (almost finished),
(Tried to re-do Arabic also, since some translation didn't look good-enough to me TBH 😅 )
And have also started Urdu and Pashto

[mmokhi]

@PJ-Finlay A question about wiki-generate script...
Does it matter in which order we pass the json file args?

Like if we are training EN -> AR or AR -> EN should we have ./generate_wiktionary_data.py kaikki.org-dictionary-English.json kaikki.org-dictionary-Arabic.json and ./generate_wiktionary_data.py kaikki.org-dictionary-Arabic.json kaikki.org-dictionary-English.json accordingly?

[PJ-Finlay]

Order doesn't matter, but you do need to re-source the config if you are reusing language data in the opposite direction.

Deleted it with reset.sh to make this more clear:
argosopentech/argos-train@7733eff

[mmokhi]

Order doesn't matter, but you do need to re-source the config if you are reusing language data in the opposite direction.

👍 Great, Thanks for clarification

Deleted it with reset.sh to make this more clear:
argosopentech/onmt-models@7733eff

Aha, great idea! 🙂 🙌

[liakhandrii]

@PJ-Finlay

What data did you use? And did you run the script to generate wiktionary data? I've never trained a model past 50k steps but it's possible it would improve translation quality. The main thing I think is better data, I currently use all of the permissively licensed data from OPUS and have a script for generating data from wiktionary. In the future I'd also like to put more than two languages through the models which hopefully will improve quality but we're not there yet.

I used the Wiki OPUS data (basically just ran your training script), and tbh that data is kinda garbage. 1/20 of the sentences are some random stuff. E.g. in English it's gonna be about some geopolitics, while in Russian it's gonna be about guitar picks. I'd love to get some more quality data, but it's super hard to find, I couldn't even find where to buy this kinda stuff.

Also can you send me the packaged model so I can potentially post it on the index? Our current Russian model was trained before the OpenNMT-py port.

Yep, here it is, https://drive.google.com/file/d/16j1xiRyhVnepUrB1a0Txk8-um3-sw7Vv/view?usp=sharing

[liakhandrii]

I used 2M sentences for this, different data sets from OPUS: opensubtitles, UNPC, CCAligned, and Wiki. 500K sentences from each. But it's not even close to the original en -> ru model you made, what did you use for that one?

Sure, here's an early stopping pull request. It's basically just adding a single line to the config.
argosopentech/argos-train#6

[mmokhi]

@liakhandrii for the PR, wouldn't a value more than 4 be safer for early_stopping?
I'm no expert on this really, but in some of the trainings I did, I have witnessed a change in curvature after more than 4 steps...

Still that might not be a perfect example, so I'm just asking here 🙂

[liakhandrii]

I honestly have no idea.

[liakhandrii]

Oh, it doesn’t mean 4 steps. It means 4x 5000, the valid_steps.

[mmokhi]

Ahha
Okay then 😁, now 20,000 makes more sense 👍 😊

[ruzko]

I'm very new to scripting, nmt and all this stuff. This might be my first github comment. But i do have a couple old computers (ubuntu, manjaro) which could be put to good use training language models. They do not have separate GPUs. I followed instructions for the training script (apart from installing CUDA), and encountered an error when trying to run train.sh:

Splitting train and valid data Traceback (most recent call last): File "./split_train_and_valid.py", line 17, in <module> target_data_file = open(args.target_data, 'r') FileNotFoundError: [Errno 2] No such file or directory: 'raw_data/source.zh' ./train.sh: line 6: split_data/all.txt: No such file or directory . . . trainer_interface.cc(174) LOG(INFO) Loading corpus: split_data/all.txt spm_train_main.cc(252) [_status.ok()] Not found: "split_data/all.txt": No such file or directory Error #2 Program terminated with an unrecoverable error. ./train.sh: line 13: onmt_build_vocab: command not found rm: cannot remove 'split_data/all.txt': No such file or directory Done with tokenization ./train.sh: line 19: onmt_train: command not found

so i checked out the config.sh script and changed export tl="zh" to export tl="es" This solved the error.
Figured I'd mention it here, since it makes sense for the training scripts to default to the languages provided in the tutorial.

[mmokhi]

I think it will probably fail when it reaches to line 19.
The reason you see it's working now, is that now it can find that sources.es file that was missing before.

What happened earlier, was couple of failing-and-going-to-next command chain, so you saw all failures at once. Now one of the errors is solved, and the step is being done, until it reaches to the next failing step

Not sure how to check that

Make sure that you haven't missed this step. then try to run onmt_train --help just to test and see if the command is found

[ruzko]

as you say, it failed at line 19. I will try to uninstall and reinstall OpenNMT-py later.
pip3 check OpenNMT-py results in launchpadlib 1.10.13 requires testresources, which is not installed. so i installed testresources. Now pip3 check OpenNMT-py results in No broken requirements found.
onmt_train --help returns command not found

[mmokhi]

👍
If you (re)install OpenNMT-py the way it's explained here, make sure that the commands will be visible in your $PATH

[PJ-Finlay]

Just a heads up, I think you will have pretty hard time training models without a GPU.

[ruzko]

Yea, it will probably be real slow. But it's not like i currently have any other use for my spare computers. Seems like uninstall -> reinstall OpenNMT-py fixed something, because i can use the onmt_train --help command now. I'll get thermald working properly to prevent overheating, then retry train.sh.

[liakhandrii]

@PJ-Finlay, hey. I finally managed to train a good en->ru model with the new guide you made.
Here, you can have a look / use the model in your model repository: https://drive.google.com/drive/folders/1UGNfUI-AdJDC_UhalTUwHZQwA_k9gWkW?usp=sharing
The 25(k) one turned out the best.

It's actually better than the legacy one you have, except for curse words. What data did you use so the legacy one nails curse words so well?

[igieon]

@liakhandrii can you also train model ru -> en. Thanks

[PJ-Finlay]

Great thanks! I'll check it out and add it to the index. Is the old one better with curse words? That's pretty funny. I just used the data from OPUS, I'm guessing curse words came from OpenSubtitles. Probably not a lot of curse words in the United Nations corpus. The legacy one was trained using OpenNMT-tf so there could be differences in how the models work but the new one should be mostly better.

[PJ-Finlay]

Live!

[mzeidhassan]

Hi @PJ-Finlay
First, thanks for creating such amazing library. This is super cool.
Can you please let me know what datasets you have used to train the EN<>Arabic model? I know it is Opus data, but I would like to know which datasets exactly? Have you used everything on OPUS?

[PJ-Finlay]

It should say in the model README