Release v0.3.5

thammegowda · Mar 11, 2022 · 5a9c034 · 5a9c034
1 parent 9ffba89
commit 5a9c034
Show file tree

Hide file tree

Showing 16 changed files with 133,937 additions and 64 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,17 +1,20 @@
 # Change Log
 
-## v0.3.5 - WIP
+## v0.3.5 - 20220310 
 
 - Parallel download support `-j/--n-jobs` argument (with default `4`)
 - Add histogram to web search interface (Thanks, @sgowdaks)
 - Update OPUS index. Use OPUS API to download all datasets
   - A lot of new datasets are added. 
-  - WARNING: Some of OPUS IDs are not backward compatible (version number mismatch) 
+  - WARNING: Some OPUS IDs are not backward compatible (version number mismatch) 
 - Fix: JESC dataset language IDs were wrong  
 - New datasets:
    - jpn-eng: add paracrawl v3, and wmt19 TED
    - backtranslation datasets for en2ru ru2en en2ru
-- Option to set `MTDATA_RECIPES` dir (default is $PWD). All files matching the glob `${MTDATA_RECIPES}/mtdata.recipes*.yml` are loaded  
+- Option to set `MTDATA_RECIPES` dir (default is $PWD). All files matching the glob `${MTDATA_RECIPES}/mtdata.recipes*.yml` are loaded
+- WMT22 recipes added 
+- JW300 is disabled [#77](https://github.com/thammegowda/mtdata/issues/77)
+- Automatically create references.bib file based on datasets selected
 
 ## v0.3.4 - 20220206
 

diff --git a/README.md b/README.md
@@ -45,35 +45,35 @@ pip install --editable .
 We have added some commonly used datasets - you are welcome to add more! 
 These are the summary of datasets from various sources (Updated: Feb 2022).
 
-
-|                  Source | Dataset Count |
-|------------------------:|--------------:|
-|       OPUS<sup>$1</sup> |        80,830 |
-| OPUS_JW300<sup>$2</sup> |        91,248 |
-|                  Neulab |         4,455 |
-|                Facebook |         1,617 |
-|                    ELRC |         1,341 |
-|                      EU |         1,178 |
-|                  Statmt |           699 |
-|                   Tilde |           519 |
-|             LinguaTools |           253 |
-|                 Anuvaad |           196 |
-|              AI4Bharath |           192 |
-|               ParaCrawl |           126 |
-|                  Lindat |            56 |
-|         UN<sup>$3</sup> |            30 |
-|               JoshuaDec |            29 |
-|                Phontron |             4 |
-|                  NRC_CA |             4 |
-|                    IITB |             3 |
-|                     WAT |             3 |
-|             StanfordNLP |             3 |
-|                    KECL |             1 |
-|                 *Total* |      *182.8K* |
+|                         Source | Dataset Count |
+|-------------------------------:|--------------:|
+|              OPUS<sup>$1</sup> |        80,830 |
+|                         Neulab |         4,455 |
+|                       Facebook |         1,617 |
+|                           ELRC |         1,394 |
+|                             EU |         1,178 |
+|                         Statmt |           750 |
+|                          Tilde |           519 |
+|                    LinguaTools |           253 |
+|                        Anuvaad |           196 |
+|                     AI4Bharath |           192 |
+|                      ParaCrawl |           126 |
+|                         Lindat |            56 |
+|                UN<sup>$3</sup> |            30 |
+|                      JoshuaDec |            29 |
+|                    StanfordNLP |            15 |
+|                         ParIce |             8 |
+|                       Phontron |             4 |
+|                         NRC_CA |             4 |
+|                           IITB |             3 |
+|                            WAT |             3 |
+|                           KECL |             2 |
+|                      Masakhane |             2 |
+|                        *Total* |     *131,301* |
+
 
 - <sup>$1</sup> - OPUS contains duplicate entries from other listed sources, but they are often older releases of corpus.
-- <sup>$2</sup> - ~~JW300 is also retrieved from OPUS, however handled differently due to the difference in the scale and internal format. It has two versions: `v1` (tokenized) and `v1c` (raw)~~ This dataset has been taken down at source
-- <sup>$3</sup> - Only test sets are included
+- <sup>$2</sup> - Only test sets are included
 
 # CLI Usage
 - After pip installation, the CLI can be called using `mtdata` command  or `python -m mtdata`

diff --git a/docs/asciidoctor.css b/docs/asciidoctor.css
@@ -1 +1 @@
-v0.3.4/asciidoctor.css
+v0.3.5/asciidoctor.css
diff --git a/docs/dids.txt b/docs/dids.txt
@@ -1 +1 @@
-v0.3.4/dids.txt
+v0.3.5/dids.txt
diff --git a/docs/how-to-release.md b/docs/how-to-release.md
@@ -6,8 +6,9 @@ Using twine : https://twine.readthedocs.io/en/latest/
    Clear `rm -r build dist *.egg-info`   if those dir exist.
 2. Build :: `$ python setup.py sdist bdist_wheel`   
    where `sdist` is source code; `bdist_wheel` is universal ie. for all platforms
-3. Upload to **testpypi** ::  `$ twine upload -r testpypi dist/*`
-4. Upload to **pypi** ::  `$ twine upload -r pypi dist/*`
+3.  Make docs: `docs/make-docs.sh`
+4. Upload to **testpypi** ::  `$ twine upload -r testpypi dist/*`
+5. Upload to **pypi** ::  `$ twine upload -r pypi dist/*`
 
 
 ### The `.pypirc` file

diff --git a/docs/index.adoc b/docs/index.adoc
@@ -68,28 +68,29 @@ Here is  a summary of datasets from various sources (Updated: Feb 2022).
 
 |===
 | Source | Dataset Count
-| OPUS<sup>$1</sup> |        80,830
-| OPUS_JW300<sup>$2</sup> |        91,248
-| Neulab |         4,455
-| Facebook |         1,617
-| ELRC |         1,341
-| EU |         1,178
-| Statmt |           699
-| Tilde |           519
-| LinguaTools |           253
-| Anuvaad |           196
-| AI4Bharath |           192
-| ParaCrawl |           126
-| Lindat |            56
-| UN<sup>$3</sup> |            30
-| JoshuaDec |            29
-| Phontron |             4
-| NRC_CA |             4
-| IITB |             3
-| WAT |             3
-| StanfordNLP |             3
-| KECL |             1
-| *Total* |      *182.8K*
+| OPUS | 120,465
+| Neulab | 4,455
+| Facebook | 1,617
+| ELRC | 1,394
+| EU | 1,178
+| Statmt | 750
+| Tilde | 519
+| LinguaTools | 253
+| Anuvaad | 196
+| AI4Bharath | 192
+| ParaCrawl | 126
+| Lindat | 56
+| UN | 30
+| JoshuaDec | 29
+| StanfordNLP | 15
+| ParIce | 8
+| Phontron | 4
+| NRC_CA | 4
+| IITB | 3
+| WAT | 3
+| KECL | 2
+| Masakhane | 2
+| *Total* | *131,301*
 |===
 
 - <sup>$1</sup> - OPUS contains duplicate entries from other listed sources, but they are often older releases of corpus.

diff --git a/docs/index.html b/docs/index.html
@@ -1 +1 @@
-v0.3.4/index.html
+v0.3.5/index.html
diff --git a/docs/rouge-github.css b/docs/rouge-github.css
@@ -1 +1 @@
-v0.3.4/rouge-github.css
+v0.3.5/rouge-github.css
diff --git a/docs/task-wmt22.adoc b/docs/task-wmt22.adoc
@@ -12,13 +12,10 @@ This document helps download datasets for WMT22 General MT task using `mtdata`.
 
 == Setup
 
-NOTE:  mtdata v0.3.5 is required which is currently under testing. So install it from develop. We will release it to PyPi once testing is complete.
-
 [source,bash]
 ----
-# pip install mtdata==0.3.5
-# Install from develop branch
-pip install https://github.com/thammegowda/mtdata/archive/develop.zip
+pip install mtdata==0.3.5
+# pip install https://github.com/thammegowda/mtdata/archive/develop.zip  # Install from develop branch
 ----
 
 == Get Recipes File