mtdata downloader

[XapaJIaMnu]

This is just a draft as I am not exactly sure how to hook it up to the GUI, but I have coded the backend bits necessary for

1. Deduplicating datasets (only taking the latest version of a dataset)
2. Downloading datasets (in parallel) using the mtdata cli.

I imagine it should be some "tab" such as "discover" datasets, where we get a list of them and we can manually exclude some/label them clean/medium/dirty. The downloader automatically splits train/test/dev based on the dataset id provided by mtdata.

This should fix #6 eventually.

[XapaJIaMnu]

This is duplicated with the function above. I assume that this is part of the javascript interface. @jelmervdl is that how it's supposed to work?

[XapaJIaMnu]

I propose we merge this for now so that it's in the tree and it can be picked up when we have a downloader interface.

[jelmervdl]

Stand-alone this doesn't add much actual functionality. I'll take this pull request over, and add some sort of minimal interface for it at least before merging it.

[jelmervdl]

Current route:

I'm really considering adding Bootstrap to the project for a bit more UI. Also a filter toggle for only the latest version of each dataset.

[XapaJIaMnu]

Looks good to me, pending on the answer of thammegowda/mtdata#129 . Ideally i'd like to see how much i am downloading before starting to download (and also we had the issue with the mozilla pipeline that downloads would fail because we would be throttled, so we should be able to limit the number of parallel downloads.)

[XapaJIaMnu]

Also, crash on exit:

^CINFO:     Shutting down
2022-10-05 17:30:27 server.shutdown:252 INFO:: Shutting down
INFO:     Finished server process [14370]
2022-10-05 17:30:27 server.serve:85 INFO:: Finished server process [14370]
ERROR:    Traceback (most recent call last):
  File "/home/dheart/uni_stuff/postdoc/empty-train/.env/lib/python3.10/site-packages/starlette/routing.py", line 638, in lifespan
    await receive()
  File "/home/dheart/uni_stuff/postdoc/empty-train/.env/lib/python3.10/site-packages/uvicorn/lifespan/on.py", line 135, in receive
    return await self.receive_queue.get()
  File "/usr/lib/python3.10/asyncio/queues.py", line 159, in get
    await getter
asyncio.exceptions.CancelledError

2022-10-05 17:30:27 on.send:132 ERROR:: Traceback (most recent call last):
  File "/home/dheart/uni_stuff/postdoc/empty-train/.env/lib/python3.10/site-packages/starlette/routing.py", line 638, in lifespan
    await receive()
  File "/home/dheart/uni_stuff/postdoc/empty-train/.env/lib/python3.10/site-packages/uvicorn/lifespan/on.py", line 135, in receive
    return await self.receive_queue.get()
  File "/usr/lib/python3.10/asyncio/queues.py", line 159, in get
    await getter
asyncio.exceptions.CancelledError

[jelmervdl]

Downloads are currently limited to two concurrent downloads.

Maybe I can get it do a HEAD request to get the size (or Content-Length really) of the download. It would be infeasible to do this for all datasets that are listed, but should be doable for the ones in your "shopping list" at least.