Will update instructions to use latest wikiextractor #203
Conversation
There was a problem hiding this comment.
Hi, and thanks for submitting this PR! Happy to hear that newer versions of the WikiExtractor result in more sentences. Have you noticed any caveats we need to be aware of in terms of quality of these sentences? I think it might also be worth it to test that upgrade on the English dump.
Generally I would prefer referring to a specific commit though, as that would insure that local tests reflect what will eventually be run in the final extraction that will be added to CV. The version of the automation pipeline is specified in the scripts in https://github.com/common-voice/cv-sentence-extractor/tree/main/scripts/providers. These should match the same commit as we use in the README to have a least -surprises approach for contributors.
|
Hard to tell if the updated wikiextractor is meaningfully better. I noticed that it is processing gallery templates in the articles better, so you get some sentences from those. Maybe other things. If someone follows along and does some tests for some other language, this is something to test. Will leave this PR as a note and will update it to a specific commit if I happen to do some further work on this. |
Adjusted the instructions in the Readme to use a more recent version of wikiextractor. It seems to be able to extract more content. In my tests for the Latvian, I am able to get about 5% more sentences if the updated wikiextractor is used.