English entry filtering

[eldrin]

non-English lyrics should be filtered properly. Currently, (at least for MVP project) we don't stick to advanced machine learning algorithms avoiding the pipeline as shallow as possible so that we can confidently validate every step although each step has a bit more error.

The logic is fairly simple:

1. count non-English words from each lyric (based on nltk corpus)
2. based on the non-English word count distribution, we select a threshold (i.e. percentile)
3. filter out entries where the number of non-English words more than the threshold

But we need validation for optimal threshold!!

[eldrin]

One way to tackle this issue is to use topic modeling as a tool for the anomaly detector. Namely, we can learn k topics without filtering, pick m topics indicating they are non-english clusters, then filter them. One can do this procedure iteratively but it seems only one cycle already seems okay.

The evaluation of this approach can be (roughly) measured by the total number of non-english words filtered by the nltk.

[eldrin]

Still seems there are erroneous entries. If there is enough room for checking it, the current pipeline should be evaluated as well.