New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Preprocess Text Refactoring #209

Closed

nikicc opened this issue Mar 30, 2017 · 1 comment

Labels

Contributor

nikicc commented Mar 30, 2017 •

edited by ajdapretnar

Loading

Text version

2.4.dev

Proposed Layout

Transformation
Tokenization
Stopwords, Lexicon, Regex filtering
POS tagging
Normalization
N-Grams
DF, Max Tokens filter [Output]

Fixes

Filtering on tokens not lemmas.
DF, Max Tokens filtering on N-Grams.

Doesn't Fix

N-Grams are built from non-consecutive words in text due to filtering in step 3.

nikicc added DH2017 enhancement labels

kernc mentioned this issue

Preprocess Text: Refactor #210

Closed

ajdapretnar removed the DH2017 label

Collaborator

ajdapretnar commented Jan 17, 2020

This would be made redundant by #210.

ajdapretnar closed this as completed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment