Skip to content

wushixian/Keyphrase-extraction

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Keyphrase extraction

Keyphrase extraction meant to extract meaningful attributes from fashion products. Python3 based. What implemented so far:

  • Graph based methods
  • Embedding based methods

TODO:

  • Remove meaningless but frequent words and rebuild the LDA model
  • Maybe try some DL supervised methods

Installation of dependencies

pip3 install -r requirements.txt
python -m nltk.downloader stopwords
python -m nltk.download('averaged_perceptron_tagger')

To enable the use of Embedding-based method, follow innstructions here to download Sent2vec model and store it in the 'EmbeddingRank' folder. Name it as 's2v.bin' for example.

Since I have already trained with LDA, I stored the word-to-topic distribution, document-to-topic distribution and feauture_names as pickle file inside LDA folder. Feel free to train with your own corpus using lda.py and replace them with your trained distributions.

Minimal example

After download the whole repo and named it as kkExtract:

# Example of single-tpr extraction
import kkExtract
# tpr_rank = kkExtract.graph_methods.singleTPR(doc_index = 12, str_input=test_str) # you need to specify the document index in the corpus, build from string
tpr_rank = singleTPR(doc_index = 12, input_file=path/to/your/txt_file) # build from txt file
tpr_rank.weight_node()
res = tpr_rank.get_top_phrases(k=6)
print(res)
# Example of EmbedRank extraction
path_to_sen2vecmodel = 's2v.bin'
emb_rank = EmbedRank(k=5, embedModel = path_to_sen2vecmodel)
emb_rank.load_text(test_str)
res = emb_rank.topK_phrase()

Detailed examples are provided in the end of each methods python file

Currently implements the following keyphrase extraction models:

About

Attribute extraction from text

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%