all_you_need_to_know.ipynb

{
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0,
  "cells": [
    {
      "cell_type": "markdown",
      "source": "**A brief look at various techniques in paraphrase detection**",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "Identifying duplicate questions is a wonderful problem, one which I've personally faced many a time on Quora. Lets have a crack at it, shall we? Rather, lets understand the ways to go about it!",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "**1. Cosine, Jaccard and Shingling**\n\nThe first, naive approach towards identifying question pairs -- Strip the stopwords, stem the remaining and do a simple Cosine/Jaccard Test. [K-Shingling](https://www.cs.utah.edu/~jeffp/teaching/cs5955/L4-Jaccard+Shingle.pdf) is also a popular technique, where continuous subsets of \"k\" words are matched between the two documents.\n\nHowever, a major drawback with the above is that of a lack of semantic understanding -- There might be two questions with a high percentage of common words, but different meanings.\n\nHere is a [kernel](https://www.kaggle.com/demetripananos/d/quora/question-pairs-dataset/duplicate-questions-have-high-cosine-similarity) from a fellow kaggler trying the cosine similarity technique.",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "**2. Semantic Similarity via Wordnet**\n\nWordnet is a huge library of synsets for almost all words in the English dictionary. The synsets for each word describe its meaning, part of speeches, and synonyms/antonyms. The synonyms help in identifying the semantic meaning of the sentence, when all words are taken together.\n\n[This paper](http://staffwww.dcs.shef.ac.uk/people/S.Fernando/pubs/clukPaper.pdf) describes how wordnet is used to calculate a matrix similarity between two sentences. Later a thresholding for paraphrases is done, they could come up with a F-Score of 82.4 on the Microsoft Research Paraphrase Corpus, the industry standard.",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "**3. Regular ML Models**\n\nAnother naive model -- Using the training data to build a Decision Tree, or a Random Forest, or even a simple Linear Regression to identify the duplicate pairs. \n\nHere is a [kernel](https://www.kaggle.com/arathee2/d/quora/question-pairs-dataset/predictive-modelling-a-simple-approach) from a kaggler on the same -- with an accuracy of 66%.",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "**4. Word Embeddings**\n\nA recent trend in the Deep NLP community, starting with the famous Word2Vec and CBOW, and now  Doc2Vec, Paragraph2Vec, [skip-thought vectors](https://gab41.lab41.org/lab41-reading-group-skip-thought-vectors-fec68c05aa92#.ptcit7c0y) coming along! These are extremely powerful models which have changed the scope of NLP models in the last 3-4 years.\n\n 1. *CNN over Word Embeddings*:\nThis [research paper](https://aclweb.org/anthology/K15-1013) explains the approach of applying Convolutional Neural Nets over the word embeddings (using a large collection of unlabeled data), building vector representations for question pairs. They tested their results over AskUbuntu, witnessing a 92.4% test accuracy!\n\n 2. *Skip-thought Vectors*:\nThis model backs upon its ability to semantically understand a sentence, thus the transition from the old *skip-gram* to *skip-thought*. Ryan Kiros is the lead developer behind this model, do have a look at his [Github Repo](https://github.com/ryankiros/skip-thoughts) and its application on Paraphrase Detection. And let me know if you can replicate his results!",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "**5. Explicit Semantic Analysis**\n\nESA is a really interesting methodology to go around this problem -- Train your model on the whole of Wikipedia (into a Bag-of-Words), and then convert the questions into a set of concepts(with probabilities). Higher the intersection of concepts, the more similar are the two questions.\n\nHere is a [comprehensive resource](http://www.gabrilovich.com/resources/code/esa/esa.html) for ESA.",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "**6. Recursive Autoencoders with Dynamic Pooling**\n\nA famous [paper](https://papers.nips.cc/paper/4204-dynamic-pooling-and-unfolding-recursive-autoencoders-for-paraphrase-detection.pdf) by Socher introduces the concept of training RAE for paraphrase detection. RAEs basically take a set of inputs, and try to output the same, minimizing the reconstruction loss -- here the inputs are syntax trees generated by the Standford Parser.\n\nDynamic Pooling allows us to compare questions of varying lengths, by mapping them to a standard matrix(of nXn) -- a type of feature extraction.\n\n I haven't been able to understand the paraphrasing algorithm clearly, working on it!",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": "**Conclusion**\n\nHere is the [list](https://aclweb.org/aclwiki/index.php?title=Paraphrase_Identification_(State_of_the_art)) of all techniques(under the sun) and their performances on the Microsoft Research Paraphrase Corpus.\n\nExcited to see more kernels here!",
      "metadata": {}
    }
  ]
}