04_word_vectors.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "04_word_vectors.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "mount_file_id": "1Hz9rXJHZDZeEY4pbWkLYFhZftVekADdY",
      "authorship_tag": "ABX9TyMEntsjwuRkCZb0BptFdoNT",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/tomonari-masada/course2021-nlp/blob/main/04_word_vectors.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6t-iVvqcVnB5"
      },
      "source": [
        "# **単語ベクトル**\n",
        "* 前回のように文書をbag-of-wordsモデリングによってベクトルとして表現することは、最近はほとんど行われない。\n",
        "* まず単語のベクトル表現を得て、それを使って文書のベクトル表現を得る、という手順をとることのほうが多い。\n",
        " * こうなったのは、[word2vec](https://arxiv.org/abs/1301.3781)と呼ばれる手法が登場して以降。\n",
        "  * https://en.wikipedia.org/wiki/Word2vec\n",
        "* ここでは、spaCyを使って単語ベクトルを得る方法を示す。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y3F0XNxABxH_"
      },
      "source": [
        "## 1. spaCyの単語ベクトル\n",
        "* 今回は英語テキストのみを扱う。\n",
        "* 小規模のモデル（名前が__`_sm`__で終わるモデル）は単語ベクトルを含まない。\n",
        "* 大規模モデルはダウンロードに時間がかかる。\n",
        "* そのため、ここでは中規模モデルをインストールする。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jAMsWKXbOkoM"
      },
      "source": [
        "* 自分のGoogle Driveの適当なディレクトリへにワーキング・ディレクトリを変更する\n",
        " * pathは必ず「/content/drive/MyDrive/」で始まるはず。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vXYGiHJQOdPB"
      },
      "source": [
        "%cd /content/drive/MyDrive/2021Courses/NLP/spaCy/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a4qKeUfdCDNu"
      },
      "source": [
        "### spaCyの中規模or大規模なモデルをダウンロード\n",
        "* Google Driveにワーキング・ディレクトリを変更しておけば、ダウンロードしたモデルのファイルが残せる。\n",
        "* 次からは、あらためてモデル・ファイルをダウンロードしなくても、このファイルをpipでinstallすればよい。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B_n5sy-aQZg2"
      },
      "source": [
        "!wget -nc \"https://github.com/explosion/spacy-models/releases/download/en_core_web_md-2.2.5/en_core_web_md-2.2.5.tar.gz\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iJsxDGzgCnbG"
      },
      "source": [
        "!pip install /content/drive/MyDrive/2021Courses/NLP/spaCy/en_core_web_md-2.2.5.tar.gz"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ev9j-vQaDOL2"
      },
      "source": [
        "* 注：次のセルで「モデルがロードできない」というエラーが出たら、上のColabのメニューの「ランタイム」から「ランタイムを再起動」をクリックして、ランタイムを再起動する。その後、次のセルから実行を再開する（戻らなくてよい）。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MojyJdS4Cbbd"
      },
      "source": [
        "import spacy\n",
        "nlp = spacy.load('en_core_web_md')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pfg2RgvJwX_U"
      },
      "source": [
        "## 2. 単語ベクトルを使ってみる"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LXK0UdENE-jT"
      },
      "source": [
        "### テキストをtokenizeする"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FxAuMKmbVlnW"
      },
      "source": [
        "tokens = nlp(\"dog cat banana afskfsd\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JMi6zTI3FPYa"
      },
      "source": [
        "### トークンを列挙する"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t2SzAM9aDvXP"
      },
      "source": [
        "for token in tokens:\n",
        "  print((f'単語:{token.text}, ベクトルの有無:{token.has_vector},'\n",
        "        f'ベクトルのL2ノルム:{token.vector_norm:.4f}, \bベクトルがOoVか否か:{token.is_oov}'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Di8hwjB9FUhg"
      },
      "source": [
        "### 単語ベクトルの型と中身を確認"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XYCI_KqmWodS"
      },
      "source": [
        "* 単語ベクトルはNumPyのndarray"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4gY23wgHEkij"
      },
      "source": [
        "type(tokens[0].vector)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HcjSXEg-Wq_t"
      },
      "source": [
        "* 'dog'の単語ベクトルの要素を確認"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-Gb5jxhID0S1"
      },
      "source": [
        "tokens[0].vector"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dg0pELuZFe0H"
      },
      "source": [
        "* OoV(Out of Vocabulary)の単語ベクトルはゼロベクトル"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UhObUeE0Eg9H"
      },
      "source": [
        "print(f'{tokens[3].text}\\n{tokens[3].vector}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0MkGyUTgWwT1"
      },
      "source": [
        "* トークン列のベクトルを求めると、OoVのゼロベクトルも含めて平均が計算されるようだ。\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xvd4L57pwmpa"
      },
      "source": [
        "import numpy as np\n",
        "np.allclose(tokens[0:3].vector * 3, tokens[0:4].vector * 4)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R9cxfLvGGYLY"
      },
      "source": [
        "* 多義語であっても単語ベクトルはひとつだけ\n",
        " * 意味の数だけ別々のベクトルが用意されていたりはしない。\n",
        " * 参考 https://www.youtube.com/watch?v=RB9uDpJPZdc\n",
        " * spaCyでは単語ベクトルどうしのコサイン類似度を__`.similarity()`__で計算できる。\n",
        " * https://spacy.io/api/token#similarity"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5_UQP6WVW3s0"
      },
      "source": [
        "* 例：社名のアップルであろうと、りんごのアップルであろうと、単語ベクトルは同一\n",
        " * よって、コサイン類似度はぴったり1"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h7i_BHd5FaTa"
      },
      "source": [
        "doc = nlp(\"Apple shares rose on the news. Apple pie is delicious.\")\n",
        "print(doc[0].similarity(doc[7]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B7uGjrrAIlvV"
      },
      "source": [
        "### 文書類似度の計算"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F63Ofq-MIpAB"
      },
      "source": [
        "doc1 = nlp(\"It's a warm summer day\")\n",
        "doc2 = nlp(\"It's sunny outside\")\n",
        "doc3 = nlp(\"It's definitely cold outside\")\n",
        "\n",
        "print(doc1.similarity(doc2))\n",
        "print(doc1.similarity(doc3))\n",
        "print(doc2.similarity(doc3))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0EIlDmklyYaP"
      },
      "source": [
        "## 3. 単語ベクトルを使った文書分類\n",
        "* 文書に含まれる単語の単語ベクトルから文書のベクトル表現を得る。\n",
        "* 文書のベクトル表現を使って2値分類問題を解く。\n",
        "* sentiment analysisの有名なデータセットであるIMDbを使う。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7OMyhtDIbg0r"
      },
      "source": [
        "### IMDbデータセット\n",
        "\n",
        "* データセットの基本情報\n",
        " * Webサイト: https://ai.stanford.edu/~amaas/data/sentiment/\n",
        " * 作成者: Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng and Christopher Potts\n",
        " * タイトル: Large Movie Review Dataset (aka. IMDb Review Dataset)\n",
        " * 公開日: Jun, 2011"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v_qATgIcyxlk"
      },
      "source": [
        "### データのロード\n",
        "* Thincというツールを使う。\n",
        " * https://thinc.ai/"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pnceAxpVG8s2"
      },
      "source": [
        "import thinc.extra.datasets\n",
        "train_data, test_data = thinc.extra.datasets.imdb()\n",
        "print(f'{len(train_data)} {len(test_data)}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y6-_KL5z28ce"
      },
      "source": [
        "train_texts, train_labels = zip(*train_data)\n",
        "test_texts, test_labels = zip(*test_data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MlQ4ISYZ4Txs"
      },
      "source": [
        "train_texts[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IjIIZ7YQ5Vaf"
      },
      "source": [
        "print(train_labels)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sTFG6tRlGQYY"
      },
      "source": [
        "## 4. fasttextの単語ベクトルを使う\n",
        "* spaCyの単語ベクトルは、Pythonで実装されているので、遅い。\n",
        "* fasttextは、C++で実装されているので、速い。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vrjq9WQT_evz"
      },
      "source": [
        "### fasttextのインストール"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GTn2oj1NGuW7"
      },
      "source": [
        "!pip install fasttext"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d0HhBNNX_lSr"
      },
      "source": [
        "### 言語モデルのダウンロード\n",
        " * 7GB強のサイズがあるので、非常に時間がかかる。\n",
        " * 今回は、諦める。\n",
        " * 手順だけを下のセルに示す。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AoN3GLJVGa_A"
      },
      "source": [
        "import fasttext.util\n",
        "fasttext.util.download_model('en', if_exists='ignore')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2QE2IOVRqTP1"
      },
      "source": [
        "### 文書のベクトル化\n",
        "* モデルをロードしてから、ベクトル化。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xNbYKpQhGm3R"
      },
      "source": [
        "model = fasttext.load_model('cc.en.300.bin')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y5763OiNqKsD"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "X_train = []\n",
        "for text in train_texts:\n",
        "    X_train.append(model.get_sentence_vector(text.replace(\"\\n\",\" \")))\n",
        "X_train = np.array(X_train)\n",
        "\n",
        "X_test = []\n",
        "for text in test_texts:\n",
        "    X_test.append(model.get_sentence_vector(text.replace(\"\\n\",\" \")))\n",
        "X_test = np.array(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m0uU0v1CAEpj"
      },
      "source": [
        "* ベクトル化した結果をファイルとして保存"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xs37au0nqij5"
      },
      "source": [
        "with open('train.npy', 'wb') as f:\n",
        "    np.save(f, X_train)\n",
        "with open('test.npy', 'wb') as f:\n",
        "    np.save(f, X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kok7PUdvrT_-"
      },
      "source": [
        "with open('train_labels.npy', 'wb') as f:\n",
        "    np.save(f, np.array(train_labels))\n",
        "with open('test_labels.npy', 'wb') as f:\n",
        "    np.save(f, np.array(test_labels))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T4_RQVCxksgG"
      },
      "source": [
        "* 時間がかかるので、事前にベクトル化してある。\n",
        " * BlackboardにあるIMDbデータを、あらかじめGoogle Driveに置いておく。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h7A-j1BDeN6f"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "PATH = '/content/drive/MyDrive/2021Courses/NLP/'\n",
        "texts = dict()\n",
        "labels = dict()\n",
        "for tag in ['train', 'test']:\n",
        "  with open(f'{PATH}{tag}.npy', 'rb') as f:\n",
        "    texts[tag] = np.load(f)\n",
        "  with open(f'{PATH}{tag}_labels.npy', 'rb') as f:\n",
        "    labels[tag] = np.load(f)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GtFQvkcOrb6x"
      },
      "source": [
        "texts['train'][0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Im8djYmZkB3b"
      },
      "source": [
        "## 5. 事前学習済みBERTで文書をベクトル化\n",
        "* BERTの説明はしない。とりあえず使う。\n",
        "* BERTを単なるエンコーダとして使う。\n",
        "* 今回は、下記のspaCy向けのモデルを使う。\n",
        " * https://newreleases.io/project/github/explosion/spacy-models/release/en_trf_bertbaseuncased_lg-2.3.0\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HJ8MWBBPlUIJ"
      },
      "source": [
        "!python -m spacy download en_trf_bertbaseuncased_lg"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dI6Dd2pZmoi8"
      },
      "source": [
        "* ここでランタイムを再起動してから、次のセルを実行する。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VoJVdkZQk801"
      },
      "source": [
        "import spacy\n",
        "nlp = spacy.load(\"en_trf_bertbaseuncased_lg\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LalaUJ22k9dU"
      },
      "source": [
        "apple1 = nlp(\"Apple shares rose on the news.\")\n",
        "apple2 = nlp(\"Apple sold fewer iPhones this quarter.\")\n",
        "apple3 = nlp(\"Apple pie is delicious.\")\n",
        "\n",
        "# sentence similarity\n",
        "print(apple1.similarity(apple2)) #0.69861203\n",
        "print(apple1.similarity(apple3)) #0.5404963\n",
        "\n",
        "# sentence embeddings\n",
        "apple1.vector  # or apple1.tensor.sum(axis=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rBiyaMmB-5oZ"
      },
      "source": [
        "* 何の工夫もしないと、単語ベクトルを利用する場合に比べ、非常に時間がかかる。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VoOyuhcGmwN8"
      },
      "source": [
        "import tqdm\n",
        "\n",
        "X_train = []\n",
        "for text in tqdm.tqdm(train_texts):\n",
        "  doc = nlp(text)\n",
        "  X_train.append(doc.vector)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ss1gBFpoATIR"
      },
      "source": [
        "# 課題４\n",
        "* 春学期に習った分類手法を使って、IMDbデータセットの感情分析をしてみよう。\n",
        " * training set / test setの分割は、そのまま使う。\n",
        " * training setをどのように使うかはお任せします。（交差検証など。）\n",
        " * test setでの分類性能をArea under the ROC curveで報告。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gnK61odfnND7"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}