_1_자연어_처리(Natural_Language_Processing).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "_1 자연어 처리(Natural Language Processing).ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ingu627/TIL-THEORY/blob/master/_1_%EC%9E%90%EC%97%B0%EC%96%B4_%EC%B2%98%EB%A6%AC(Natural_Language_Processing).ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YLv9iQK0HOlX"
      },
      "source": [
        "# 자연어 처리(Natural Language Processing)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "inehXda7HZhF"
      },
      "source": [
        "* 자연어는 일상 생활에서 사용하는 언어\n",
        "* 자연어 처리는 자연어의 의미를 분석 처리하는 일\n",
        "* 텍스트 분류, 감성 분석, 문서 요약, 번역, 질의 응답, 음성 인식, 챗봇과 같은 응용"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W53tu5dXuZR_"
      },
      "source": [
        "## 텍스트 처리"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7gXKJ2H15wF_"
      },
      "source": [
        "s = 'No Pain no gain'"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eDpKqMwU6G6M",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c3e8a888-4fd6-491b-ab5a-21018677b0b0"
      },
      "source": [
        "'pain' in s"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1kXoIXAD6a7J",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1255fe4b-ee8f-4ff2-82f1-f530ad880a7c"
      },
      "source": [
        "s.split()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['No', 'Pain', 'no', 'gain']"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "71btJnM_6Jp5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "de929914-a67d-4c4c-9e57-3dcc9ff8d7ed"
      },
      "source": [
        "s.split().index('gain')"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IRcGQS4h6nnH",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "4d7837f3-0dd3-4bde-cb16-02413cf54b26"
      },
      "source": [
        "s[-4:] # index 접근"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'gain'"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "igK5VIBI6PMo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "e28beeb1-aaaf-4da9-d7b8-14ee201a0ae8"
      },
      "source": [
        "s.split()[1]"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Pain'"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bjIMwR776SIv",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "6e8c44e8-2b41-4ae2-b988-4bec3d43c98b"
      },
      "source": [
        "s.split()[2][::-1]"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'on'"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KLusoOBWkvhI"
      },
      "source": [
        "s = '한글도 처리 가능'"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dy75iFEMkycR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a3a51ba3-78a9-4d3c-9edf-6dc6e820996f"
      },
      "source": [
        "'처리' in s"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JrAyEDZdk1Jf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d04b75a6-00a6-4fe6-ce95-53e5bcaefe61"
      },
      "source": [
        "s.split()"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['한글도', '처리', '가능']"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "Bb5fWUtRFnxx",
        "outputId": "be718a53-501a-4a13-d334-ca88a1bae340"
      },
      "source": [
        "s.split()[0]"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'한글도'"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "--ANCtg2uPwW"
      },
      "source": [
        "## 영어 처리"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gxPYRMKXuiAM"
      },
      "source": [
        "#### 대소문자 통합"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jkwMuEpRvFI3"
      },
      "source": [
        "* 대소문자를 통합하지 않는다면 컴퓨터는 같은 단어를 다르게 받아들임\n",
        "* 파이썬의 내장 함수 `lower()`, `upper()`를 통해 간단하게 통합 가능"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q9oloj0ivF34",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3393663b-2eb5-4a76-8d07-eb19a0935eaf"
      },
      "source": [
        "s = 'AbCdEfGh'\n",
        "str_lower = s.lower()\n",
        "str_upper = s.upper()\n",
        "print(str_lower, str_upper)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "abcdefgh ABCDEFGH\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PwDDC4HjPaWn"
      },
      "source": [
        "### 정규화(Normalization)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yRo-_ohoPdAz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4ce04eb8-4aa7-4979-a3ff-10b93ed90fb1"
      },
      "source": [
        "s = 'I visited UK from US on 22-09-20'\n",
        "print(s)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "I visited UK from US on 22-09-20\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4nP3cRP_Pp09",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5e5252b9-bc2f-4127-94f5-647ee276df50"
      },
      "source": [
        "new_s = s.replace(\"UK\", 'United Kindom').replace('US', 'United States').replace('-20', '-2020')\n",
        "print(new_s)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "I visited United Kindom from United States on 22-09-2020\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mwfz36YS1FCU"
      },
      "source": [
        "### 정규표현식"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "myBZJRSL1JLE"
      },
      "source": [
        "* 정규 표현식은 특정 문자들을 편리하게 지정하고 추가, 삭제 가능\n",
        "* 데이터 전처리에서 정규 표현식을 많이 사용\n",
        "* 파이썬에서는 정규 표현식을 지원하는 `re` 패키지 제공"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lLT-dbGQ1NSe"
      },
      "source": [
        "* 정규 표현식 문법\n",
        "  \n",
        "| 특수문자 | 설명 |\n",
        "| - | - |\n",
        "| `.` | 앞의 문자 1개를 표현 |\n",
        "| `?` | 문자 한개를 표현하나 존재할 수도, 존재하지 않을 수도 있음(0개 또는 1개) |\n",
        "| `*` | 앞의 문자가 0개 이상 |\n",
        "| `+` | 앞의 문자가 최소 1개 이상 |\n",
        "| `^` | 뒤의 문자로 문자열이 시작 |\n",
        "| `\\$` | 앞의 문자로 문자열이 끝남 |\n",
        "| `\\{n\\}` | `n`번만큼 반복 |\n",
        "| `\\{n1, n2\\}` | `n1` 이상, `n2` 이하만큼 반복, n2를 지정하지 않으면 `n1` 이상만 반복 |\n",
        "| `\\[ abc \\]` | 안에 문자들 중 한 개의 문자와 매치, a-z처럼 범위도 지정 가능 |\n",
        "| `\\[ ^a \\]` | 해당 문자를 제외하고 매치 |\n",
        "| `a\\|b` | `a` 또는 `b`를 나타냄 |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JGv0DLZu1PqE"
      },
      "source": [
        "* 정규 표현식에 자주 사용하는 역슬래시(\\\\)를 이용한 문자 규칙\n",
        "\n",
        "| 문자 | 설명 |\n",
        "| - | - |\n",
        "| `\\\\` | 역슬래시 자체를 의미 |\n",
        "| `\\d` | 모든 숫자를 의미, [0-9]와 동일 |\n",
        "| `\\D` | 숫자를 제외한 모든 문자를 의미, [^0-9]와 동일 |\n",
        "| `\\s` | 공백을 의미, [ \\t\\n\\r\\f\\v]와 동일|\n",
        "| `\\S` | 공백을 제외한 모든 문자를 의미, [^ \\t\\n\\r\\f\\v]와 동일 |\n",
        "| `\\w` | 문자와 숫자를 의미, [a-zA-Z0-9]와 동일 |\n",
        "| `\\W` | 문자와 숫자를 제외한 다른 문자를 의미, [^a-zA-Z0-9]와 동일 |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CEIH8rWo1bJf"
      },
      "source": [
        "#### match\n",
        "\n",
        "* 컴파일한 정규 표현식을 이용해 문자열이 정규 표현식과 맞는지 검사"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V8UD0NCw8UVm"
      },
      "source": [
        "# re = regular expression \n",
        "import re"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lxzhmTi11wS1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "45a33ea3-ba6f-4c18-c18e-717a0aa3969f"
      },
      "source": [
        "check = 'ab.'\n",
        "\n",
        "print(re.match(check, 'abc'))\n",
        "print(re.match(check, 'c'))\n",
        "print(re.match(check, 'ab'))"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<re.Match object; span=(0, 3), match='abc'>\n",
            "None\n",
            "None\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lICVJ-q_1c_M"
      },
      "source": [
        "#### compile\n",
        "\n",
        "* compile을 사용하면 여러 번 사용할 경우 일반 사용보다 더 빠른 속도를 보임\n",
        "* compile을 통해 정규 표현식을 사용할 경우 `re`가 아닌 컴파일한 객체 이름을 통해 사용해야 함\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jDTSl5S31wms",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "24ccd055-669f-49bc-adc1-f25e78723383"
      },
      "source": [
        "import time \n",
        "\n",
        "normal_s_time = time.time()\n",
        "r = 'ab.'\n",
        "for i in range(1000):\n",
        "  re.match(check, 'abc')\n",
        "print('일반 사용시 소요 시간: ', time.time() - normal_s_time)\n",
        "\n",
        "compile_s_time = time.time()\n",
        "r = re.compile('ab.')\n",
        "for i in range(1000):\n",
        "  r.match(check)\n",
        "print('컴파일 사용시 소요 시간: ', time.time() - compile_s_time)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "일반 사용시 소요 시간:  0.0010993480682373047\n",
            "컴파일 사용시 소요 시간:  0.00046896934509277344\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z6OcEzN81fNV"
      },
      "source": [
        "#### search\n",
        "\n",
        "* match와 다르게, search는 문자열의 전체를 검사"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mg1iU0Kg1w70",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "29335233-98ca-4a4a-eee9-b1061a1694f9"
      },
      "source": [
        "check = 'ab?'\n",
        "\n",
        "print(re.search('a', check))\n",
        "print(re.match('kkkab', check))\n",
        "print(re.search('kkkab', check))\n",
        "print(re.match('ab', check))\n"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<re.Match object; span=(0, 1), match='a'>\n",
            "None\n",
            "None\n",
            "<re.Match object; span=(0, 2), match='ab'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nmkvc1741g5P"
      },
      "source": [
        "#### split\n",
        "\n",
        "* 정규표현식에 해당하는 문자열을 기준으로 문자열을 나눔"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6bLf5Hgs1xZk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "28262df2-bfc9-4368-8435-66577d251c98"
      },
      "source": [
        "r = re.compile(' ')\n",
        "print(r.split('abc abbc abcbab'))\n",
        "\n",
        "r= re.compile('c')\n",
        "print(r.split('abc abbc abcbab'))\n",
        "\n",
        "r = re.compile('[1-9]') # 이걸 기준으로 스플릿 해봐라\n",
        "print(r.split('slabc 2v3s 4sss 5a'))"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['abc', 'abbc', 'abcbab']\n",
            "['ab', ' abb', ' ab', 'bab']\n",
            "['slabc ', 'v', 's ', 'sss ', 'a']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fTor-yg_1h49"
      },
      "source": [
        "#### sub\n",
        "\n",
        "* 정규 표현식과 일치하는 부분을 다른 문자열로 교체\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZlqOk9DY1xtF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bac5134e-2a1a-4909-a63b-7021ee929d7a"
      },
      "source": [
        "print(re.sub('[a-z]', 'abcdefg', '1')) \n",
        "# abcdefg는 패턴속에 일치된다. 일치되는 것을 1로 바꿔라 \n",
        "\n",
        "print(re.sub('[^a-z]', 'abc defg', '1'))\n",
        "\n",
        "print(re.sub('[a-z]', 'abcdefg24', '1'))\n",
        "\n",
        "# 위는 잘못. \n",
        "print(re.sub('[a-z]', '1', 'abcdefg24'))\n",
        "\n",
        "print(re.sub('[^a-z]', '1', 'abd defg'))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1\n",
            "abc defg\n",
            "1\n",
            "111111124\n",
            "abd1defg\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BUzY6Vz31i8U"
      },
      "source": [
        "#### findall\n",
        "\n",
        "* 컴파일한 정규 표현식을 이용해 정규 표현식과 맞는 모든 문자(열)을 리스트로 반환\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Osi3UYYv1yHP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "369ea857-2f4a-444c-963a-c54a4ed7f6d2"
      },
      "source": [
        "print(re.findall('[\\d]', 'lab 2cd 3ef 4g'))\n",
        "# \\d = decimal = 숫자만 찾아줘\n",
        "\n",
        "print(re.findall('[\\W]', '!abcd@@#'))\n",
        "# #W 문자 숫자가 아닌 특수문자만 찾아줘"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['2', '3', '4']\n",
            "['!', '@', '@', '#']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NxLLmwJu1kTt"
      },
      "source": [
        "#### finditer\n",
        "\n",
        "* 컴파일한 정규 표현식을 이용해 정규 표현식과 맞는 모든 문자(열)을 `iterator` 객체로 반환\n",
        "* `iterator` 객체를 이용하면 생성된 객체를 하나씩 자동으로 가져올 수 있어 처리가 간편함"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6NOFG3Hy1ycd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c8475744-1cab-4cd7-c851-4bd38f8f8611"
      },
      "source": [
        "iter1 = re.finditer('[\\d]', 'lab 2cd 3ef 4g')\n",
        "print(iter1)\n",
        "for i in iter1:\n",
        "  print(i)\n",
        "\n",
        "iter2 = re.finditer('[\\W]', '!abcd@@#')\n",
        "print(iter)\n",
        "for i in iter2:\n",
        "  print(i)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<callable_iterator object at 0x7f656b755750>\n",
            "<re.Match object; span=(4, 5), match='2'>\n",
            "<re.Match object; span=(8, 9), match='3'>\n",
            "<re.Match object; span=(12, 13), match='4'>\n",
            "<callable_iterator object at 0x7f656b755910>\n",
            "<re.Match object; span=(0, 1), match='!'>\n",
            "<re.Match object; span=(5, 6), match='@'>\n",
            "<re.Match object; span=(6, 7), match='@'>\n",
            "<re.Match object; span=(7, 8), match='#'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sn4C67oivxxN"
      },
      "source": [
        "### 토큰화(Tokenization)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eqPZHQZPzFvc"
      },
      "source": [
        "* 특수문자에 대한 처리\n",
        "\n",
        "  + 단어에 일반적으로 사용되는 알파벳, 숫자와는 다르게 특수문자는 별도의 처리가 필요            \n",
        "  + 일괄적으로 단어의 특수문자를 제거하는 방법도 있지만 특수문자가 단어에 특별한 의미를 가질 때 이를 학습에 반영시키지 못할 수도 있음\n",
        "  + 특수문자에 대한 일괄적인 제거보다는 데이터의 특성을 파악하고, 처리를 하는 것이 중요\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FrZTwnlTyzYV"
      },
      "source": [
        "* 특정 단어에 대한 토큰 분리 방법\n",
        "\n",
        "  + 한 단어지만 토큰으로 분리할 때 판단되는 문자들로 이루어진 we're, United Kingdom 등의 단어는 어떻게 분리해야 할지 선택이 필요   \n",
        "  + we're은 한 단어이나 분리해도 단어의 의미에 별 영향을 끼치진 않지만 United Kingdom은 두 단어가 모여 특정 의미를 가리켜 분리해선 안됨\n",
        "  + 사용자가 단어의 특성을 고려해 토큰을 분리하는 것이 학습에 유리"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U9_k9rA6v0KM"
      },
      "source": [
        "#### 단어 토큰화"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t02uSgcSv6IQ"
      },
      "source": [
        "* 파이썬 내장 함수인 `split`을 활용해 단어 토큰화\n",
        "* 공백을 기준으로 단어를 분리"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bV6CNv1Sv62F",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c4f99c2c-45fd-4103-9276-e1bd560741e9"
      },
      "source": [
        "sentence = 'Time is gold'\n",
        "tokens = [x for x in sentence.split(' ')]\n",
        "tokens"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Time', 'is', 'gold']"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x_PDCM7YwEi9"
      },
      "source": [
        "* 토큰화는 `nltk` 패키지의 `tokenize` 모듈을 사용해 손쉽게 구현 가능\n",
        "* 단어 토큰화는 `word_tokenize()` 함수를 사용해 구현 가능"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "isgXAhV7wEo7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7d3e5731-f162-4eed-e8cf-01076a957e52"
      },
      "source": [
        "# nltk = natural language toolkit\n",
        "import nltk\n",
        "nltk.download('punkt')"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vXkXYwoywHAl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4f24a364-1893-4e55-8e45-1e25a12a0fbd"
      },
      "source": [
        "from nltk.tokenize import word_tokenize\n",
        "\n",
        "tokens = word_tokenize(sentence)\n",
        "tokens"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Time', 'is', 'gold']"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VcmIjFyRwJpW"
      },
      "source": [
        "#### 문장 토큰화"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Jkvs6Jov7QF"
      },
      "source": [
        "* 문장 토큰화는 줄바꿈 문자('\\n')를 기준으로 문장을 분리"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MkJ60tlTv7Ux",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aa0f16bf-ccc7-44bb-984f-e0b000f9917d"
      },
      "source": [
        "sentences = 'The world is a beautiful book.\\nBut of little use to him who cannot read it.'\n",
        "print(sentences)\n",
        "\n",
        "tokens = [x for x in sentences.split('\\n')]\n",
        "tokens"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The world is a beautiful book.\n",
            "But of little use to him who cannot read it.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['The world is a beautiful book.',\n",
              " 'But of little use to him who cannot read it.']"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nDSxHUHTwVzG"
      },
      "source": [
        "* 문장 토큰화는 `sent_tokenize()` 함수를 사용해 구현 가능"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0oR8iHWiwV6f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "227f0553-4d2a-4840-e800-ac158c22f971"
      },
      "source": [
        "from nltk.tokenize import sent_tokenize\n",
        "\n",
        "tokens = sent_tokenize(sentences)\n",
        "tokens"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['The world is a beautiful book.',\n",
              " 'But of little use to him who cannot read it.']"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Ca3lOErwbGN"
      },
      "source": [
        "* 문장 토큰화에서는 온점(.)의 처리를 위해 이진 분류기를 사용할 수도 있음\n",
        "* 온점은 문장과 문장을 구분해줄 수도, 문장에 포함된 단어를 구성할 수도 있기 때문에 이를 이진 분류기로 분류해 더욱 좋은 토큰화를 구현할 수도 있음"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hVPmqCnV2Ydt"
      },
      "source": [
        "#### 정규 표현식을 이용한 토큰화"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_hkxXlZb2j0O"
      },
      "source": [
        "* 토큰화 기능을 직접 구현할 수도 있지만 정규 표현식을 이용해 간단하게 구현할 수도 있음\n",
        "* `nltk` 패키지는 정규 표현식을 사용하는 토큰화 도구인 `RegexpTokenizer`를 제공"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9KOEXR7R2cyV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6271cf5d-1509-49a7-e6bf-a73d9c2c89fe"
      },
      "source": [
        "from nltk.tokenize import RegexpTokenizer\n",
        "\n",
        "sentence = 'Where there\\'s a will, there\\'s a way'\n",
        "\n",
        "tokenizer = RegexpTokenizer(\"[\\w]+\") # 이걸 기준으로 토큰화를 하겠다\n",
        "tokens = tokenizer.tokenize(sentence) # '는 문자 숫자가 아니기에 제외\n",
        "tokens"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Where', 'there', 's', 'a', 'will', 'there', 's', 'a', 'way']"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lhdy85nr2fTt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7ce11f58-d53b-413c-c048-8e12d58bc0c7"
      },
      "source": [
        "tokenizer = RegexpTokenizer('[\\s]+', gaps=True)\n",
        "# \\s = 공백을 기준으로 반환 \n",
        "# gaps=true는 해당 정규 표현식을 토큰으로 나누기 위한 기준으로 사용한다는 의미입니다.\n",
        "# 만약 gaps=True라는 부분을 기재하지 않는다면, 토큰화의 결과는 공백들만 나오게 됩니다.\n",
        "tokens = tokenizer.tokenize(sentence)\n",
        "tokens"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Where', \"there's\", 'a', 'will,', \"there's\", 'a', 'way']"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KMmCGvTFirsi"
      },
      "source": [
        "#### 케라스를 이용한 토큰화"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pghLPLfSirOJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "501cb7cd-d83a-46ae-b6f0-a63be5111625"
      },
      "source": [
        "from keras.preprocessing.text import text_to_word_sequence\n",
        "\n",
        "sentence = 'Where there\\'s a will, there\\'s a way'\n",
        "\n",
        "text_to_word_sequence(sentence)"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['where', \"there's\", 'a', 'will', \"there's\", 'a', 'way']"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wOZj2hkzi9Ki"
      },
      "source": [
        "#### TextBlob을 이용한 토큰화"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oIDyF_gLi84n",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2f2d997c-3733-4770-ded4-6e4d50ab93cb"
      },
      "source": [
        "from textblob import TextBlob\n",
        "\n",
        "sentence = 'Where there\\'s a will, there\\'s a way'\n",
        "\n",
        "blob = TextBlob(sentence)\n",
        "blob.words"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "WordList(['Where', 'there', \"'s\", 'a', 'will', 'there', \"'s\", 'a', 'way'])"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F5Vvqfynlf9Z"
      },
      "source": [
        "#### 기타 토크나이저\n",
        "\n",
        "* WhiteSpaceTokenizer: 공백을 기준으로 토큰화\n",
        "* WordPunktTokenizer: 텍스트를 알파벳 문자, 숫자, 알파벳 이외의 문자 리스트로 토큰화\n",
        "* MWETokenizer: MWE는 Multi-Word Expression의 약자로 'republic of korea'와 같이 여러 단어로 이뤄진 특정 그룹을 한 개체로 취급\n",
        "* TweetTokenizer: 트위터에서 사용되는 문장의 토큰화를 위해서 만들어졌으며, 문장 속 감성의 표현과 감정을 다룸"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pQh3deRog1It"
      },
      "source": [
        "### n-gram 추출\n",
        "\n",
        "* n-gram은 n개의 어절이나 음절을 연쇄적으로 분류해 그 빈도를 분석\n",
        "* n=1일 때는 unigram, n=2일 때는 bigram, n=3일 때는 trigram으로 불림"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Jj0RiGng7bV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fad0e003-8d48-4418-b16b-3c4df31228b1"
      },
      "source": [
        "from nltk import ngrams\n",
        "\n",
        "sentence = 'There is no royal road to learning'\n",
        "bigram = list(ngrams(sentence.split(), 2))\n",
        "print(bigram)"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('There', 'is'), ('is', 'no'), ('no', 'royal'), ('royal', 'road'), ('road', 'to'), ('to', 'learning')]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-sEZ6IfGh2q2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d25308d5-99b8-4dbb-b2d1-568f1fbc9fd8"
      },
      "source": [
        "trigram = list(ngrams(sentence.split(), 3))\n",
        "print(trigram)"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('There', 'is', 'no'), ('is', 'no', 'royal'), ('no', 'royal', 'road'), ('royal', 'road', 'to'), ('road', 'to', 'learning')]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0g1yxT62iMvA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "76f1fe78-e3b6-4f8b-9739-0ef34c8eec45"
      },
      "source": [
        "from textblob import TextBlob\n",
        "\n",
        "# blob = 방울 == token과 비슷한 개념\n",
        "blob = TextBlob(sentence)\n",
        "blob.ngrams(n=2)"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[WordList(['There', 'is']),\n",
              " WordList(['is', 'no']),\n",
              " WordList(['no', 'royal']),\n",
              " WordList(['royal', 'road']),\n",
              " WordList(['road', 'to']),\n",
              " WordList(['to', 'learning'])]"
            ]
          },
          "metadata": {},
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2hkjduP0idli",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "254ab669-8d31-4231-c304-65f3b5eeeec6"
      },
      "source": [
        "blob.ngrams(n=3)"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[WordList(['There', 'is', 'no']),\n",
              " WordList(['is', 'no', 'royal']),\n",
              " WordList(['no', 'royal', 'road']),\n",
              " WordList(['royal', 'road', 'to']),\n",
              " WordList(['road', 'to', 'learning'])]"
            ]
          },
          "metadata": {},
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1iRYvPrOZbGQ"
      },
      "source": [
        "### PoS(Parts of Speech) 태깅 "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2xRajzx4_5ng"
      },
      "source": [
        "* PoS는 품사를 의미하며, PoS 태깅은 문장 내에서 단어에 해당하는 각 품사를 태깅"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vo4n1kbt_4Pt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "32605dff-101d-45ff-de58-9f21d489c9d4"
      },
      "source": [
        "import nltk\n",
        "nltk.download('punkt')\n",
        "\n",
        "from nltk import word_tokenize"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OcVS13f8ATMT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9cf505ca-86a4-40fd-d446-470e1b63e08e"
      },
      "source": [
        "words = word_tokenize('Think like man of action and act like man of thought.')\n",
        "words"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Think',\n",
              " 'like',\n",
              " 'man',\n",
              " 'of',\n",
              " 'action',\n",
              " 'and',\n",
              " 'act',\n",
              " 'like',\n",
              " 'man',\n",
              " 'of',\n",
              " 'thought',\n",
              " '.']"
            ]
          },
          "metadata": {},
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YvoGzfP2AjqJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0f38f336-0d95-4b6f-e66b-afd7f59d8f05"
      },
      "source": [
        "nltk.download('averaged_perceptron_tagger')\n",
        "\n",
        "nltk.pos_tag(words) # 품사 태그"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Think', 'VBP'),\n",
              " ('like', 'IN'),\n",
              " ('man', 'NN'),\n",
              " ('of', 'IN'),\n",
              " ('action', 'NN'),\n",
              " ('and', 'CC'),\n",
              " ('act', 'NN'),\n",
              " ('like', 'IN'),\n",
              " ('man', 'NN'),\n",
              " ('of', 'IN'),\n",
              " ('thought', 'NN'),\n",
              " ('.', '.')]"
            ]
          },
          "metadata": {},
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5AaI3nFwB_qL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "27211381-6056-4810-d722-555ba905d8fd"
      },
      "source": [
        "nltk.pos_tag(word_tokenize('A rolling stone gathers no moss'))"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('A', 'DT'),\n",
              " ('rolling', 'VBG'),\n",
              " ('stone', 'NN'),\n",
              " ('gathers', 'NNS'),\n",
              " ('no', 'DT'),\n",
              " ('moss', 'NN')]"
            ]
          },
          "metadata": {},
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AWUanibELmmi"
      },
      "source": [
        "* PoS 태그 리스트"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8RGD-0sVAwaV"
      },
      "source": [
        "| Number | Tag | Description | 설명 |\n",
        "| -- | -- | -- | -- |\n",
        "| 1 | `CC` | Coordinating conjunction |\n",
        "| 2 | `CD` | Cardinal number |\n",
        "| 3 | `DT` | Determiner | 한정사\n",
        "| 4 | `EX` | Existential there |\n",
        "| 5 | `FW` | Foreign word | 외래어 |\n",
        "| 6 | `IN` | Preposition or subordinating conjunction | 전치사 또는 종속 접속사 |\n",
        "| 7 | `JJ` | Adjective | 형용사 |\n",
        "| 8 | `JJR` | Adjective, comparative | 헝용사, 비교급 |\n",
        "| 9 | `JJS` | Adjective, superlative | 형용사, 최상급 |\n",
        "| 10 | `LS` | List item marker |\n",
        "| 11 | `MD` | Modal |\n",
        "| 12 | `NN` | Noun, singular or mass | 명사, 단수형 |\n",
        "| 13 | `NNS` | Noun, plural | 명사, 복수형 |\n",
        "| 14 | `NNP` | Proper noun, singular | 고유명사, 단수형 |\n",
        "| 15 | `NNPS` | Proper noun, plural | 고유명사, 복수형 |\n",
        "| 16 | `PDT` | Predeterminer | 전치한정사 |\n",
        "| 17 | `POS` | Possessive ending | 소유형용사 |\n",
        "| 18 | `PRP` | Personal pronoun | 인칭 대명사 |\n",
        "| 19 | `PRP$` | Possessive pronoun | 소유 대명사 |\n",
        "| 20 | `RB` | Adverb | 부사 |\n",
        "| 21 | `RBR` | Adverb, comparative | 부사, 비교급 |\n",
        "| 22 | `RBS` | Adverb, superlative | 부사, 최상급 |\n",
        "| 23 | `RP` | Particle |\n",
        "| 24 | `SYM` | Symbol | 기호\n",
        "| 25 | `TO` | to |\n",
        "| 26 | `UH` | Interjection | 감탄사 |\n",
        "| 27 | `VB` | Verb, base form | 동사, 원형 |\n",
        "| 28 | `VBD` | Verb, past tense | 동사, 과거형 |\n",
        "| 29 | `VBG` | Verb, gerund or present participle | 동사, 현재분사 |\n",
        "| 30 | `VBN` | Verb, past participle | 동사, 과거분사 |\n",
        "| 31 | `VBP` | Verb, non-3rd person singular present | 동사, 비3인칭 단수 |\n",
        "| 32 | `VBZ` | Verb, 3rd person singular present | 동사, 3인칭 단수 |\n",
        "| 33 | `WDT` | Wh-determiner |\n",
        "| 34 | `WP` | Wh-pronoun |\n",
        "| 35 | `WP$` | Possessive wh-pronoun |\n",
        "| 36 | `WRB` | Wh-adverb |\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fmDuyhjxugFN"
      },
      "source": [
        "### 불용어 제거"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fj2rvWzwuy7o"
      },
      "source": [
        "* 영어의 전치사(on, in), 한국어의 조사(을, 를) 등은 분석에 필요하지 않은 경우가 많음\n",
        "* 길이가 짧은 단어, 등장 빈도 수가 적은 단어들도 분석에 큰 영향을 주지 않음\n",
        "* 일반적으로 사용되는 도구들은 해당 단어들을 제거해주지만 완벽하게 제거되지는 않음\n",
        "* 사용자가 불용어 사전을 만들어 해당 단어들을 제거하는 것이 좋음\n",
        "* 도구들이 걸러주지 않는 전치사, 조사 등을 불용어 사전을 만들어 불필요한 단어들을 제거\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hjFp9u9quy_t",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7c1adabe-cc2b-4162-c77a-6e94e6adaff1"
      },
      "source": [
        "stop_words = 'on in the'\n",
        "stop_words = stop_words.split(' ')\n",
        "stop_words"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['on', 'in', 'the']"
            ]
          },
          "metadata": {},
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9nKwwtR6uzCr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7878f061-6e46-4372-bee7-22997ceba64a"
      },
      "source": [
        "sentence = 'singer on the stage'\n",
        "sentence = sentence.split(' ')\n",
        "nouns = []\n",
        "for noun in sentence:\n",
        "  if noun not in stop_words:\n",
        "    nouns.append(noun)\n",
        "  \n",
        "nouns"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['singer', 'stage']"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ykTS2yeMMEXF"
      },
      "source": [
        "* `nltk` 패키지에 불용어 리스트 사용"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XlDpO3e-MHwa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7543b733-4007-4b5f-dfff-3dd2ad24bb1f"
      },
      "source": [
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "\n",
        "from nltk import word_tokenize\n",
        "from nltk.corpus import stopwords"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1jC408N2MRR0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "219dd38a-7f13-43de-9904-9c6d9850955e"
      },
      "source": [
        "stop_words = stopwords.words('english') # 영어로 된 불용어\n",
        "print(stop_words)"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zShCTozjMYWG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "be66f064-4217-4520-a1f7-2e440489d833"
      },
      "source": [
        "s = 'If you do not walk today, you will have to run tomorrow.'\n",
        "words = word_tokenize(s)\n",
        "print(words)"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['If', 'you', 'do', 'not', 'walk', 'today', ',', 'you', 'will', 'have', 'to', 'run', 'tomorrow', '.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qd_gC78EOPdv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f13eba3e-0ba2-47b9-ba37-72a7ea1f86ee"
      },
      "source": [
        "no_stopwords = []\n",
        "for w in words:\n",
        "  if w not in stop_words:\n",
        "    no_stopwords.append(w)\n",
        "\n",
        "print(no_stopwords)"
      ],
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['If', 'walk', 'today', ',', 'run', 'tomorrow', '.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LXO6E3E--gQy"
      },
      "source": [
        "### 철자 교정"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AM6lTU2w-q7A"
      },
      "source": [
        "* 텍스트에 오탈자가 존재하는 경우가 있음\n",
        "* 예를 들어, 단어 'apple'을 'aplpe'과 같이 철자 순서가 바뀌거나 spple 같이 철자가 틀릴 수 있음\n",
        "* 사람이 적절한 추정을 통해 이해하는데는 문제가 없지만, 컴퓨터는 이러한 단어를 그대로 받아들여 처리가 필요\n",
        "* 철자 교정 알고리즘은 이미 개발되어 워드 프로세서나 다양한 서비스에서 많이 적용됨\n",
        "됨"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m3hUcYaYRwVP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "505d2a1b-265f-414c-c6de-f3d80b92d5d7"
      },
      "source": [
        "!pip install autocorrect"
      ],
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting autocorrect\n",
            "  Downloading autocorrect-2.5.0.tar.gz (622 kB)\n",
            "\u001b[K     |████████████████████████████████| 622 kB 4.1 MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: autocorrect\n",
            "  Building wheel for autocorrect (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for autocorrect: filename=autocorrect-2.5.0-py3-none-any.whl size=621851 sha256=10bf39018f242d857c501c22c486fa516f45a3f90dad50756735ef63d50c4de6\n",
            "  Stored in directory: /root/.cache/pip/wheels/3d/8e/bd/f6fd900a056a031bf710a00bca338d86f43b83f0c25ab5242f\n",
            "Successfully built autocorrect\n",
            "Installing collected packages: autocorrect\n",
            "Successfully installed autocorrect-2.5.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QtqlKspEfpK5"
      },
      "source": [
        "from autocorrect import Speller"
      ],
      "execution_count": 59,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4H4TaYE7REcI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "67173248-b49d-4b3f-e67d-84258e54c93a"
      },
      "source": [
        "spell = Speller('en')\n",
        "\n",
        "print(spell('peoplle'))\n",
        "print(spell('peope'))\n",
        "print(spell('peopae'))"
      ],
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "people\n",
            "people\n",
            "people\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VO03TM_KSL4V",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cd1aa130-caca-4fcd-f8b4-08d275ac7169"
      },
      "source": [
        "s = word_tokenize('Earlly biird catchess the womm.')\n",
        "print(s)\n",
        "ss = ' '.join(spell(s) for s in s)\n",
        "print(ss) # 철자 교정"
      ],
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Earlly', 'biird', 'catchess', 'the', 'womm', '.']\n",
            "Early bird catches the worm .\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H4_svyI2myE3"
      },
      "source": [
        "### 언어의 단수화와 복수화"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3tDE8Aeumxml",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e26805d8-1005-4686-8601-3c08f582fca3"
      },
      "source": [
        "from textblob import TextBlob\n",
        "\n",
        "words = 'apples bananas oranges'\n",
        "tb = TextBlob(words)\n",
        "\n",
        "print(tb.words)\n",
        "print(tb.words.singularize()) # 단수화 취급"
      ],
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['apples', 'bananas', 'oranges']\n",
            "['apple', 'banana', 'orange']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WLqLAg6Snfi8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bbe5a858-ceb6-4e9f-98dd-0c9601632508"
      },
      "source": [
        "words = 'car train airplane'\n",
        "tb = TextBlob(words)\n",
        "\n",
        "print(tb.words)\n",
        "print(tb.words.pluralize())"
      ],
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['car', 'train', 'airplane']\n",
            "['cars', 'trains', 'airplanes']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q-9i7ubNX0mD"
      },
      "source": [
        "### 어간(Stemming) 추출"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3_28SNykTEbf"
      },
      "source": [
        "import nltk\n",
        "\n",
        "stemmer = nltk.stem.PorterStemmer()"
      ],
      "execution_count": 65,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LCwnOPFNTP2v",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "0839d315-e4d4-412d-ea75-73fe2b7dafd0"
      },
      "source": [
        "stemmer.stem('application')"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'applic'"
            ]
          },
          "metadata": {},
          "execution_count": 66
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O6O5VoWcTQuO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "4d7dce47-bac8-4797-c33a-2361069135e6"
      },
      "source": [
        "stemmer.stem('beginning')"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'begin'"
            ]
          },
          "metadata": {},
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aHvL3FirTUQO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "99197873-3449-451d-fde0-024c0b9dbdef"
      },
      "source": [
        "stemmer.stem('catches')"
      ],
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'catch'"
            ]
          },
          "metadata": {},
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4B7VXlfMTiAj",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "ec40b8da-2e56-4912-8535-9b9bf33ff0b0"
      },
      "source": [
        "stemmer.stem('education')"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'educ'"
            ]
          },
          "metadata": {},
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9-zauW83XwJ8"
      },
      "source": [
        "### 표제어(Lemmatization) 추출"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FTHh2W06TlPm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6aa95b9d-f994-471a-de3f-d5dc1b006b85"
      },
      "source": [
        "import nltk\n",
        "nltk.download('wordnet')\n",
        "from nltk.stem.wordnet import WordNetLemmatizer\n",
        "lemmatizer = WordNetLemmatizer()"
      ],
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/wordnet.zip.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LF_jyPwaTzV5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "e5dc514b-5a8e-4f99-e6e9-f76817a0b1be"
      },
      "source": [
        "lemmatizer.lemmatize('application')"
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'application'"
            ]
          },
          "metadata": {},
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WrFilM4jWF9p",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "740b5757-de4a-418c-cc92-819bc54aca31"
      },
      "source": [
        "lemmatizer.lemmatize('beginning')"
      ],
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'beginning'"
            ]
          },
          "metadata": {},
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T1pwiZSgWV8Y",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "b5f92b79-8b5d-41bd-b39e-515df7a3983f"
      },
      "source": [
        "lemmatizer.lemmatize('catches')"
      ],
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'catch'"
            ]
          },
          "metadata": {},
          "execution_count": 74
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MoKJNaJ8WT2x",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "42de98ef-e041-47b7-b958-67c03b41ad58"
      },
      "source": [
        "lemmatizer.lemmatize('education')"
      ],
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'education'"
            ]
          },
          "metadata": {},
          "execution_count": 75
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YauGaSZHWg3R"
      },
      "source": [
        "### 개체명 인식(Named Entity Recognition)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t5tcQw9FW8fi",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "09e4939a-c4e7-4c32-b988-f976525b6a7e"
      },
      "source": [
        "import nltk \n",
        "from nltk import word_tokenize\n",
        "nltk.download('maxent_ne_chunker')\n",
        "nltk.download('words')\n",
        "\n",
        "# 개체 = 국가, 등 특수 단어"
      ],
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package maxent_ne_chunker to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping chunkers/maxent_ne_chunker.zip.\n",
            "[nltk_data] Downloading package words to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/words.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WqWjHuthXWp4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6ac2c70d-247f-4300-faca-0ac8dd89ac0f"
      },
      "source": [
        "s = 'Rome was not built in a day.'\n",
        "print(s)"
      ],
      "execution_count": 77,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Rome was not built in a day.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NnnBqd-YXkwa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a6117f45-fae7-472a-c111-44f60b827622"
      },
      "source": [
        "tags = nltk.pos_tag(word_tokenize(s))\n",
        "print(tags)"
      ],
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('Rome', 'NNP'), ('was', 'VBD'), ('not', 'RB'), ('built', 'VBN'), ('in', 'IN'), ('a', 'DT'), ('day', 'NN'), ('.', '.')]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kjnWKrpPYWax",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "84df88d6-6b03-4828-e00f-792a88ad4f94"
      },
      "source": [
        "entities = nltk.ne_chunk(tags, binary=True)\n",
        "print(entities)\n",
        "\n",
        "# NE = named entity\n",
        "# ne_chunk = 객체명 인식하는 함수"
      ],
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(S (NE Rome/NNP) was/VBD not/RB built/VBN in/IN a/DT day/NN ./.)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IOVaBRHZauv8"
      },
      "source": [
        "### 단어 중의성(Lexical Ambiguity)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j9Y7u8Kqawho",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0ee4b9e0-ac0e-4b40-a45b-29201ff298a6"
      },
      "source": [
        "import nltk\n",
        "from nltk.wsd import lesk\n",
        "\n",
        "s = 'I saw bats.'\n",
        "\n",
        "print(word_tokenize(s))\n",
        "print(lesk(word_tokenize(s), 'saw'))\n",
        "print(lesk(word_tokenize(s), 'bats'))\n",
        "\n",
        "# Synset 중의성을 가지는 것에 대한 셋트를 가지고 있는데"
      ],
      "execution_count": 81,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['I', 'saw', 'bats', '.']\n",
            "Synset('saw.v.01')\n",
            "Synset('squash_racket.n.01')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FU4T_80TuSdp"
      },
      "source": [
        "## 한국어 처리"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8qeUwjUq3lkH"
      },
      "source": [
        "### 정규표현식"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1HSV73e530v8"
      },
      "source": [
        "* 한국어 정규 표현식도 대부분의 문법은 영어 정규 표현식과 같음\n",
        "* 한국어는 자음과 모음이 분리되어 있기 때문에, 문법을 지정할 때는 자음과 모음을 동시에 고려해야 함"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DKYR7b3U44RM"
      },
      "source": [
        "#### match\n",
        "\n",
        "* 컴파일한 정규 표현식을 이용해 문자열이 정규 표현식과 맞는지 검사"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AgwMiWUe4zJN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1e8f7d82-920b-4d9d-f451-985f3cc824c9"
      },
      "source": [
        "import re\n",
        "\n",
        "check = '[ㄱ-ㅎ]+'\n",
        "\n",
        "print(re.match(check, 'ㅎ 안녕하세요.'))\n",
        "print(re.match(check, '안녕하세요. ㅎ'))"
      ],
      "execution_count": 84,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<re.Match object; span=(0, 1), match='ㅎ'>\n",
            "None\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oVxRcA8Z457T"
      },
      "source": [
        "#### search\n",
        "\n",
        "* match와 다르게, search는 문자열의 전체를 검사"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WWS4rgxq48N7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4c159e68-9b8a-4bea-e184-6fb6c6e66431"
      },
      "source": [
        "check = '[ㄱ-ㅎ|ㅏ-ㅣ]+'\n",
        "\n",
        "print(re.search(check, 'ㄱㅏ 안녕하세요'))\n",
        "print(re.match(check, '안 ㄱㅏ'))\n",
        "print(re.search(check, '안 ㄱㅏ'))"
      ],
      "execution_count": 86,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<re.Match object; span=(0, 2), match='ㄱㅏ'>\n",
            "None\n",
            "<re.Match object; span=(2, 4), match='ㄱㅏ'>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eCWlVO4S48Ts"
      },
      "source": [
        "#### sub\n",
        "\n",
        "* 정규 표현식과 일치하는 부분을 다른 문자열로 교체"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5Kv_oNRs48fU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "29cf3c31-abf1-4a64-8362-daee59f74a8b"
      },
      "source": [
        "print(re.sub('[가-힣]', '1', '가나다라마바사'))\n",
        "print(re.sub('[^가-힣]', '1', '가나다라마바사'))"
      ],
      "execution_count": 88,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1111111\n",
            "가나다라마바사\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pe_7bBBdvlXQ"
      },
      "source": [
        "### 토큰화(Tokenization)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6aEfI5YHysxv"
      },
      "source": [
        "* 한국어를 학습 데이터를 사용할 때는 언어의 특성으로 인해 추가로 고려해야 할 사항이 존재\n",
        "* 한국어는 띄어쓰기를 준수하지 않아도 의미가 전달되는 경우가 많아 띄어쓰기가 지켜지지 않을 가능성이 존재\n",
        "* 띄어쓰기가 지켜지지 않으면 정상적인 토큰 분리가 이루어지지 않음\n",
        "* 한국어는 형태소라는 개념이 존재해 추가로 고려해주어야만 함\n",
        "* '그는', '그가' 등의 단어들은 같은 의미를 가리키지만 텍스트 처리에서는 다르게 받아들일 수 있어 처리를 해줘야 함"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "765_lRMqwlZg"
      },
      "source": [
        "#### 한국어 자연어 처리 konlpy와 형태소 분석기 MeCab 설치\n",
        "\n",
        "* https://raw.githubusercontent.com/konlpy/konlpy/master/scripts/mecab.sh"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8CoLF80pwnc9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f6d0d81a-21cb-41ae-f33a-e03bcdd3f53d"
      },
      "source": [
        "# korea natural language \n",
        "!set -x \\\n",
        "&& pip install konlpy \\\n",
        "&& curl -s https://raw.githubusercontent.com/konlpy/konlpy/master/scripts/mecab.sh | bash -x"
      ],
      "execution_count": 89,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+ pip install konlpy\n",
            "Collecting konlpy\n",
            "  Downloading konlpy-0.5.2-py2.py3-none-any.whl (19.4 MB)\n",
            "\u001b[K     |████████████████████████████████| 19.4 MB 1.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.6 in /usr/local/lib/python3.7/dist-packages (from konlpy) (1.19.5)\n",
            "Collecting beautifulsoup4==4.6.0\n",
            "  Downloading beautifulsoup4-4.6.0-py3-none-any.whl (86 kB)\n",
            "\u001b[K     |████████████████████████████████| 86 kB 4.3 MB/s \n",
            "\u001b[?25hCollecting JPype1>=0.7.0\n",
            "  Downloading JPype1-1.3.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (448 kB)\n",
            "\u001b[K     |████████████████████████████████| 448 kB 46.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: tweepy>=3.7.0 in /usr/local/lib/python3.7/dist-packages (from konlpy) (3.10.0)\n",
            "Collecting colorama\n",
            "  Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)\n",
            "Requirement already satisfied: lxml>=4.1.0 in /usr/local/lib/python3.7/dist-packages (from konlpy) (4.2.6)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from JPype1>=0.7.0->konlpy) (3.7.4.3)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from tweepy>=3.7.0->konlpy) (1.3.0)\n",
            "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from tweepy>=3.7.0->konlpy) (1.15.0)\n",
            "Requirement already satisfied: requests[socks]>=2.11.1 in /usr/local/lib/python3.7/dist-packages (from tweepy>=3.7.0->konlpy) (2.23.0)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->tweepy>=3.7.0->konlpy) (3.1.1)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests[socks]>=2.11.1->tweepy>=3.7.0->konlpy) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests[socks]>=2.11.1->tweepy>=3.7.0->konlpy) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests[socks]>=2.11.1->tweepy>=3.7.0->konlpy) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests[socks]>=2.11.1->tweepy>=3.7.0->konlpy) (2021.5.30)\n",
            "Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /usr/local/lib/python3.7/dist-packages (from requests[socks]>=2.11.1->tweepy>=3.7.0->konlpy) (1.7.1)\n",
            "Installing collected packages: JPype1, colorama, beautifulsoup4, konlpy\n",
            "  Attempting uninstall: beautifulsoup4\n",
            "    Found existing installation: beautifulsoup4 4.6.3\n",
            "    Uninstalling beautifulsoup4-4.6.3:\n",
            "      Successfully uninstalled beautifulsoup4-4.6.3\n",
            "Successfully installed JPype1-1.3.0 beautifulsoup4-4.6.0 colorama-0.4.4 konlpy-0.5.2\n",
            "+ bash -x\n",
            "+ curl -s https://raw.githubusercontent.com/konlpy/konlpy/master/scripts/mecab.sh\n",
            "+ mecab_dicdir=/usr/local/lib/mecab/dic/mecab-ko-dic\n",
            "+ set -e\n",
            "++ uname\n",
            "+ os=Linux\n",
            "+ [[ ! Linux == \\L\\i\\n\\u\\x ]]\n",
            "+ hash sudo\n",
            "+ sudo=sudo\n",
            "+ python=python3\n",
            "+ hash pyenv\n",
            "+ at_user_site=\n",
            "++ check_python_site_location_is_writable\n",
            "++ python3 -\n",
            "+ [[ 1 == \\0 ]]\n",
            "+ hash automake\n",
            "+ echo 'Installing automake (A dependency for mecab-ko)'\n",
            "Installing automake (A dependency for mecab-ko)\n",
            "+ install_automake\n",
            "+ '[' Linux == Linux ']'\n",
            "++ grep -Ei 'debian|buntu|mint' /etc/lsb-release /etc/os-release\n",
            "+ '[' '/etc/lsb-release:DISTRIB_ID=Ubuntu\n",
            "/etc/lsb-release:DISTRIB_DESCRIPTION=\"Ubuntu 18.04.5 LTS\"\n",
            "/etc/os-release:NAME=\"Ubuntu\"\n",
            "/etc/os-release:ID=ubuntu\n",
            "/etc/os-release:ID_LIKE=debian\n",
            "/etc/os-release:PRETTY_NAME=\"Ubuntu 18.04.5 LTS\"\n",
            "/etc/os-release:HOME_URL=\"https://www.ubuntu.com/\"\n",
            "/etc/os-release:SUPPORT_URL=\"https://help.ubuntu.com/\"\n",
            "/etc/os-release:BUG_REPORT_URL=\"https://bugs.launchpad.net/ubuntu/\"\n",
            "/etc/os-release:PRIVACY_POLICY_URL=\"https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\"\n",
            "/etc/os-release:UBUNTU_CODENAME=bionic' ']'\n",
            "+ sudo apt-get update\n",
            "Get:1 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease [3,626 B]\n",
            "Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]\n",
            "Get:3 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease [15.9 kB]\n",
            "Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease\n",
            "Ign:5 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease\n",
            "Ign:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease\n",
            "Hit:7 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release\n",
            "Get:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]\n",
            "Hit:9 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release\n",
            "Hit:11 http://ppa.launchpad.net/cran/libgit2/ubuntu bionic InRelease\n",
            "Get:12 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2,365 kB]\n",
            "Get:14 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic InRelease [15.9 kB]\n",
            "Get:15 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]\n",
            "Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2,210 kB]\n",
            "Hit:17 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease\n",
            "Get:18 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,431 kB]\n",
            "Get:19 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main Sources [1,802 kB]\n",
            "Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2,801 kB]\n",
            "Get:21 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main amd64 Packages [922 kB]\n",
            "Get:22 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic/main amd64 Packages [40.8 kB]\n",
            "Fetched 11.9 MB in 7s (1,648 kB/s)\n",
            "Reading package lists... Done\n",
            "+ sudo apt-get install -y automake\n",
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "The following additional packages will be installed:\n",
            "  autoconf autotools-dev libsigsegv2 m4\n",
            "Suggested packages:\n",
            "  autoconf-archive gnu-standards autoconf-doc libtool gettext m4-doc\n",
            "The following NEW packages will be installed:\n",
            "  autoconf automake autotools-dev libsigsegv2 m4\n",
            "0 upgraded, 5 newly installed, 0 to remove and 39 not upgraded.\n",
            "Need to get 1,082 kB of archives.\n",
            "After this operation, 3,994 kB of additional disk space will be used.\n",
            "Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsigsegv2 amd64 2.12-1 [14.7 kB]\n",
            "Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 m4 amd64 1.4.18-1 [197 kB]\n",
            "Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 autoconf all 2.69-11 [322 kB]\n",
            "Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 autotools-dev all 20180224.1 [39.6 kB]\n",
            "Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 automake all 1:1.15.1-3ubuntu2 [509 kB]\n",
            "Fetched 1,082 kB in 3s (344 kB/s)\n",
            "debconf: unable to initialize frontend: Dialog\n",
            "debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 5.)\n",
            "debconf: falling back to frontend: Readline\n",
            "debconf: unable to initialize frontend: Readline\n",
            "debconf: (This frontend requires a controlling tty.)\n",
            "debconf: falling back to frontend: Teletype\n",
            "dpkg-preconfigure: unable to re-open stdin: \n",
            "Selecting previously unselected package libsigsegv2:amd64.\n",
            "(Reading database ... 155047 files and directories currently installed.)\n",
            "Preparing to unpack .../libsigsegv2_2.12-1_amd64.deb ...\n",
            "Unpacking libsigsegv2:amd64 (2.12-1) ...\n",
            "Selecting previously unselected package m4.\n",
            "Preparing to unpack .../archives/m4_1.4.18-1_amd64.deb ...\n",
            "Unpacking m4 (1.4.18-1) ...\n",
            "Selecting previously unselected package autoconf.\n",
            "Preparing to unpack .../autoconf_2.69-11_all.deb ...\n",
            "Unpacking autoconf (2.69-11) ...\n",
            "Selecting previously unselected package autotools-dev.\n",
            "Preparing to unpack .../autotools-dev_20180224.1_all.deb ...\n",
            "Unpacking autotools-dev (20180224.1) ...\n",
            "Selecting previously unselected package automake.\n",
            "Preparing to unpack .../automake_1%3a1.15.1-3ubuntu2_all.deb ...\n",
            "Unpacking automake (1:1.15.1-3ubuntu2) ...\n",
            "Setting up libsigsegv2:amd64 (2.12-1) ...\n",
            "Setting up m4 (1.4.18-1) ...\n",
            "Setting up autotools-dev (20180224.1) ...\n",
            "Setting up autoconf (2.69-11) ...\n",
            "Setting up automake (1:1.15.1-3ubuntu2) ...\n",
            "update-alternatives: using /usr/bin/automake-1.15 to provide /usr/bin/automake (automake) in auto mode\n",
            "Processing triggers for libc-bin (2.27-3ubuntu1.3) ...\n",
            "/sbin/ldconfig.real: /usr/local/lib/python3.7/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link\n",
            "\n",
            "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...\n",
            "+ hash mecab\n",
            "+ echo 'Install mecab-ko-dic'\n",
            "Install mecab-ko-dic\n",
            "+ install_mecab_ko\n",
            "+ cd /tmp\n",
            "+ curl -LO https://bitbucket.org/eunjeon/mecab-ko/downloads/mecab-0.996-ko-0.9.2.tar.gz\n",
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
            "100 1381k  100 1381k    0     0   488k      0  0:00:02  0:00:02 --:--:--  928k\n",
            "+ tar zxfv mecab-0.996-ko-0.9.2.tar.gz\n",
            "mecab-0.996-ko-0.9.2/\n",
            "mecab-0.996-ko-0.9.2/example/\n",
            "mecab-0.996-ko-0.9.2/example/example.cpp\n",
            "mecab-0.996-ko-0.9.2/example/example_lattice.cpp\n",
            "mecab-0.996-ko-0.9.2/example/example_lattice.c\n",
            "mecab-0.996-ko-0.9.2/example/example.c\n",
            "mecab-0.996-ko-0.9.2/example/thread_test.cpp\n",
            "mecab-0.996-ko-0.9.2/mecab-config.in\n",
            "mecab-0.996-ko-0.9.2/man/\n",
            "mecab-0.996-ko-0.9.2/man/Makefile.am\n",
            "mecab-0.996-ko-0.9.2/man/mecab.1\n",
            "mecab-0.996-ko-0.9.2/man/Makefile.in\n",
            "mecab-0.996-ko-0.9.2/mecab.iss.in\n",
            "mecab-0.996-ko-0.9.2/config.guess\n",
            "mecab-0.996-ko-0.9.2/README\n",
            "mecab-0.996-ko-0.9.2/COPYING\n",
            "mecab-0.996-ko-0.9.2/CHANGES.md\n",
            "mecab-0.996-ko-0.9.2/README.md\n",
            "mecab-0.996-ko-0.9.2/INSTALL\n",
            "mecab-0.996-ko-0.9.2/config.sub\n",
            "mecab-0.996-ko-0.9.2/configure.in\n",
            "mecab-0.996-ko-0.9.2/swig/\n",
            "mecab-0.996-ko-0.9.2/swig/Makefile\n",
            "mecab-0.996-ko-0.9.2/swig/version.h.in\n",
            "mecab-0.996-ko-0.9.2/swig/version.h\n",
            "mecab-0.996-ko-0.9.2/swig/MeCab.i\n",
            "mecab-0.996-ko-0.9.2/aclocal.m4\n",
            "mecab-0.996-ko-0.9.2/LGPL\n",
            "mecab-0.996-ko-0.9.2/Makefile.am\n",
            "mecab-0.996-ko-0.9.2/configure\n",
            "mecab-0.996-ko-0.9.2/tests/\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/test\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/t9/\n",
            "mecab-0.996-ko-0.9.2/tests/t9/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/t9/ipadic.pl\n",
            "mecab-0.996-ko-0.9.2/tests/t9/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/t9/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/t9/test\n",
            "mecab-0.996-ko-0.9.2/tests/t9/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/t9/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/t9/mkdic.pl\n",
            "mecab-0.996-ko-0.9.2/tests/t9/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/ipa.train\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/ipa.test\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/rewrite.def\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/feature.def\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/run-eval.sh\n",
            "mecab-0.996-ko-0.9.2/tests/run-cost-train.sh\n",
            "mecab-0.996-ko-0.9.2/tests/Makefile.am\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/test\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/eval/\n",
            "mecab-0.996-ko-0.9.2/tests/eval/answer\n",
            "mecab-0.996-ko-0.9.2/tests/eval/system\n",
            "mecab-0.996-ko-0.9.2/tests/eval/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/test\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/mkdic.pl\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/latin/\n",
            "mecab-0.996-ko-0.9.2/tests/latin/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/latin/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/latin/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/latin/test\n",
            "mecab-0.996-ko-0.9.2/tests/latin/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/latin/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/latin/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/test\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/run-dics.sh\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/test\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/Makefile.in\n",
            "mecab-0.996-ko-0.9.2/ltmain.sh\n",
            "mecab-0.996-ko-0.9.2/config.rpath\n",
            "mecab-0.996-ko-0.9.2/config.h.in\n",
            "mecab-0.996-ko-0.9.2/mecabrc.in\n",
            "mecab-0.996-ko-0.9.2/GPL\n",
            "mecab-0.996-ko-0.9.2/Makefile.train\n",
            "mecab-0.996-ko-0.9.2/ChangeLog\n",
            "mecab-0.996-ko-0.9.2/install-sh\n",
            "mecab-0.996-ko-0.9.2/AUTHORS\n",
            "mecab-0.996-ko-0.9.2/doc/\n",
            "mecab-0.996-ko-0.9.2/doc/bindings.html\n",
            "mecab-0.996-ko-0.9.2/doc/posid.html\n",
            "mecab-0.996-ko-0.9.2/doc/unk.html\n",
            "mecab-0.996-ko-0.9.2/doc/learn.html\n",
            "mecab-0.996-ko-0.9.2/doc/format.html\n",
            "mecab-0.996-ko-0.9.2/doc/libmecab.html\n",
            "mecab-0.996-ko-0.9.2/doc/mecab.css\n",
            "mecab-0.996-ko-0.9.2/doc/feature.html\n",
            "mecab-0.996-ko-0.9.2/doc/Makefile.am\n",
            "mecab-0.996-ko-0.9.2/doc/soft.html\n",
            "mecab-0.996-ko-0.9.2/doc/en/\n",
            "mecab-0.996-ko-0.9.2/doc/en/bindings.html\n",
            "mecab-0.996-ko-0.9.2/doc/dic-detail.html\n",
            "mecab-0.996-ko-0.9.2/doc/flow.png\n",
            "mecab-0.996-ko-0.9.2/doc/mecab.html\n",
            "mecab-0.996-ko-0.9.2/doc/index.html\n",
            "mecab-0.996-ko-0.9.2/doc/result.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tab_a.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/globals_eval.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/classMeCab_1_1Tagger-members.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/functions_vars.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/doxygen.css\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tab_r.gif\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/classMeCab_1_1Lattice.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/functions.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/classMeCab_1_1Tagger.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/mecab_8h_source.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tabs.css\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/nav_f.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tab_b.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/globals.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/nav_h.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tab_h.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/classMeCab_1_1Model.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/globals_func.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/closed.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tab_l.gif\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/structmecab__path__t-members.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/functions_func.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/globals_type.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/classMeCab_1_1Lattice-members.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/structmecab__node__t.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/namespacemembers_func.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tab_s.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/structmecab__dictionary__info__t-members.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/namespacemembers_type.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/classMeCab_1_1Model-members.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/structmecab__dictionary__info__t.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/namespaces.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/namespacemembers.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/namespaceMeCab.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/structmecab__path__t.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/files.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/structmecab__node__t-members.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/index.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/annotated.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/globals_defs.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/classes.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/mecab_8h-source.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/doxygen.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tab_b.gif\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/bc_s.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/open.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/mecab_8h.html\n",
            "mecab-0.996-ko-0.9.2/doc/dic.html\n",
            "mecab-0.996-ko-0.9.2/doc/partial.html\n",
            "mecab-0.996-ko-0.9.2/doc/feature.png\n",
            "mecab-0.996-ko-0.9.2/doc/Makefile.in\n",
            "mecab-0.996-ko-0.9.2/missing\n",
            "mecab-0.996-ko-0.9.2/BSD\n",
            "mecab-0.996-ko-0.9.2/NEWS\n",
            "mecab-0.996-ko-0.9.2/mkinstalldirs\n",
            "mecab-0.996-ko-0.9.2/src/\n",
            "mecab-0.996-ko-0.9.2/src/dictionary.h\n",
            "mecab-0.996-ko-0.9.2/src/writer.h\n",
            "mecab-0.996-ko-0.9.2/src/utils.h\n",
            "mecab-0.996-ko-0.9.2/src/string_buffer.cpp\n",
            "mecab-0.996-ko-0.9.2/src/tokenizer.cpp\n",
            "mecab-0.996-ko-0.9.2/src/make.bat\n",
            "mecab-0.996-ko-0.9.2/src/mecab.h\n",
            "mecab-0.996-ko-0.9.2/src/freelist.h\n",
            "mecab-0.996-ko-0.9.2/src/string_buffer.h\n",
            "mecab-0.996-ko-0.9.2/src/learner_tagger.h\n",
            "mecab-0.996-ko-0.9.2/src/dictionary_compiler.cpp\n",
            "mecab-0.996-ko-0.9.2/src/eval.cpp\n",
            "mecab-0.996-ko-0.9.2/src/mecab-system-eval.cpp\n",
            "mecab-0.996-ko-0.9.2/src/darts.h\n",
            "mecab-0.996-ko-0.9.2/src/param.h\n",
            "mecab-0.996-ko-0.9.2/src/char_property.h\n",
            "mecab-0.996-ko-0.9.2/src/learner_node.h\n",
            "mecab-0.996-ko-0.9.2/src/mecab-dict-gen.cpp\n",
            "mecab-0.996-ko-0.9.2/src/mecab-dict-index.cpp\n",
            "mecab-0.996-ko-0.9.2/src/winmain.h\n",
            "mecab-0.996-ko-0.9.2/src/thread.h\n",
            "mecab-0.996-ko-0.9.2/src/context_id.cpp\n",
            "mecab-0.996-ko-0.9.2/src/Makefile.am\n",
            "mecab-0.996-ko-0.9.2/src/connector.h\n",
            "mecab-0.996-ko-0.9.2/src/common.h\n",
            "mecab-0.996-ko-0.9.2/src/dictionary_rewriter.cpp\n",
            "mecab-0.996-ko-0.9.2/src/Makefile.msvc.in\n",
            "mecab-0.996-ko-0.9.2/src/dictionary_rewriter.h\n",
            "mecab-0.996-ko-0.9.2/src/feature_index.h\n",
            "mecab-0.996-ko-0.9.2/src/iconv_utils.cpp\n",
            "mecab-0.996-ko-0.9.2/src/char_property.cpp\n",
            "mecab-0.996-ko-0.9.2/src/mecab-test-gen.cpp\n",
            "mecab-0.996-ko-0.9.2/src/tagger.cpp\n",
            "mecab-0.996-ko-0.9.2/src/mecab-cost-train.cpp\n",
            "mecab-0.996-ko-0.9.2/src/learner.cpp\n",
            "mecab-0.996-ko-0.9.2/src/dictionary.cpp\n",
            "mecab-0.996-ko-0.9.2/src/lbfgs.cpp\n",
            "mecab-0.996-ko-0.9.2/src/ucs.h\n",
            "mecab-0.996-ko-0.9.2/src/writer.cpp\n",
            "mecab-0.996-ko-0.9.2/src/learner_tagger.cpp\n",
            "mecab-0.996-ko-0.9.2/src/lbfgs.h\n",
            "mecab-0.996-ko-0.9.2/src/libmecab.cpp\n",
            "mecab-0.996-ko-0.9.2/src/tokenizer.h\n",
            "mecab-0.996-ko-0.9.2/src/mecab.cpp\n",
            "mecab-0.996-ko-0.9.2/src/utils.cpp\n",
            "mecab-0.996-ko-0.9.2/src/dictionary_generator.cpp\n",
            "mecab-0.996-ko-0.9.2/src/param.cpp\n",
            "mecab-0.996-ko-0.9.2/src/context_id.h\n",
            "mecab-0.996-ko-0.9.2/src/mmap.h\n",
            "mecab-0.996-ko-0.9.2/src/viterbi.h\n",
            "mecab-0.996-ko-0.9.2/src/viterbi.cpp\n",
            "mecab-0.996-ko-0.9.2/src/stream_wrapper.h\n",
            "mecab-0.996-ko-0.9.2/src/feature_index.cpp\n",
            "mecab-0.996-ko-0.9.2/src/nbest_generator.h\n",
            "mecab-0.996-ko-0.9.2/src/ucstable.h\n",
            "mecab-0.996-ko-0.9.2/src/nbest_generator.cpp\n",
            "mecab-0.996-ko-0.9.2/src/iconv_utils.h\n",
            "mecab-0.996-ko-0.9.2/src/connector.cpp\n",
            "mecab-0.996-ko-0.9.2/src/Makefile.in\n",
            "mecab-0.996-ko-0.9.2/src/scoped_ptr.h\n",
            "mecab-0.996-ko-0.9.2/Makefile.in\n",
            "+ cd mecab-0.996-ko-0.9.2\n",
            "+ ./configure\n",
            "checking for a BSD-compatible install... /usr/bin/install -c\n",
            "checking whether build environment is sane... yes\n",
            "checking for a thread-safe mkdir -p... /bin/mkdir -p\n",
            "checking for gawk... no\n",
            "checking for mawk... mawk\n",
            "checking whether make sets $(MAKE)... yes\n",
            "checking for gcc... gcc\n",
            "checking whether the C compiler works... yes\n",
            "checking for C compiler default output file name... a.out\n",
            "checking for suffix of executables... \n",
            "checking whether we are cross compiling... no\n",
            "checking for suffix of object files... o\n",
            "checking whether we are using the GNU C compiler... yes\n",
            "checking whether gcc accepts -g... yes\n",
            "checking for gcc option to accept ISO C89... none needed\n",
            "checking for style of include used by make... GNU\n",
            "checking dependency style of gcc... none\n",
            "checking for g++... g++\n",
            "checking whether we are using the GNU C++ compiler... yes\n",
            "checking whether g++ accepts -g... yes\n",
            "checking dependency style of g++... none\n",
            "checking how to run the C preprocessor... gcc -E\n",
            "checking for grep that handles long lines and -e... /bin/grep\n",
            "checking for egrep... /bin/grep -E\n",
            "checking whether gcc needs -traditional... no\n",
            "checking whether make sets $(MAKE)... (cached) yes\n",
            "checking build system type... x86_64-unknown-linux-gnu\n",
            "checking host system type... x86_64-unknown-linux-gnu\n",
            "checking how to print strings... printf\n",
            "checking for a sed that does not truncate output... /bin/sed\n",
            "checking for fgrep... /bin/grep -F\n",
            "checking for ld used by gcc... /usr/bin/ld\n",
            "checking if the linker (/usr/bin/ld) is GNU ld... yes\n",
            "checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B\n",
            "checking the name lister (/usr/bin/nm -B) interface... BSD nm\n",
            "checking whether ln -s works... yes\n",
            "checking the maximum length of command line arguments... 1572864\n",
            "checking whether the shell understands some XSI constructs... yes\n",
            "checking whether the shell understands \"+=\"... yes\n",
            "checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop\n",
            "checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop\n",
            "checking for /usr/bin/ld option to reload object files... -r\n",
            "checking for objdump... objdump\n",
            "checking how to recognize dependent libraries... pass_all\n",
            "checking for dlltool... dlltool\n",
            "checking how to associate runtime and link libraries... printf %s\\n\n",
            "checking for ar... ar\n",
            "checking for archiver @FILE support... @\n",
            "checking for strip... strip\n",
            "checking for ranlib... ranlib\n",
            "checking command to parse /usr/bin/nm -B output from gcc object... ok\n",
            "checking for sysroot... no\n",
            "./configure: line 7378: /usr/bin/file: No such file or directory\n",
            "checking for mt... no\n",
            "checking if : is a manifest tool... no\n",
            "checking for ANSI C header files... yes\n",
            "checking for sys/types.h... yes\n",
            "checking for sys/stat.h... yes\n",
            "checking for stdlib.h... yes\n",
            "checking for string.h... yes\n",
            "checking for memory.h... yes\n",
            "checking for strings.h... yes\n",
            "checking for inttypes.h... yes\n",
            "checking for stdint.h... yes\n",
            "checking for unistd.h... yes\n",
            "checking for dlfcn.h... yes\n",
            "checking for objdir... .libs\n",
            "checking if gcc supports -fno-rtti -fno-exceptions... no\n",
            "checking for gcc option to produce PIC... -fPIC -DPIC\n",
            "checking if gcc PIC flag -fPIC -DPIC works... yes\n",
            "checking if gcc static flag -static works... yes\n",
            "checking if gcc supports -c -o file.o... yes\n",
            "checking if gcc supports -c -o file.o... (cached) yes\n",
            "checking whether the gcc linker (/usr/bin/ld) supports shared libraries... yes\n",
            "checking whether -lc should be explicitly linked in... no\n",
            "checking dynamic linker characteristics... GNU/Linux ld.so\n",
            "checking how to hardcode library paths into programs... immediate\n",
            "checking whether stripping libraries is possible... yes\n",
            "checking if libtool supports shared libraries... yes\n",
            "checking whether to build shared libraries... yes\n",
            "checking whether to build static libraries... yes\n",
            "checking how to run the C++ preprocessor... g++ -E\n",
            "checking for ld used by g++... /usr/bin/ld\n",
            "checking if the linker (/usr/bin/ld) is GNU ld... yes\n",
            "checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes\n",
            "checking for g++ option to produce PIC... -fPIC -DPIC\n",
            "checking if g++ PIC flag -fPIC -DPIC works... yes\n",
            "checking if g++ static flag -static works... yes\n",
            "checking if g++ supports -c -o file.o... yes\n",
            "checking if g++ supports -c -o file.o... (cached) yes\n",
            "checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes\n",
            "checking dynamic linker characteristics... (cached) GNU/Linux ld.so\n",
            "checking how to hardcode library paths into programs... immediate\n",
            "checking for library containing strerror... none required\n",
            "checking whether byte ordering is bigendian... no\n",
            "checking for ld used by GCC... /usr/bin/ld\n",
            "checking if the linker (/usr/bin/ld) is GNU ld... yes\n",
            "checking for shared library run path origin... done\n",
            "checking for iconv... yes\n",
            "checking for working iconv... yes\n",
            "checking for iconv declaration... \n",
            "         extern size_t iconv (iconv_t cd, char * *inbuf, size_t *inbytesleft, char * *outbuf, size_t *outbytesleft);\n",
            "checking for ANSI C header files... (cached) yes\n",
            "checking for an ANSI C-conforming const... yes\n",
            "checking whether byte ordering is bigendian... (cached) no\n",
            "checking for string.h... (cached) yes\n",
            "checking for stdlib.h... (cached) yes\n",
            "checking for unistd.h... (cached) yes\n",
            "checking fcntl.h usability... yes\n",
            "checking fcntl.h presence... yes\n",
            "checking for fcntl.h... yes\n",
            "checking for stdint.h... (cached) yes\n",
            "checking for sys/stat.h... (cached) yes\n",
            "checking sys/mman.h usability... yes\n",
            "checking sys/mman.h presence... yes\n",
            "checking for sys/mman.h... yes\n",
            "checking sys/times.h usability... yes\n",
            "checking sys/times.h presence... yes\n",
            "checking for sys/times.h... yes\n",
            "checking for sys/types.h... (cached) yes\n",
            "checking dirent.h usability... yes\n",
            "checking dirent.h presence... yes\n",
            "checking for dirent.h... yes\n",
            "checking ctype.h usability... yes\n",
            "checking ctype.h presence... yes\n",
            "checking for ctype.h... yes\n",
            "checking for sys/types.h... (cached) yes\n",
            "checking io.h usability... no\n",
            "checking io.h presence... no\n",
            "checking for io.h... no\n",
            "checking windows.h usability... no\n",
            "checking windows.h presence... no\n",
            "checking for windows.h... no\n",
            "checking pthread.h usability... yes\n",
            "checking pthread.h presence... yes\n",
            "checking for pthread.h... yes\n",
            "checking for off_t... yes\n",
            "checking for size_t... yes\n",
            "checking size of char... 1\n",
            "checking size of short... 2\n",
            "checking size of int... 4\n",
            "checking size of long... 8\n",
            "checking size of long long... 8\n",
            "checking size of size_t... 8\n",
            "checking for size_t... (cached) yes\n",
            "checking for unsigned long long int... yes\n",
            "checking for stdlib.h... (cached) yes\n",
            "checking for unistd.h... (cached) yes\n",
            "checking for sys/param.h... yes\n",
            "checking for getpagesize... yes\n",
            "checking for working mmap... yes\n",
            "checking for main in -lstdc++... yes\n",
            "checking for pthread_create in -lpthread... yes\n",
            "checking for pthread_join in -lpthread... yes\n",
            "checking for getenv... yes\n",
            "checking for opendir... yes\n",
            "checking whether make is GNU Make... yes\n",
            "checking if g++ supports stl <vector> (required)... yes\n",
            "checking if g++ supports stl <list> (required)... yes\n",
            "checking if g++ supports stl <map> (required)... yes\n",
            "checking if g++ supports stl <set> (required)... yes\n",
            "checking if g++ supports stl <queue> (required)... yes\n",
            "checking if g++ supports stl <functional> (required)... yes\n",
            "checking if g++ supports stl <algorithm> (required)... yes\n",
            "checking if g++ supports stl <string> (required)... yes\n",
            "checking if g++ supports stl <iostream> (required)... yes\n",
            "checking if g++ supports stl <sstream> (required)... yes\n",
            "checking if g++ supports stl <fstream> (required)... yes\n",
            "checking if g++ supports template <class T> (required)... yes\n",
            "checking if g++ supports const_cast<> (required)... yes\n",
            "checking if g++ supports static_cast<> (required)... yes\n",
            "checking if g++ supports reinterpret_cast<> (required)... yes\n",
            "checking if g++ supports namespaces (required) ... yes\n",
            "checking if g++ supports __thread (optional)... yes\n",
            "checking if g++ supports template <class T> (required)... yes\n",
            "checking if g++ supports GCC native atomic operations (optional)... yes\n",
            "checking if g++ supports OSX native atomic operations (optional)... no\n",
            "checking if g++ environment provides all required features... yes\n",
            "configure: creating ./config.status\n",
            "config.status: creating Makefile\n",
            "config.status: creating src/Makefile\n",
            "config.status: creating src/Makefile.msvc\n",
            "config.status: creating man/Makefile\n",
            "config.status: creating doc/Makefile\n",
            "config.status: creating tests/Makefile\n",
            "config.status: creating swig/version.h\n",
            "config.status: creating mecab.iss\n",
            "config.status: creating mecab-config\n",
            "config.status: creating mecabrc\n",
            "config.status: creating config.h\n",
            "config.status: executing depfiles commands\n",
            "config.status: executing libtool commands\n",
            "config.status: executing default commands\n",
            "+ make\n",
            "make  all-recursive\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "Making all in src\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o viterbi.lo viterbi.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c viterbi.cpp  -fPIC -DPIC -o .libs/viterbi.o\n",
            "In file included from \u001b[01m\u001b[Kviterbi.cpp:14:0\u001b[m\u001b[K:\n",
            "\u001b[01m\u001b[Kparam.h:30:13:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K'\u001b[01m\u001b[KTarget {anonymous}::lexical_cast(Source) [with Target = std::__cxx11::basic_string<char>; Source = std::__cxx11::basic_string<char>]\u001b[m\u001b[K' defined but not used [\u001b[01;35m\u001b[K-Wunused-function\u001b[m\u001b[K]\n",
            " std::string \u001b[01;35m\u001b[Klexical_cast<std::string, std::string>\u001b[m\u001b[K(std::string arg) {\n",
            "             \u001b[01;35m\u001b[K^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\u001b[m\u001b[K\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c viterbi.cpp -o viterbi.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o tagger.lo tagger.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c tagger.cpp  -fPIC -DPIC -o .libs/tagger.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c tagger.cpp -o tagger.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o utils.lo utils.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c utils.cpp  -fPIC -DPIC -o .libs/utils.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c utils.cpp -o utils.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o eval.lo eval.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c eval.cpp  -fPIC -DPIC -o .libs/eval.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c eval.cpp -o eval.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o iconv_utils.lo iconv_utils.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c iconv_utils.cpp  -fPIC -DPIC -o .libs/iconv_utils.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c iconv_utils.cpp -o iconv_utils.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o dictionary_rewriter.lo dictionary_rewriter.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c dictionary_rewriter.cpp  -fPIC -DPIC -o .libs/dictionary_rewriter.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c dictionary_rewriter.cpp -o dictionary_rewriter.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o dictionary_generator.lo dictionary_generator.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c dictionary_generator.cpp  -fPIC -DPIC -o .libs/dictionary_generator.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c dictionary_generator.cpp -o dictionary_generator.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o dictionary_compiler.lo dictionary_compiler.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c dictionary_compiler.cpp  -fPIC -DPIC -o .libs/dictionary_compiler.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c dictionary_compiler.cpp -o dictionary_compiler.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o context_id.lo context_id.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c context_id.cpp  -fPIC -DPIC -o .libs/context_id.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c context_id.cpp -o context_id.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o connector.lo connector.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c connector.cpp  -fPIC -DPIC -o .libs/connector.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c connector.cpp -o connector.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o nbest_generator.lo nbest_generator.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c nbest_generator.cpp  -fPIC -DPIC -o .libs/nbest_generator.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c nbest_generator.cpp -o nbest_generator.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o writer.lo writer.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c writer.cpp  -fPIC -DPIC -o .libs/writer.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c writer.cpp -o writer.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o string_buffer.lo string_buffer.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c string_buffer.cpp  -fPIC -DPIC -o .libs/string_buffer.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c string_buffer.cpp -o string_buffer.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o param.lo param.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c param.cpp  -fPIC -DPIC -o .libs/param.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c param.cpp -o param.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o tokenizer.lo tokenizer.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c tokenizer.cpp  -fPIC -DPIC -o .libs/tokenizer.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c tokenizer.cpp -o tokenizer.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o char_property.lo char_property.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c char_property.cpp  -fPIC -DPIC -o .libs/char_property.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c char_property.cpp -o char_property.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o dictionary.lo dictionary.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c dictionary.cpp  -fPIC -DPIC -o .libs/dictionary.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c dictionary.cpp -o dictionary.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o feature_index.lo feature_index.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c feature_index.cpp  -fPIC -DPIC -o .libs/feature_index.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c feature_index.cpp -o feature_index.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o lbfgs.lo lbfgs.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c lbfgs.cpp  -fPIC -DPIC -o .libs/lbfgs.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c lbfgs.cpp -o lbfgs.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o learner_tagger.lo learner_tagger.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c learner_tagger.cpp  -fPIC -DPIC -o .libs/learner_tagger.o\n",
            "\u001b[01m\u001b[Klearner_tagger.cpp:25:7:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[K'\u001b[01m\u001b[Kchar* MeCab::{anonymous}::mystrdup(const string&)\u001b[m\u001b[K' defined but not used [\u001b[01;35m\u001b[K-Wunused-function\u001b[m\u001b[K]\n",
            " char *\u001b[01;35m\u001b[Kmystrdup\u001b[m\u001b[K(const std::string &str) {\n",
            "       \u001b[01;35m\u001b[K^~~~~~~~\u001b[m\u001b[K\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c learner_tagger.cpp -o learner_tagger.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o learner.lo learner.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c learner.cpp  -fPIC -DPIC -o .libs/learner.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c learner.cpp -o learner.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o libmecab.lo libmecab.cpp\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c libmecab.cpp  -fPIC -DPIC -o .libs/libmecab.o\n",
            "libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102 -DMECAB_DEFAULT_RC=\\\"/usr/local/etc/mecabrc\\\" -O3 -Wall -c libmecab.cpp -o libmecab.o >/dev/null 2>&1\n",
            "/bin/bash ../libtool --tag=CXX   --mode=link g++  -O3 -Wall  -no-undefined -version-info 2:0:0  -o libmecab.la -rpath /usr/local/lib viterbi.lo tagger.lo utils.lo eval.lo iconv_utils.lo dictionary_rewriter.lo dictionary_generator.lo dictionary_compiler.lo context_id.lo connector.lo nbest_generator.lo writer.lo string_buffer.lo param.lo tokenizer.lo char_property.lo dictionary.lo feature_index.lo lbfgs.lo learner_tagger.lo learner.lo libmecab.lo  -lpthread -lpthread  -lstdc++ \n",
            "libtool: link: g++  -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/7/crtbeginS.o  .libs/viterbi.o .libs/tagger.o .libs/utils.o .libs/eval.o .libs/iconv_utils.o .libs/dictionary_rewriter.o .libs/dictionary_generator.o .libs/dictionary_compiler.o .libs/context_id.o .libs/connector.o .libs/nbest_generator.o .libs/writer.o .libs/string_buffer.o .libs/param.o .libs/tokenizer.o .libs/char_property.o .libs/dictionary.o .libs/feature_index.o .libs/lbfgs.o .libs/learner_tagger.o .libs/learner.o .libs/libmecab.o   -lpthread -L/usr/lib/gcc/x86_64-linux-gnu/7 -L/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/local/cuda/lib64/stubs -L/usr/lib/gcc/x86_64-linux-gnu/7/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/7/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/crtn.o  -O3   -Wl,-soname -Wl,libmecab.so.2 -o .libs/libmecab.so.2.0.0\n",
            "libtool: link: (cd \".libs\" && rm -f \"libmecab.so.2\" && ln -s \"libmecab.so.2.0.0\" \"libmecab.so.2\")\n",
            "libtool: link: (cd \".libs\" && rm -f \"libmecab.so\" && ln -s \"libmecab.so.2.0.0\" \"libmecab.so\")\n",
            "libtool: link: ar cru .libs/libmecab.a  viterbi.o tagger.o utils.o eval.o iconv_utils.o dictionary_rewriter.o dictionary_generator.o dictionary_compiler.o context_id.o connector.o nbest_generator.o writer.o string_buffer.o param.o tokenizer.o char_property.o dictionary.o feature_index.o lbfgs.o learner_tagger.o learner.o libmecab.o\n",
            "ar: `u' modifier ignored since `D' is the default (see `U')\n",
            "libtool: link: ranlib .libs/libmecab.a\n",
            "libtool: link: ( cd \".libs\" && rm -f \"libmecab.la\" && ln -s \"../libmecab.la\" \"libmecab.la\" )\n",
            "g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o mecab.o mecab.cpp\n",
            "/bin/bash ../libtool --tag=CXX   --mode=link g++  -O3 -Wall    -o mecab mecab.o libmecab.la -lpthread -lpthread  -lstdc++ \n",
            "libtool: link: g++ -O3 -Wall -o .libs/mecab mecab.o  ./.libs/libmecab.so -lpthread -lstdc++\n",
            "g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o mecab-dict-index.o mecab-dict-index.cpp\n",
            "/bin/bash ../libtool --tag=CXX   --mode=link g++  -O3 -Wall    -o mecab-dict-index mecab-dict-index.o libmecab.la -lpthread -lpthread  -lstdc++ \n",
            "libtool: link: g++ -O3 -Wall -o .libs/mecab-dict-index mecab-dict-index.o  ./.libs/libmecab.so -lpthread -lstdc++\n",
            "g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o mecab-dict-gen.o mecab-dict-gen.cpp\n",
            "/bin/bash ../libtool --tag=CXX   --mode=link g++  -O3 -Wall    -o mecab-dict-gen mecab-dict-gen.o libmecab.la -lpthread -lpthread  -lstdc++ \n",
            "libtool: link: g++ -O3 -Wall -o .libs/mecab-dict-gen mecab-dict-gen.o  ./.libs/libmecab.so -lpthread -lstdc++\n",
            "g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o mecab-cost-train.o mecab-cost-train.cpp\n",
            "/bin/bash ../libtool --tag=CXX   --mode=link g++  -O3 -Wall    -o mecab-cost-train mecab-cost-train.o libmecab.la -lpthread -lpthread  -lstdc++ \n",
            "libtool: link: g++ -O3 -Wall -o .libs/mecab-cost-train mecab-cost-train.o  ./.libs/libmecab.so -lpthread -lstdc++\n",
            "g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o mecab-system-eval.o mecab-system-eval.cpp\n",
            "/bin/bash ../libtool --tag=CXX   --mode=link g++  -O3 -Wall    -o mecab-system-eval mecab-system-eval.o libmecab.la -lpthread -lpthread  -lstdc++ \n",
            "libtool: link: g++ -O3 -Wall -o .libs/mecab-system-eval mecab-system-eval.o  ./.libs/libmecab.so -lpthread -lstdc++\n",
            "g++ -DHAVE_CONFIG_H -I. -I.. -DDIC_VERSION=102   -DMECAB_DEFAULT_RC=\"\\\"/usr/local/etc/mecabrc\\\"\"    -O3 -Wall  -c -o mecab-test-gen.o mecab-test-gen.cpp\n",
            "/bin/bash ../libtool --tag=CXX   --mode=link g++  -O3 -Wall    -o mecab-test-gen mecab-test-gen.o libmecab.la -lpthread -lpthread  -lstdc++ \n",
            "libtool: link: g++ -O3 -Wall -o .libs/mecab-test-gen mecab-test-gen.o  ./.libs/libmecab.so -lpthread -lstdc++\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "Making all in man\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "make[2]: Nothing to be done for 'all'.\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "Making all in doc\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "make[2]: Nothing to be done for 'all'.\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "Making all in tests\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[2]: Nothing to be done for 'all'.\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "+ make check\n",
            "Making check in src\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "make[1]: Nothing to be done for 'check'.\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "Making check in man\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "make[1]: Nothing to be done for 'check'.\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "Making check in doc\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "make[1]: Nothing to be done for 'check'.\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "Making check in tests\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make  check-TESTS\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./unk.def ... 2\n",
            "emitting double-array: 100% |###########################################| \n",
            "./model.def is not found. skipped.\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./dic.csv ... 177\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading ./matrix.def ... 178x178\n",
            "emitting matrix      : 100% |###########################################| \n",
            "\n",
            "done!\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./unk.def ... 2\n",
            "emitting double-array: 100% |###########################################| \n",
            "./model.def is not found. skipped.\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./dic.csv ... 83\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading ./matrix.def ... 84x84\n",
            "emitting matrix      : 100% |###########################################| \n",
            "\n",
            "done!\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./unk.def ... 2\n",
            "emitting double-array: 100% |###########################################| \n",
            "./model.def is not found. skipped.\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./dic.csv ... 450\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading ./matrix.def ... 1x1\n",
            "\n",
            "done!\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./unk.def ... 2\n",
            "emitting double-array: 100% |###########################################| \n",
            "./model.def is not found. skipped.\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./dic.csv ... 162\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading ./matrix.def ... 3x3\n",
            "emitting matrix      : 100% |###########################################| \n",
            "\n",
            "done!\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./unk.def ... 2\n",
            "emitting double-array: 100% |###########################################| \n",
            "./model.def is not found. skipped.\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./dic.csv ... 4\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading ./matrix.def ... 1x1\n",
            "\n",
            "done!\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./unk.def ... 11\n",
            "emitting double-array: 100% |###########################################| \n",
            "./model.def is not found. skipped.\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./dic.csv ... 1\n",
            "reading ./matrix.def ... 1x1\n",
            "\n",
            "done!\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./unk.def ... 2\n",
            "emitting double-array: 100% |###########################################| \n",
            "./model.def is not found. skipped.\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./dic.csv ... 1\n",
            "reading ./matrix.def ... 1x1\n",
            "\n",
            "done!\n",
            "PASS: run-dics.sh\n",
            "PASS: run-eval.sh\n",
            "seed/pos-id.def is not found. minimum setting is used\n",
            "reading seed/unk.def ... 40\n",
            "emitting double-array: 100% |###########################################| \n",
            "seed/model.def is not found. skipped.\n",
            "seed/pos-id.def is not found. minimum setting is used\n",
            "reading seed/dic.csv ... 4335\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading seed/matrix.def ... 1x1\n",
            "\n",
            "done!\n",
            "reading corpus ...\n",
            "Number of sentences: 34\n",
            "Number of features:  64108\n",
            "eta:                 0.00005\n",
            "freq:                1\n",
            "eval-size:           6\n",
            "unk-eval-size:       4\n",
            "threads:             1\n",
            "charset:             EUC-JP\n",
            "C(sigma^2):          1.00000\n",
            "\n",
            "iter=0 err=1.00000 F=0.35771 target=2406.28355 diff=1.00000\n",
            "iter=1 err=0.97059 F=0.65652 target=1484.25231 diff=0.38318\n",
            "iter=2 err=0.91176 F=0.79331 target=863.32765 diff=0.41834\n",
            "iter=3 err=0.85294 F=0.89213 target=596.72480 diff=0.30881\n",
            "iter=4 err=0.61765 F=0.95467 target=336.30744 diff=0.43641\n",
            "iter=5 err=0.50000 F=0.96702 target=246.53039 diff=0.26695\n",
            "iter=6 err=0.35294 F=0.95472 target=188.93963 diff=0.23361\n",
            "iter=7 err=0.20588 F=0.99106 target=168.62665 diff=0.10751\n",
            "iter=8 err=0.05882 F=0.99777 target=158.64865 diff=0.05917\n",
            "iter=9 err=0.08824 F=0.99665 target=154.14530 diff=0.02839\n",
            "iter=10 err=0.08824 F=0.99665 target=151.94257 diff=0.01429\n",
            "iter=11 err=0.02941 F=0.99888 target=147.20825 diff=0.03116\n",
            "iter=12 err=0.00000 F=1.00000 target=147.34956 diff=0.00096\n",
            "iter=13 err=0.02941 F=0.99888 target=146.32592 diff=0.00695\n",
            "iter=14 err=0.00000 F=1.00000 target=145.77299 diff=0.00378\n",
            "iter=15 err=0.02941 F=0.99888 target=145.24641 diff=0.00361\n",
            "iter=16 err=0.00000 F=1.00000 target=144.96490 diff=0.00194\n",
            "iter=17 err=0.02941 F=0.99888 target=144.90246 diff=0.00043\n",
            "iter=18 err=0.00000 F=1.00000 target=144.75959 diff=0.00099\n",
            "iter=19 err=0.00000 F=1.00000 target=144.71727 diff=0.00029\n",
            "iter=20 err=0.00000 F=1.00000 target=144.66337 diff=0.00037\n",
            "iter=21 err=0.00000 F=1.00000 target=144.61349 diff=0.00034\n",
            "iter=22 err=0.00000 F=1.00000 target=144.62987 diff=0.00011\n",
            "iter=23 err=0.00000 F=1.00000 target=144.60060 diff=0.00020\n",
            "iter=24 err=0.00000 F=1.00000 target=144.59125 diff=0.00006\n",
            "iter=25 err=0.00000 F=1.00000 target=144.58619 diff=0.00004\n",
            "iter=26 err=0.00000 F=1.00000 target=144.58219 diff=0.00003\n",
            "iter=27 err=0.00000 F=1.00000 target=144.58059 diff=0.00001\n",
            "\n",
            "Done! writing model file ... \n",
            "model-ipadic.c1.0.f1.model is not a binary model. reopen it as text mode...\n",
            "reading seed/unk.def ... 40\n",
            "reading seed/dic.csv ... 4335\n",
            "emitting model-ipadic.c1.0.f1.dic/left-id.def/ model-ipadic.c1.0.f1.dic/right-id.def\n",
            "emitting model-ipadic.c1.0.f1.dic/unk.def ... 40\n",
            "emitting model-ipadic.c1.0.f1.dic/dic.csv ... 4335\n",
            "emitting matrix      : 100% |###########################################| \n",
            "copying seed/char.def to model-ipadic.c1.0.f1.dic/char.def\n",
            "copying seed/rewrite.def to model-ipadic.c1.0.f1.dic/rewrite.def\n",
            "copying seed/dicrc to model-ipadic.c1.0.f1.dic/dicrc\n",
            "copying seed/feature.def to model-ipadic.c1.0.f1.dic/feature.def\n",
            "copying model-ipadic.c1.0.f1.model to model-ipadic.c1.0.f1.dic/model.def\n",
            "\n",
            "done!\n",
            "model-ipadic.c1.0.f1.dic/pos-id.def is not found. minimum setting is used\n",
            "reading model-ipadic.c1.0.f1.dic/unk.def ... 40\n",
            "emitting double-array: 100% |###########################################| \n",
            "model-ipadic.c1.0.f1.dic/pos-id.def is not found. minimum setting is used\n",
            "reading model-ipadic.c1.0.f1.dic/dic.csv ... 4335\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading model-ipadic.c1.0.f1.dic/matrix.def ... 346x346\n",
            "emitting matrix      : 100% |###########################################| \n",
            "\n",
            "done!\n",
            "              precision          recall         F\n",
            "LEVEL 0:    12.8959(57/442) 11.8998(57/479) 12.3779\n",
            "LEVEL 1:    12.2172(54/442) 11.2735(54/479) 11.7264\n",
            "LEVEL 2:    11.7647(52/442) 10.8559(52/479) 11.2921\n",
            "LEVEL 4:    11.7647(52/442) 10.8559(52/479) 11.2921\n",
            "PASS: run-cost-train.sh\n",
            "==================\n",
            "All 3 tests passed\n",
            "==================\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "+ sudo make install\n",
            "Making install in src\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "test -z \"/usr/local/lib\" || /bin/mkdir -p \"/usr/local/lib\"\n",
            " /bin/bash ../libtool   --mode=install /usr/bin/install -c   libmecab.la '/usr/local/lib'\n",
            "libtool: install: /usr/bin/install -c .libs/libmecab.so.2.0.0 /usr/local/lib/libmecab.so.2.0.0\n",
            "libtool: install: (cd /usr/local/lib && { ln -s -f libmecab.so.2.0.0 libmecab.so.2 || { rm -f libmecab.so.2 && ln -s libmecab.so.2.0.0 libmecab.so.2; }; })\n",
            "libtool: install: (cd /usr/local/lib && { ln -s -f libmecab.so.2.0.0 libmecab.so || { rm -f libmecab.so && ln -s libmecab.so.2.0.0 libmecab.so; }; })\n",
            "libtool: install: /usr/bin/install -c .libs/libmecab.lai /usr/local/lib/libmecab.la\n",
            "libtool: install: /usr/bin/install -c .libs/libmecab.a /usr/local/lib/libmecab.a\n",
            "libtool: install: chmod 644 /usr/local/lib/libmecab.a\n",
            "libtool: install: ranlib /usr/local/lib/libmecab.a\n",
            "libtool: finish: PATH=\"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/sbin\" ldconfig -n /usr/local/lib\n",
            "----------------------------------------------------------------------\n",
            "Libraries have been installed in:\n",
            "   /usr/local/lib\n",
            "\n",
            "If you ever happen to want to link against installed libraries\n",
            "in a given directory, LIBDIR, you must either use libtool, and\n",
            "specify the full pathname of the library, or use the `-LLIBDIR'\n",
            "flag during linking and do at least one of the following:\n",
            "   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable\n",
            "     during execution\n",
            "   - add LIBDIR to the `LD_RUN_PATH' environment variable\n",
            "     during linking\n",
            "   - use the `-Wl,-rpath -Wl,LIBDIR' linker flag\n",
            "   - have your system administrator add LIBDIR to `/etc/ld.so.conf'\n",
            "\n",
            "See any operating system documentation about shared libraries for\n",
            "more information, such as the ld(1) and ld.so(8) manual pages.\n",
            "----------------------------------------------------------------------\n",
            "test -z \"/usr/local/bin\" || /bin/mkdir -p \"/usr/local/bin\"\n",
            "  /bin/bash ../libtool   --mode=install /usr/bin/install -c mecab '/usr/local/bin'\n",
            "libtool: install: /usr/bin/install -c .libs/mecab /usr/local/bin/mecab\n",
            "test -z \"/usr/local/libexec/mecab\" || /bin/mkdir -p \"/usr/local/libexec/mecab\"\n",
            "  /bin/bash ../libtool   --mode=install /usr/bin/install -c mecab-dict-index mecab-dict-gen mecab-cost-train mecab-system-eval mecab-test-gen '/usr/local/libexec/mecab'\n",
            "libtool: install: /usr/bin/install -c .libs/mecab-dict-index /usr/local/libexec/mecab/mecab-dict-index\n",
            "libtool: install: /usr/bin/install -c .libs/mecab-dict-gen /usr/local/libexec/mecab/mecab-dict-gen\n",
            "libtool: install: /usr/bin/install -c .libs/mecab-cost-train /usr/local/libexec/mecab/mecab-cost-train\n",
            "libtool: install: /usr/bin/install -c .libs/mecab-system-eval /usr/local/libexec/mecab/mecab-system-eval\n",
            "libtool: install: /usr/bin/install -c .libs/mecab-test-gen /usr/local/libexec/mecab/mecab-test-gen\n",
            "test -z \"/usr/local/include\" || /bin/mkdir -p \"/usr/local/include\"\n",
            " /usr/bin/install -c -m 644 mecab.h '/usr/local/include'\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "Making install in man\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "make[2]: Nothing to be done for 'install-exec-am'.\n",
            "test -z \"/usr/local/share/man/man1\" || /bin/mkdir -p \"/usr/local/share/man/man1\"\n",
            " /usr/bin/install -c -m 644 mecab.1 '/usr/local/share/man/man1'\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "Making install in doc\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "make[2]: Nothing to be done for 'install-exec-am'.\n",
            "make[2]: Nothing to be done for 'install-data-am'.\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "Making install in tests\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[2]: Nothing to be done for 'install-exec-am'.\n",
            "make[2]: Nothing to be done for 'install-data-am'.\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "test -z \"/usr/local/bin\" || /bin/mkdir -p \"/usr/local/bin\"\n",
            " /usr/bin/install -c mecab-config '/usr/local/bin'\n",
            "test -z \"/usr/local/etc\" || /bin/mkdir -p \"/usr/local/etc\"\n",
            " /usr/bin/install -c -m 644 mecabrc '/usr/local/etc'\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "+ [[ -d /usr/local/lib/mecab/dic/mecab-ko-dic ]]\n",
            "+ echo 'Install mecab-ko-dic'\n",
            "Install mecab-ko-dic\n",
            "+ install_mecab_ko_dic\n",
            "+ echo 'Install mecab-ko-dic'\n",
            "Install mecab-ko-dic\n",
            "+ cd /tmp\n",
            "+ curl -LO https://bitbucket.org/eunjeon/mecab-ko-dic/downloads/mecab-ko-dic-2.1.1-20180720.tar.gz\n",
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\n",
            "100 47.4M  100 47.4M    0     0  7860k      0  0:00:06  0:00:06 --:--:--  9.8M\n",
            "+ tar -zxvf mecab-ko-dic-2.1.1-20180720.tar.gz\n",
            "mecab-ko-dic-2.1.1-20180720/\n",
            "mecab-ko-dic-2.1.1-20180720/configure\n",
            "mecab-ko-dic-2.1.1-20180720/COPYING\n",
            "mecab-ko-dic-2.1.1-20180720/autogen.sh\n",
            "mecab-ko-dic-2.1.1-20180720/Place-station.csv\n",
            "mecab-ko-dic-2.1.1-20180720/NNG.csv\n",
            "mecab-ko-dic-2.1.1-20180720/README\n",
            "mecab-ko-dic-2.1.1-20180720/EF.csv\n",
            "mecab-ko-dic-2.1.1-20180720/MAG.csv\n",
            "mecab-ko-dic-2.1.1-20180720/Preanalysis.csv\n",
            "mecab-ko-dic-2.1.1-20180720/NNB.csv\n",
            "mecab-ko-dic-2.1.1-20180720/Person-actor.csv\n",
            "mecab-ko-dic-2.1.1-20180720/VV.csv\n",
            "mecab-ko-dic-2.1.1-20180720/Makefile.in\n",
            "mecab-ko-dic-2.1.1-20180720/matrix.def\n",
            "mecab-ko-dic-2.1.1-20180720/EC.csv\n",
            "mecab-ko-dic-2.1.1-20180720/NNBC.csv\n",
            "mecab-ko-dic-2.1.1-20180720/clean\n",
            "mecab-ko-dic-2.1.1-20180720/ChangeLog\n",
            "mecab-ko-dic-2.1.1-20180720/J.csv\n",
            "mecab-ko-dic-2.1.1-20180720/.keep\n",
            "mecab-ko-dic-2.1.1-20180720/feature.def\n",
            "mecab-ko-dic-2.1.1-20180720/Foreign.csv\n",
            "mecab-ko-dic-2.1.1-20180720/XPN.csv\n",
            "mecab-ko-dic-2.1.1-20180720/EP.csv\n",
            "mecab-ko-dic-2.1.1-20180720/NR.csv\n",
            "mecab-ko-dic-2.1.1-20180720/left-id.def\n",
            "mecab-ko-dic-2.1.1-20180720/Place.csv\n",
            "mecab-ko-dic-2.1.1-20180720/Symbol.csv\n",
            "mecab-ko-dic-2.1.1-20180720/dicrc\n",
            "mecab-ko-dic-2.1.1-20180720/NP.csv\n",
            "mecab-ko-dic-2.1.1-20180720/ETM.csv\n",
            "mecab-ko-dic-2.1.1-20180720/IC.csv\n",
            "mecab-ko-dic-2.1.1-20180720/Place-address.csv\n",
            "mecab-ko-dic-2.1.1-20180720/Group.csv\n",
            "mecab-ko-dic-2.1.1-20180720/model.def\n",
            "mecab-ko-dic-2.1.1-20180720/XSN.csv\n",
            "mecab-ko-dic-2.1.1-20180720/INSTALL\n",
            "mecab-ko-dic-2.1.1-20180720/rewrite.def\n",
            "mecab-ko-dic-2.1.1-20180720/Inflect.csv\n",
            "mecab-ko-dic-2.1.1-20180720/configure.ac\n",
            "mecab-ko-dic-2.1.1-20180720/NNP.csv\n",
            "mecab-ko-dic-2.1.1-20180720/CoinedWord.csv\n",
            "mecab-ko-dic-2.1.1-20180720/XSV.csv\n",
            "mecab-ko-dic-2.1.1-20180720/pos-id.def\n",
            "mecab-ko-dic-2.1.1-20180720/Makefile.am\n",
            "mecab-ko-dic-2.1.1-20180720/unk.def\n",
            "mecab-ko-dic-2.1.1-20180720/missing\n",
            "mecab-ko-dic-2.1.1-20180720/VCP.csv\n",
            "mecab-ko-dic-2.1.1-20180720/install-sh\n",
            "mecab-ko-dic-2.1.1-20180720/Hanja.csv\n",
            "mecab-ko-dic-2.1.1-20180720/MAJ.csv\n",
            "mecab-ko-dic-2.1.1-20180720/XSA.csv\n",
            "mecab-ko-dic-2.1.1-20180720/Wikipedia.csv\n",
            "mecab-ko-dic-2.1.1-20180720/tools/\n",
            "mecab-ko-dic-2.1.1-20180720/tools/add-userdic.sh\n",
            "mecab-ko-dic-2.1.1-20180720/tools/mecab-bestn.sh\n",
            "mecab-ko-dic-2.1.1-20180720/tools/convert_for_using_store.sh\n",
            "mecab-ko-dic-2.1.1-20180720/user-dic/\n",
            "mecab-ko-dic-2.1.1-20180720/user-dic/nnp.csv\n",
            "mecab-ko-dic-2.1.1-20180720/user-dic/place.csv\n",
            "mecab-ko-dic-2.1.1-20180720/user-dic/person.csv\n",
            "mecab-ko-dic-2.1.1-20180720/user-dic/README.md\n",
            "mecab-ko-dic-2.1.1-20180720/NorthKorea.csv\n",
            "mecab-ko-dic-2.1.1-20180720/VX.csv\n",
            "mecab-ko-dic-2.1.1-20180720/right-id.def\n",
            "mecab-ko-dic-2.1.1-20180720/VA.csv\n",
            "mecab-ko-dic-2.1.1-20180720/char.def\n",
            "mecab-ko-dic-2.1.1-20180720/NEWS\n",
            "mecab-ko-dic-2.1.1-20180720/MM.csv\n",
            "mecab-ko-dic-2.1.1-20180720/ETN.csv\n",
            "mecab-ko-dic-2.1.1-20180720/AUTHORS\n",
            "mecab-ko-dic-2.1.1-20180720/Person.csv\n",
            "mecab-ko-dic-2.1.1-20180720/XR.csv\n",
            "mecab-ko-dic-2.1.1-20180720/VCN.csv\n",
            "+ cd mecab-ko-dic-2.1.1-20180720\n",
            "+ ./autogen.sh\n",
            "Looking in current directory for macros.\n",
            "configure.ac:2: warning: AM_INIT_AUTOMAKE: two- and three-arguments forms are deprecated.  For more info, see:\n",
            "configure.ac:2: http://www.gnu.org/software/automake/manual/automake.html#Modernize-AM_005fINIT_005fAUTOMAKE-invocation\n",
            "+ ./configure\n",
            "checking for a BSD-compatible install... /usr/bin/install -c\n",
            "checking whether build environment is sane... yes\n",
            "/tmp/mecab-ko-dic-2.1.1-20180720/missing: Unknown `--is-lightweight' option\n",
            "Try `/tmp/mecab-ko-dic-2.1.1-20180720/missing --help' for more information\n",
            "configure: WARNING: 'missing' script is too old or missing\n",
            "checking for a thread-safe mkdir -p... /bin/mkdir -p\n",
            "checking for gawk... no\n",
            "checking for mawk... mawk\n",
            "checking whether make sets $(MAKE)... yes\n",
            "checking whether make supports nested variables... yes\n",
            "checking for mecab-config... /usr/local/bin/mecab-config\n",
            "checking that generated files are newer than configure... done\n",
            "configure: creating ./config.status\n",
            "config.status: creating Makefile\n",
            "+ [[ Linux == \\L\\i\\n\\u\\x ]]\n",
            "+ mecab-config --libs-only-L\n",
            "+ sudo tee /etc/ld.so.conf.d/mecab.conf\n",
            "/usr/local/lib\n",
            "+ sudo ldconfig\n",
            "/sbin/ldconfig.real: /usr/local/lib/python3.7/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link\n",
            "\n",
            "+ make\n",
            "/usr/local/libexec/mecab/mecab-dict-index -d . -o . -f UTF-8 -t UTF-8\n",
            "reading ./unk.def ... 13\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading ./Wikipedia.csv ... 36762\n",
            "reading ./XSA.csv ... 19\n",
            "reading ./J.csv ... 416\n",
            "reading ./ETM.csv ... 133\n",
            "reading ./NNG.csv ... 208524\n",
            "reading ./NP.csv ... 342\n",
            "reading ./ETN.csv ... 14\n",
            "reading ./VA.csv ... 2360\n",
            "reading ./IC.csv ... 1305\n",
            "reading ./Place-address.csv ... 19301\n",
            "reading ./EC.csv ... 2547\n",
            "reading ./NNP.csv ... 2371\n",
            "reading ./VCP.csv ... 9\n",
            "reading ./MAG.csv ... 14242\n",
            "reading ./NNB.csv ... 140\n",
            "reading ./XSN.csv ... 124\n",
            "reading ./Person.csv ... 196459\n",
            "reading ./EF.csv ... 1820\n",
            "reading ./Hanja.csv ... 125750\n",
            "reading ./NR.csv ... 482\n",
            "reading ./VX.csv ... 125\n",
            "reading ./MAJ.csv ... 240\n",
            "reading ./EP.csv ... 51\n",
            "reading ./XR.csv ... 3637\n",
            "reading ./Person-actor.csv ... 99230\n",
            "reading ./Inflect.csv ... 44820\n",
            "reading ./Place.csv ... 30303\n",
            "reading ./XPN.csv ... 83\n",
            "reading ./MM.csv ... 453\n",
            "reading ./Preanalysis.csv ... 5\n",
            "reading ./NNBC.csv ... 677\n",
            "reading ./Symbol.csv ... 16\n",
            "reading ./CoinedWord.csv ... 148\n",
            "reading ./NorthKorea.csv ... 3\n",
            "reading ./Group.csv ... 3176\n",
            "reading ./Foreign.csv ... 11690\n",
            "reading ./XSV.csv ... 23\n",
            "reading ./Place-station.csv ... 1145\n",
            "reading ./VV.csv ... 7331\n",
            "reading ./VCN.csv ... 7\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading ./matrix.def ... 3822x2693\n",
            "emitting matrix      : 100% |###########################################| \n",
            "\n",
            "done!\n",
            "echo To enable dictionary, rewrite /usr/local/etc/mecabrc as \\\"dicdir = /usr/local/lib/mecab/dic/mecab-ko-dic\\\"\n",
            "To enable dictionary, rewrite /usr/local/etc/mecabrc as \"dicdir = /usr/local/lib/mecab/dic/mecab-ko-dic\"\n",
            "+ sudo sh -c 'echo \"dicdir=/usr/local/lib/mecab/dic/mecab-ko-dic\" > /usr/local/etc/mecabrc'\n",
            "+ sudo make install\n",
            "make[1]: Entering directory '/tmp/mecab-ko-dic-2.1.1-20180720'\n",
            "make[1]: Nothing to be done for 'install-exec-am'.\n",
            " /bin/mkdir -p '/usr/local/lib/mecab/dic/mecab-ko-dic'\n",
            " /usr/bin/install -c -m 644 model.bin matrix.bin char.bin sys.dic unk.dic left-id.def right-id.def rewrite.def pos-id.def dicrc '/usr/local/lib/mecab/dic/mecab-ko-dic'\n",
            "make[1]: Leaving directory '/tmp/mecab-ko-dic-2.1.1-20180720'\n",
            "++ python3 -c 'import pkgutil; print(1 if pkgutil.find_loader(\"MeCab\") else 0)'\n",
            "+ [[ 0 == \\1 ]]\n",
            "+ echo 'Install mecab-python'\n",
            "Install mecab-python\n",
            "+ install_mecab_python\n",
            "+ pushd /tmp\n",
            "/tmp /tmp/mecab-ko-dic-2.1.1-20180720\n",
            "+ [[ ! -d mecab-python-0.996 ]]\n",
            "+ git clone https://bitbucket.org/eunjeon/mecab-python-0.996.git\n",
            "Cloning into 'mecab-python-0.996'...\n",
            "Unpacking objects: 100% (17/17), done.\n",
            "+ popd\n",
            "/tmp/mecab-ko-dic-2.1.1-20180720\n",
            "+ python3 -m pip install /tmp/mecab-python-0.996\n",
            "Processing /tmp/mecab-python-0.996\n",
            "\u001b[33m  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.\n",
            "   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.\u001b[0m\n",
            "Building wheels for collected packages: mecab-python\n",
            "  Building wheel for mecab-python (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for mecab-python: filename=mecab_python-0.996_ko_0.9.2-cp37-cp37m-linux_x86_64.whl size=141813 sha256=94621c34969b833a5188f1e365872f45654922aaf29f70694b7767ecb3b9cada\n",
            "  Stored in directory: /root/.cache/pip/wheels/40/7b/9f/2922869bef86c3354ae7034f7a3647c573ee1997c2dad0290a\n",
            "\u001b[33m  WARNING: Built wheel for mecab-python is invalid: Metadata 1.2 mandates PEP 440 version, but '0.996-ko-0.9.2' is not\u001b[0m\n",
            "Failed to build mecab-python\n",
            "Installing collected packages: mecab-python\n",
            "    Running setup.py install for mecab-python ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[33m  DEPRECATION: mecab-python was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.\u001b[0m\n",
            "Successfully installed mecab-python-0.996-ko-0.9.2\n",
            "+ echo Done.\n",
            "Done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hJhzBOckvqr9"
      },
      "source": [
        "#### 단어 토큰화"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P-34oqwZwsC3"
      },
      "source": [
        "* 한국어는 공백으로 단어를 분리해도 조사, 접속사 등이 남아 분석에 어려움이 있음\n",
        "* 이를 해결해주는 한국어 토큰화는 조사, 접속사를 분리해주거나 제거\n",
        "* 한국어 토큰화를 사용하기 위해선 `konlpy`와 `mecab`이라는 라이브러리가 필요"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8m9Inqu0wrd5"
      },
      "source": [
        "from konlpy.tag import Mecab\n",
        "tagger = Mecab()"
      ],
      "execution_count": 91,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mwoU6RioB14R",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bd648bc1-66b8-4204-d425-be3fde4555e8"
      },
      "source": [
        "sentence = '언제나 현재에 집중할 수 있다면 행복할것이다.'\n",
        "tagger.pos(sentence)"
      ],
      "execution_count": 92,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('언제나', 'MAG'),\n",
              " ('현재', 'NNG'),\n",
              " ('에', 'JKB'),\n",
              " ('집중', 'NNG'),\n",
              " ('할', 'XSV+ETM'),\n",
              " ('수', 'NNB'),\n",
              " ('있', 'VV'),\n",
              " ('다면', 'EC'),\n",
              " ('행복', 'NNG'),\n",
              " ('할', 'XSV+ETM'),\n",
              " ('것', 'NNB'),\n",
              " ('이', 'VCP'),\n",
              " ('다', 'EF'),\n",
              " ('.', 'SF')]"
            ]
          },
          "metadata": {},
          "execution_count": 92
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WKg9p-SlwyRe"
      },
      "source": [
        "* 토큰화만 실행할 때는 `tagger.morphs()`라는 함수를 이용"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ist7zW_RwvcO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8c253df8-7cf0-4579-db33-99fd9ad77874"
      },
      "source": [
        "tagger.morphs(sentence)"
      ],
      "execution_count": 93,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['언제나', '현재', '에', '집중', '할', '수', '있', '다면', '행복', '할', '것', '이', '다', '.']"
            ]
          },
          "metadata": {},
          "execution_count": 93
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q7u6-78hw0gD"
      },
      "source": [
        "* 형태소만 사용하고 싶을 때는 `tagger.nouns()`라는 함수를 이용해 조사, 접속사 등을 제거 가능"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1D_Bg0gewwm9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7af809bc-9d7f-4dfc-99ae-5e75ffb11652"
      },
      "source": [
        "tagger.nouns(sentence)"
      ],
      "execution_count": 94,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['현재', '집중', '수', '행복', '것']"
            ]
          },
          "metadata": {},
          "execution_count": 94
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ThPaMDOGvsiW"
      },
      "source": [
        "#### 문장 토큰화"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "okKRCWhdxMbH"
      },
      "source": [
        "* 한국어 문장을 토큰화할 때는 kss(korean sentence splitter) 라이브러리 이용"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dOKstGpcxv5-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2e01cf82-b9b2-47d1-b2b8-6591153300b7"
      },
      "source": [
        "!pip install kss"
      ],
      "execution_count": 95,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting kss\n",
            "  Downloading kss-3.2.0.tar.gz (42.4 MB)\n",
            "\u001b[K     |████████████████████████████████| 42.4 MB 56 kB/s \n",
            "\u001b[?25hCollecting emoji\n",
            "  Downloading emoji-1.6.0.tar.gz (168 kB)\n",
            "\u001b[K     |████████████████████████████████| 168 kB 53.2 MB/s \n",
            "\u001b[?25hBuilding wheels for collected packages: kss, emoji\n",
            "  Building wheel for kss (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for kss: filename=kss-3.2.0-py3-none-any.whl size=42447995 sha256=9f1e3f415f4455a86f9a7babc6328a83525e0229efc177166a4cc382e31ae4aa\n",
            "  Stored in directory: /root/.cache/pip/wheels/a1/47/9c/a5f83b5ab6096e3c4a33643fc553b26098c23e72b6539b86f4\n",
            "  Building wheel for emoji (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for emoji: filename=emoji-1.6.0-py3-none-any.whl size=168256 sha256=9a98aa596fe2102965f42b03744e1172fb5b1bd45f76934e49df24e59ebc6ea7\n",
            "  Stored in directory: /root/.cache/pip/wheels/f7/d7/74/c720aaf345a042b0c2d74361873258c5e8649b7f11b2ccce49\n",
            "Successfully built kss emoji\n",
            "Installing collected packages: emoji, kss\n",
            "Successfully installed emoji-1.6.0 kss-3.2.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A1VGyGg5yN-d"
      },
      "source": [
        "* 라이브러리를 이용해도 한국어에는 전치 표현이 존재해 제대로 토큰화가 안됨\n",
        "* 좀 더 나은 학습을 위해 사용자는 해당 부분을 따로 처리해주어야만 함"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VIlTDBv7xMg7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1bd363c6-8c40-422b-fe28-16fd6b171762"
      },
      "source": [
        "import kss \n",
        "\n",
        "text = '진짜? 내일 뭐하지. 이렇게 애매모호한 문장도? 밥은 먹었어? 나는...'\n",
        "print(kss.split_sentences(text))"
      ],
      "execution_count": 97,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['진짜? 내일 뭐하지.', '이렇게 애매모호한 문장도? 밥은 먹었어?', '나는...']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FAq7fR4r7cpA"
      },
      "source": [
        "#### 정규 표현식을 이용한 토큰화"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jkCInLL67l9M"
      },
      "source": [
        "* 한국어도 정규 표현식을 이용해 토큰화 가능"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lyscjy5a7c4r",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "13c558f2-0cc8-4957-9df4-ca838fe2b649"
      },
      "source": [
        "from nltk.tokenize import RegexpTokenizer\n",
        "\n",
        "sentence = '안녕하세요 ㅋㅋ 저는 자연어 처리(Natural Language Processing)를ㄹ!1 배우고 있습니다.'\n",
        "\n",
        "tokenizer = RegexpTokenizer('[가-힣]+')\n",
        "tokens = tokenizer.tokenize(sentence)\n",
        "tokens"
      ],
      "execution_count": 98,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['안녕하세요', '저는', '자연어', '처리', '를', '배우고', '있습니다']"
            ]
          },
          "metadata": {},
          "execution_count": 98
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gmXhfXCC7c0u",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "86eab01e-e9be-4aee-daaa-81b7cceca85a"
      },
      "source": [
        "tokenizer = RegexpTokenizer(\"[ㄱ-ㅎ]+\", gaps=True)\n",
        "tokens = tokenizer.tokenize(sentence)\n",
        "tokens"
      ],
      "execution_count": 99,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['안녕하세요 ', ' 저는 자연어 처리(Natural Language Processing)를', '!1 배우고 있습니다.']"
            ]
          },
          "metadata": {},
          "execution_count": 99
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sZdfxeFsn25k"
      },
      "source": [
        "#### 케라스를 이용한 토큰화"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-_Zn15pNn25o",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4b849f38-960f-4ec2-9c72-a619836c8612"
      },
      "source": [
        "from keras.preprocessing.text import text_to_word_sequence\n",
        "\n",
        "sentence = '성공의 비결은 단 한 가지, 잘할 수 있는 일에 광적으로 집중하는 것이다.'\n",
        "\n",
        "text_to_word_sequence(sentence)"
      ],
      "execution_count": 100,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['성공의', '비결은', '단', '한', '가지', '잘할', '수', '있는', '일에', '광적으로', '집중하는', '것이다']"
            ]
          },
          "metadata": {},
          "execution_count": 100
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pzEFiZt7n25z"
      },
      "source": [
        "#### TextBlob을 이용한 토큰화"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tgzHxLPyn252",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d53c3fee-f4d9-4ebf-c6ef-4dc207937dbc"
      },
      "source": [
        "from textblob import TextBlob\n",
        "\n",
        "blob = TextBlob(sentence)\n",
        "blob.words"
      ],
      "execution_count": 102,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "WordList(['성공의', '비결은', '단', '한', '가지', '잘할', '수', '있는', '일에', '광적으로', '집중하는', '것이다'])"
            ]
          },
          "metadata": {},
          "execution_count": 102
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3xaQF79AaKiN"
      },
      "source": [
        "## Bag of Words(BoW)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1-FpBEoS3CBY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "059b654f-5889-4155-e864-e9e64c5bb20f"
      },
      "source": [
        "# 각 단어가 몇개 포함되어 있는지\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "corpus = ['Think like a man of action and act like man of thought.']\n",
        "\n",
        "vector = CountVectorizer()\n",
        "bow = vector.fit_transform(corpus)\n",
        "\n",
        "print(bow.toarray())\n",
        "print(vector.vocabulary_)"
      ],
      "execution_count": 106,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[1 1 1 2 2 2 1 1]]\n",
            "{'think': 6, 'like': 3, 'man': 4, 'of': 5, 'action': 1, 'and': 2, 'act': 0, 'thought': 7}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VIjY4WXr48FE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b8f2d85b-3540-4e58-c15d-df4100d8e101"
      },
      "source": [
        "vector = CountVectorizer(stop_words='english')\n",
        "bow = vector.fit_transform(corpus)\n",
        "\n",
        "print(bow.toarray())\n",
        "print(vector.vocabulary_)"
      ],
      "execution_count": 107,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[1 1 2 2 1 1]]\n",
            "{'think': 4, 'like': 2, 'man': 3, 'action': 1, 'act': 0, 'thought': 5}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OMeaVigC4eE6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "10c4b553-e9be-4a44-e72e-822153559c6c"
      },
      "source": [
        "corpus = ['평생 살 것처럼 꿈을 꾸어라. 그리고 내일 죽을 것처럼 오늘을 살아라.']\n",
        "\n",
        "vector = CountVectorizer() # 갯수세는 것\n",
        "bow = vector.fit_transform(corpus)\n",
        "\n",
        "print(bow.toarray())\n",
        "print(vector.vocabulary_)"
      ],
      "execution_count": 108,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[2 1 1 1 1 1 1 1 1]]\n",
            "{'평생': 8, '것처럼': 0, '꿈을': 3, '꾸어라': 2, '그리고': 1, '내일': 4, '죽을': 7, '오늘을': 6, '살아라': 5}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z2ud_G3LqIpw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f8176510-4809-4bd4-a82a-115c4b741ddb"
      },
      "source": [
        "import re\n",
        "from konlpy.tag import Mecab\n",
        "tagger = Mecab()\n",
        "\n",
        "corpus = '평생 살 것처럼 꿈을 꾸어라. 그리고 내일 죽을 것처럼 오늘을 살아라.'\n",
        "tokens = tagger.morphs(re.sub(\"(\\.)\",\"\", corpus))\n",
        "\n",
        "vocab = {}\n",
        "bow = []\n",
        "\n",
        "for tok in tokens:\n",
        "  if tok not in vocab.keys():\n",
        "    vocab[tok] = len(vocab)\n",
        "    bow.insert(len(vocab)-1, 1)\n",
        "  else:\n",
        "    index = vocab.get(tok) # index값을 get한다.\n",
        "    bow[index] = bow[index]+1\n",
        "\n",
        "print(bow)\n",
        "print(vocab)"
      ],
      "execution_count": 109,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[1, 2, 2, 2, 1, 3, 1, 1, 1, 1, 1, 1, 1]\n",
            "{'평생': 0, '살': 1, '것': 2, '처럼': 3, '꿈': 4, '을': 5, '꾸': 6, '어라': 7, '그리고': 8, '내일': 9, '죽': 10, '오늘': 11, '아라': 12}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5n2gkHk17NyC"
      },
      "source": [
        "## 문서 단어 행렬(DTM)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F87rTUZs7S2p"
      },
      "source": [
        "* 문서 단어 행렬(Document-Term Matrix)은 문서에 등장하는 여러 단어들의 빈도를 행렬로 표현\n",
        "* 각 문서에 대한 BoW를 하나의 행렬로 표현한 것"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c4pNYz6V7SO9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "31e052af-1cfa-4e1a-d9bc-0dd53b6d42ad"
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "corpus = ['Think like a man of action and act like man of thought.:',\n",
        "          'Try not to become a man of success but rather try to become a man of value.',\n",
        "          'Give me liberty, of give me death']\n",
        "\n",
        "vector = CountVectorizer(stop_words='english')\n",
        "bow = vector.fit_transform(corpus)\n",
        "\n",
        "print(bow.toarray())\n",
        "print(vector.vocabulary_)"
      ],
      "execution_count": 110,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[1 1 0 0 2 2 0 1 1 0 0]\n",
            " [0 0 0 0 0 2 1 0 0 2 1]\n",
            " [0 0 1 1 0 0 0 0 0 0 0]]\n",
            "{'think': 7, 'like': 4, 'man': 5, 'action': 1, 'act': 0, 'thought': 8, 'try': 9, 'success': 6, 'value': 10, 'liberty': 3, 'death': 2}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 142
        },
        "id": "CEsS4Rvp0AGm",
        "outputId": "48ba9430-3cd9-4182-f51b-de1af64bcddd"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "columns = []\n",
        "for k, v in sorted(vector.vocabulary_.items(), key=lambda item:item[1]): #값을 기준으로 정렬\n",
        "  columns.append(k)\n",
        "\n",
        "df = pd.DataFrame(bow.toarray(), columns=columns)\n",
        "df"
      ],
      "execution_count": 111,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>act</th>\n",
              "      <th>action</th>\n",
              "      <th>death</th>\n",
              "      <th>liberty</th>\n",
              "      <th>like</th>\n",
              "      <th>man</th>\n",
              "      <th>success</th>\n",
              "      <th>think</th>\n",
              "      <th>thought</th>\n",
              "      <th>try</th>\n",
              "      <th>value</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   act  action  death  liberty  like  man  success  think  thought  try  value\n",
              "0    1       1      0        0     2    2        0      1        1    0      0\n",
              "1    0       0      0        0     0    2        1      0        0    2      1\n",
              "2    0       0      1        1     0    0        0      0        0    0      0"
            ]
          },
          "metadata": {},
          "execution_count": 111
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_z-bPw94FTGb"
      },
      "source": [
        "## 어휘 빈도-문서 역빈도(TF-IDF) 분석"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k-0f-hz5Fgai"
      },
      "source": [
        "* 어휘 빈도-문서 역빈도(TF-IDF; Term Frequency-Inverse Docunment Frequency)는 단순히 빈도수가 높은 단어가 핵심어가 아닌, 특정 문서에서만 집중적으로 등장할 때 해당 단어가 문서의 주제를 잘 담고 있는 핵심어라고 가정"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z3kEwiZBJSSB"
      },
      "source": [
        "* 특정 문서에서 특정단어가 많이 등장하고 그 단어가 다른 문서에서 적게 등장할 때, 그 단어를 특정 문서의 핵심어로 간주\n",
        "* 어휘 빈도-문서 역빈도는 어휘 빈도와 역문서 빈도를 곱해 계산 가능"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s7wTnMY8JEDC"
      },
      "source": [
        "* **어휘 빈도**는 특정 문서에서 특정 단어가 많이 등장하는 것을 의미\n",
        "\n",
        "$$ tf_{x,y} $$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R8289-mSJF1p"
      },
      "source": [
        "* **역문서 빈도**는 다른 문서에서 등장하지 않는 단어 빈도를 의미\n",
        "\n",
        "$$ log(N/df_x) $$      "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3MyZYLgRJH8R"
      },
      "source": [
        "* **어휘 빈도-문서 역빈도**는 다음과 같이 표현\n",
        "\n",
        "$$ W_{x,y} = tf_{x,y} * log(N/df_x) $$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cv_23ugcKMjS"
      },
      "source": [
        "* tf-idf를 편리하게 계산하기 위해 `scikit-learn`의 `tfidfvectorizer`를 이용\n",
        "* 앞서 계산한 단어 빈도 수를 입력하여 tf-idf로 변환"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JH25kuy-J8cj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "eff944f0-16cd-40ad-c166-8fd8b8ec2422"
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "\n",
        "tfidf = TfidfVectorizer(stop_words='english').fit(corpus)\n",
        "\n",
        "print(tfidf.transform(corpus).toarray())\n",
        "print(tfidf.vocabulary_)"
      ],
      "execution_count": 112,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[0.311383   0.311383   0.         0.         0.62276601 0.4736296\n",
            "  0.         0.311383   0.311383   0.         0.        ]\n",
            " [0.         0.         0.         0.         0.         0.52753275\n",
            "  0.34682109 0.         0.         0.69364217 0.34682109]\n",
            " [0.         0.         0.70710678 0.70710678 0.         0.\n",
            "  0.         0.         0.         0.         0.        ]]\n",
            "{'think': 7, 'like': 4, 'man': 5, 'action': 1, 'act': 0, 'thought': 8, 'try': 9, 'success': 6, 'value': 10, 'liberty': 3, 'death': 2}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eoGurU81KqYD"
      },
      "source": [
        "* 좀 더 편리하게 확인하기 위해 데이터프레임으로 변환"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uSjYYeDBKGEr",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 162
        },
        "outputId": "8d01e152-1823-4d00-d63f-3e6e186b9cc8"
      },
      "source": [
        "columns = []\n",
        "for k, v in sorted(tfidf.vocabulary_.items(), key=lambda item:item[1]): #값을 기준으로 정렬\n",
        "  columns.append(k)\n",
        "\n",
        "pd.DataFrame(tfidf.transform(corpus).toarray(), columns=columns)\n"
      ],
      "execution_count": 114,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>act</th>\n",
              "      <th>action</th>\n",
              "      <th>death</th>\n",
              "      <th>liberty</th>\n",
              "      <th>like</th>\n",
              "      <th>man</th>\n",
              "      <th>success</th>\n",
              "      <th>think</th>\n",
              "      <th>thought</th>\n",
              "      <th>try</th>\n",
              "      <th>value</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.311383</td>\n",
              "      <td>0.311383</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.622766</td>\n",
              "      <td>0.473630</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.311383</td>\n",
              "      <td>0.311383</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.527533</td>\n",
              "      <td>0.346821</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.693642</td>\n",
              "      <td>0.346821</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.707107</td>\n",
              "      <td>0.707107</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        act    action     death  ...   thought       try     value\n",
              "0  0.311383  0.311383  0.000000  ...  0.311383  0.000000  0.000000\n",
              "1  0.000000  0.000000  0.000000  ...  0.000000  0.693642  0.346821\n",
              "2  0.000000  0.000000  0.707107  ...  0.000000  0.000000  0.000000\n",
              "\n",
              "[3 rows x 11 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 114
        }
      ]
    }
  ]
}