1. Transliteration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3ddfef4a",
   "metadata": {},
   "source": [
    "# Transliteration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "425c23d7",
   "metadata": {},
   "source": [
    "### 1. What?\n",
    "\n",
    "Conversion of text from one **script** to another by replacing letters, such that the sound of words remains the same\n",
    "\n",
    "### 2. Example\n",
    "\n",
    "**\"mera naam ankur hai\"** =====> **\"मेरा नाम अंकुर है\"**\n",
    "\n",
    "### 3. How is it useful?\n",
    "\n",
    "1. Helps pronounce names from different languages since names are not translated\n",
    "\n",
    "2. Helps in search with English keyboards\n",
    "\n",
    "3. Can help someone in learning a foreign language sounds using native script"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca0f9f2a",
   "metadata": {},
   "source": [
    "## Libraries\n",
    "\n",
    "1. [Indic Transliteration](https://github.com/indic-transliteration/indic_transliteration_py)\n",
    "2. [AI4Bharat Transliteration](https://github.com/AI4Bharat/IndianNLP-Transliteration)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbf7d58c",
   "metadata": {},
   "source": [
    "## 1. Indic Transliteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f97970da",
   "metadata": {},
   "outputs": [],
   "source": [
    "from indic_transliteration import sanscript\n",
    "from indic_transliteration.sanscript import transliterate, SCHEMES"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb6c1bee",
   "metadata": {},
   "source": [
    "The package supports the following scripts:\n",
    "\n",
    "- Bengali\n",
    "- Devanagari\n",
    "- Gujarati\n",
    "- Kannada\n",
    "- Malayalam\n",
    "- Telugu\n",
    "- Tamil\n",
    "- Oriya\n",
    "- Gurmukhi/ Punjabi/ Panjabi\n",
    "\n",
    "and the following romanizations:\n",
    "\n",
    "- HK = ‘hk’\n",
    "- IAST = ‘iast’\n",
    "- ITRANS = ‘itrans’\n",
    "- OPTITRANS = ‘optitrans’\n",
    "- KOLKATA = ‘kolkata’\n",
    "- SLP1 = ‘slp1’\n",
    "- VELTHUIS = ‘velthuis’\n",
    "- WX = ‘wx’\n",
    "\n",
    "Reference: https://indic-transliteration.github.io/indic_transliteration_py/build/html/indic_transliteration_sanscript.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "939a9751",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'मेरा नाम अन्कुर है'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transliterate('merA nAma ankura hai', sanscript.ITRANS, sanscript.DEVANAGARI)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2a05b1ce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "45.9 µs ± 4.14 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "transliterate('merA nAma ankur hai', sanscript.ITRANS, sanscript.DEVANAGARI)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41ec4b77",
   "metadata": {},
   "source": [
    "### Pros\n",
    "\n",
    "1. Faster conversion\n",
    "2. Supports english to indic scripts (9) and vice-versa\n",
    "3. Supports indic to indic conversion\n",
    "\n",
    "### Cons\n",
    "\n",
    "1. Rule based\n",
    "2. Does not handle running typing style. Input needs to follow script rules"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8027d8b",
   "metadata": {},
   "source": [
    "## 2. AI4Bharat Transliteration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1f94f1a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ai4bharat.transliteration import XlitEngine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1210575a",
   "metadata": {},
   "source": [
    "Supported Languages\n",
    "\n",
    "- Bengali - বাংলা\n",
    "- Gujarati - ગુજરાતી\n",
    "- Hindi - हिंदी\n",
    "- Kannada - ಕನ್ನಡ\n",
    "- Konkani Goan - कोंकणी\n",
    "- Maithili - मैथिली\n",
    "- Malayalam - മലയാളം\n",
    "- Marathi - मराठी\n",
    "- Panjabi Eastern - ਪੰਜਾਬੀ\n",
    "- Sindhi - سنڌي‎\n",
    "- Sinhala - සිංහල\n",
    "- Telugu - తెలుగు\n",
    "- Tamil - தமிழ்\n",
    "- Urdu - اُردُو"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0354e051",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading hi...\n"
     ]
    }
   ],
   "source": [
    "e = XlitEngine(\"hi\")\n",
    "out = e.translit_word(\"mera naam ankur hai\", topk=5, beam_width=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "bf33975f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'hi': ['मेरा नाम अंकुर है']}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5a0acf7c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "993 ms ± 17.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "out = e.translit_word(\"mera naam ankur hai\", topk=5, beam_width=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "634d7c78",
   "metadata": {},
   "source": [
    "### Pros\n",
    "\n",
    "1. Can handle running input style\n",
    "2. Suppots english to indic scripts (14)\n",
    "3. HTTP API for web application\n",
    "\n",
    "### Cons\n",
    "\n",
    "1. Slower than rule based library\n",
    "2. Does not support indic to English or indic to indic conversion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "592903ec",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "sans",
   "language": "python",
   "name": "sans"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}