wikicat_tutorial.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# 🐱Wikicat\n"
      ],
      "metadata": {
        "id": "47noiSfagMuo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Goal and Purpose\n",
        "Wikicat is a module that helps us extract all the pages of a given category and write them to a json file. You just need to pass a standard language code and also a valid category name and that is all!"
      ],
      "metadata": {
        "id": "gsPiaNOargZE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Prerequisites\n",
        "You may need to install tqdm and wikipediaapi libraries first.\n",
        "Use codes below to easily install via terminal:"
      ],
      "metadata": {
        "id": "oX3V6gLkiZar"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install Wikipedia-API\n",
        "!pip install tqdm"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aPO6FwuGazbe",
        "outputId": "d535149c-d537-4c59-ac8f-ddddd32d8576"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: Wikipedia-API in /usr/local/lib/python3.7/dist-packages (0.5.4)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from Wikipedia-API) (2.23.0)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->Wikipedia-API) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->Wikipedia-API) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->Wikipedia-API) (2022.6.15)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->Wikipedia-API) (1.24.3)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (4.64.1)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Libraries\n",
        "Now it's time to import our wikicat library as well as random and json libraries.\n",
        "They will help us randomly choose a json object and print it. \n"
      ],
      "metadata": {
        "id": "iQmVMu-8coDA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import wikicat\n",
        "import random\n",
        "import json"
      ],
      "metadata": {
        "id": "4c4_HkO_aJ_8"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Funcitons\n",
        "In this section we will explore each function of this package one by one:\n",
        "*   set_lang\n",
        "*   get_category_data\n",
        "*   create_category_file\n",
        "*   get_duplicate_elements"
      ],
      "metadata": {
        "id": "_zxpDon4jL2a"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## set_lang\n",
        "Let's start with ***set_lang*** function.\n",
        "\n",
        "This function is used to set the language of wikipedia.\n",
        "Check out [language editions of Wikipedia](https://en.wikipedia.org/wiki/List_of_Wikipedias) to find out what code you should use. \n",
        "\n",
        "It is also used in ***create_category_file*** function.\n",
        "\n",
        "This function gets:\n",
        "    \n",
        "*   *lang_code*: a standard language code (ex. en: English, fa: Farsi, etc.)\n",
        "\n",
        "And returns:\n",
        "\n",
        "*   wikipedia object of the language for further use.\n",
        "\n",
        "We use *'fa'* code which indicates *Farsi* language for instance. You can use any valid code."
      ],
      "metadata": {
        "id": "a_EezH6paKRE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "lang_code = 'fa'\n",
        "fa_wiki = wikicat.set_lang(lang_code)"
      ],
      "metadata": {
        "id": "JepmiFVNaNRm"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "*fa_wiki* can be used to get single page title, url, summary, full text, page sections, categories, etc.\n",
        "For more information check on [Wikipedia-API documentation](https://pypi.org/project/Wikipedia-API/).\n"
      ],
      "metadata": {
        "id": "Iq6d3lr8aOB-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## get_category_data\n",
        "this function gets:\n",
        "\n",
        "\n",
        "*   category_name: name of a valid category in a given language\n",
        "*   categorymembers: list of members of this category\n",
        "*   min_delay: (default value: 1s) minimum delay in seconds to wait in between sending requests to wikipedia\n",
        "*   max_delay: (default value: 5s) maximum delay in seconds to wait in between sending requests to wikipedia\n",
        "*  level: (default value: 0) level of the page being processed\n",
        "*  max_level: (default value: 20) maximum number of levels to be traversed\n",
        "\n",
        "And returns:\n",
        "*  list of json objects. It may contain duplicate elements. Each page has keys:\n",
        "    - title: title of the wikipedia page\n",
        "    - main category: the main category that we aim to extract its data\n",
        "    - all categories: all categories related to this page\n",
        "    - content: content of the page (usually it needs to get preprocessed)\n",
        "    - url: the url of the page\n",
        "\n",
        "A recurssive implementation has been applied. \n",
        "It traverses all subcategories (branch nodes) \n",
        "and their pages (leaf nodes) in a depth-first-search manner to get all data related to a given main category.\n",
        "\n",
        "We use \"نثر فارسی\" category to extract its data.\n",
        "As mentioned this function gets 6 arguments which the last 4 of them are optional. If you don't pass these args, the default values will be used.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "NP8hRb2adI3_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "category_name = \"نثر فارسی\"\n",
        "category = fa_wiki.page(f\"Category:{category_name}\")\n",
        "category_members = category.categorymembers\n",
        "min_delay = 1\n",
        "max_delay = 10\n",
        "level = 0\n",
        "max_level = 10\n",
        "category_data = wikicat.get_category_data(category_name, category_members, min_delay, max_delay, level, max_level)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R8Q3YmWZdJEd",
        "outputId": "6f079b08-89af-4273-b188-97bf1749ec78"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            " 88%|████████▊ | 15/17 [01:15<00:11,  5.66s/it]\n",
            "  0%|          | 0/8 [00:00<?, ?it/s]\u001b[A\n",
            " 12%|█▎        | 1/8 [00:06<00:43,  6.17s/it]\u001b[A\n",
            " 25%|██▌       | 2/8 [00:16<00:51,  8.53s/it]\u001b[A\n",
            " 38%|███▊      | 3/8 [00:21<00:34,  6.99s/it]\u001b[A\n",
            " 50%|█████     | 4/8 [00:29<00:29,  7.46s/it]\u001b[A\n",
            " 62%|██████▎   | 5/8 [00:33<00:18,  6.28s/it]\u001b[A\n",
            " 75%|███████▌  | 6/8 [00:36<00:09,  4.88s/it]\u001b[A\n",
            " 88%|████████▊ | 7/8 [00:38<00:04,  4.00s/it]\u001b[A\n",
            "100%|██████████| 8/8 [00:42<00:00,  5.30s/it]\n",
            " 94%|█████████▍| 16/17 [02:08<00:19, 19.78s/it]\n",
            "  0%|          | 0/49 [00:00<?, ?it/s]\u001b[A\n",
            "  2%|▏         | 1/49 [00:06<04:55,  6.16s/it]\u001b[A\n",
            "  4%|▍         | 2/49 [00:15<06:13,  7.95s/it]\u001b[A\n",
            "  6%|▌         | 3/49 [00:21<05:28,  7.14s/it]\u001b[A\n",
            "  8%|▊         | 4/49 [00:27<05:04,  6.77s/it]\u001b[A\n",
            " 10%|█         | 5/49 [00:32<04:32,  6.19s/it]\u001b[A\n",
            " 12%|█▏        | 6/49 [00:42<05:09,  7.21s/it]\u001b[A\n",
            " 14%|█▍        | 7/49 [00:43<03:39,  5.24s/it]\u001b[A\n",
            " 16%|█▋        | 8/49 [00:51<04:13,  6.17s/it]\u001b[A\n",
            " 18%|█▊        | 9/49 [00:52<03:04,  4.61s/it]\u001b[A\n",
            " 20%|██        | 10/49 [00:53<02:18,  3.55s/it]\u001b[A\n",
            " 22%|██▏       | 11/49 [00:59<02:45,  4.35s/it]\u001b[A\n",
            " 24%|██▍       | 12/49 [01:05<02:50,  4.60s/it]\u001b[A\n",
            " 27%|██▋       | 13/49 [01:13<03:25,  5.71s/it]\u001b[A\n",
            " 29%|██▊       | 14/49 [01:15<02:42,  4.64s/it]\u001b[A\n",
            " 31%|███       | 15/49 [01:19<02:32,  4.50s/it]\u001b[A\n",
            " 33%|███▎      | 16/49 [01:20<01:55,  3.50s/it]\u001b[A\n",
            " 35%|███▍      | 17/49 [01:27<02:17,  4.31s/it]\u001b[A\n",
            " 37%|███▋      | 18/49 [01:33<02:30,  4.87s/it]\u001b[A\n",
            " 39%|███▉      | 19/49 [01:36<02:10,  4.36s/it]\u001b[A\n",
            " 41%|████      | 20/49 [01:37<01:38,  3.40s/it]\u001b[A\n",
            " 43%|████▎     | 21/49 [01:38<01:16,  2.73s/it]\u001b[A\n",
            " 45%|████▍     | 22/49 [01:39<01:01,  2.27s/it]\u001b[A\n",
            " 47%|████▋     | 23/49 [01:48<01:45,  4.04s/it]\u001b[A\n",
            " 49%|████▉     | 24/49 [01:52<01:41,  4.08s/it]\u001b[A\n",
            " 51%|█████     | 25/49 [01:54<01:24,  3.51s/it]\u001b[A\n",
            " 53%|█████▎    | 26/49 [01:59<01:32,  4.02s/it]\u001b[A\n",
            " 55%|█████▌    | 27/49 [02:00<01:09,  3.16s/it]\u001b[A\n",
            " 57%|█████▋    | 28/49 [02:03<01:00,  2.86s/it]\u001b[A\n",
            " 59%|█████▉    | 29/49 [02:08<01:11,  3.56s/it]\u001b[A\n",
            " 61%|██████    | 30/49 [02:09<00:54,  2.85s/it]\u001b[A\n",
            " 63%|██████▎   | 31/49 [02:18<01:25,  4.75s/it]\u001b[A\n",
            " 65%|██████▌   | 32/49 [02:26<01:38,  5.78s/it]\u001b[A\n",
            " 67%|██████▋   | 33/49 [02:32<01:34,  5.90s/it]\u001b[A\n",
            " 69%|██████▉   | 34/49 [02:42<01:43,  6.92s/it]\u001b[A\n",
            " 71%|███████▏  | 35/49 [02:43<01:12,  5.20s/it]\u001b[A\n",
            " 73%|███████▎  | 36/49 [02:50<01:15,  5.80s/it]\u001b[A\n",
            " 76%|███████▌  | 37/49 [02:51<00:52,  4.41s/it]\u001b[A\n",
            " 78%|███████▊  | 38/49 [02:53<00:41,  3.74s/it]\u001b[A\n",
            " 80%|███████▉  | 39/49 [02:58<00:38,  3.88s/it]\u001b[A\n",
            " 82%|████████▏ | 40/49 [03:03<00:38,  4.27s/it]\u001b[A\n",
            " 84%|████████▎ | 41/49 [03:08<00:36,  4.57s/it]\u001b[A\n",
            " 86%|████████▌ | 42/49 [03:09<00:24,  3.55s/it]\u001b[A\n",
            " 88%|████████▊ | 43/49 [03:10<00:17,  2.84s/it]\u001b[A\n",
            " 90%|████████▉ | 44/49 [03:21<00:25,  5.04s/it]\u001b[A\n",
            " 92%|█████████▏| 45/49 [03:25<00:19,  4.78s/it]\u001b[A\n",
            "\n",
            "  0%|          | 0/4 [00:00<?, ?it/s]\u001b[A\u001b[A\n",
            "\n",
            " 25%|██▌       | 1/4 [00:10<00:30, 10.23s/it]\u001b[A\u001b[A\n",
            "\n",
            " 50%|█████     | 2/4 [00:20<00:20, 10.19s/it]\u001b[A\u001b[A\n",
            "\n",
            " 75%|███████▌  | 3/4 [00:29<00:09,  9.73s/it]\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "  0%|          | 0/3 [00:00<?, ?it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 33%|███▎      | 1/3 [00:06<00:12,  6.17s/it]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 67%|██████▋   | 2/3 [00:09<00:04,  4.40s/it]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "100%|██████████| 3/3 [00:12<00:00,  4.18s/it]\n",
            "\n",
            "\n",
            "100%|██████████| 4/4 [00:46<00:00, 11.57s/it]\n",
            "\n",
            " 94%|█████████▍| 46/49 [04:17<00:57, 19.09s/it]\u001b[A\n",
            "\n",
            "  0%|          | 0/3 [00:00<?, ?it/s]\u001b[A\u001b[A\n",
            "\n",
            " 33%|███▎      | 1/3 [00:07<00:14,  7.18s/it]\u001b[A\u001b[A\n",
            "\n",
            " 67%|██████▋   | 2/3 [00:09<00:04,  4.23s/it]\u001b[A\u001b[A\n",
            "\n",
            "100%|██████████| 3/3 [00:19<00:00,  6.53s/it]\n",
            "\n",
            " 96%|█████████▌| 47/49 [04:47<00:44, 22.29s/it]\u001b[A\n",
            "\n",
            "  0%|          | 0/16 [00:00<?, ?it/s]\u001b[A\u001b[A\n",
            "\n",
            "  6%|▋         | 1/16 [00:04<01:02,  4.16s/it]\u001b[A\u001b[A\n",
            "\n",
            " 12%|█▎        | 2/16 [00:11<01:23,  5.94s/it]\u001b[A\u001b[A\n",
            "\n",
            " 19%|█▉        | 3/16 [00:20<01:36,  7.42s/it]\u001b[A\u001b[A\n",
            "\n",
            " 25%|██▌       | 4/16 [00:24<01:13,  6.14s/it]\u001b[A\u001b[A\n",
            "\n",
            " 31%|███▏      | 5/16 [00:33<01:19,  7.23s/it]\u001b[A\u001b[A\n",
            "\n",
            " 38%|███▊      | 6/16 [00:43<01:18,  7.89s/it]\u001b[A\u001b[A\n",
            "\n",
            " 44%|████▍     | 7/16 [00:45<00:54,  6.03s/it]\u001b[A\u001b[A\n",
            "\n",
            " 50%|█████     | 8/16 [00:46<00:35,  4.48s/it]\u001b[A\u001b[A\n",
            "\n",
            " 56%|█████▋    | 9/16 [00:49<00:28,  4.07s/it]\u001b[A\u001b[A\n",
            "\n",
            " 62%|██████▎   | 10/16 [00:59<00:35,  5.95s/it]\u001b[A\u001b[A\n",
            "\n",
            " 69%|██████▉   | 11/16 [01:00<00:22,  4.50s/it]\u001b[A\u001b[A\n",
            "\n",
            " 75%|███████▌  | 12/16 [01:06<00:18,  4.70s/it]\u001b[A\u001b[A\n",
            "\n",
            " 81%|████████▏ | 13/16 [01:15<00:18,  6.06s/it]\u001b[A\u001b[A\n",
            "\n",
            " 88%|████████▊ | 14/16 [01:19<00:10,  5.49s/it]\u001b[A\u001b[A\n",
            "\n",
            " 94%|█████████▍| 15/16 [01:27<00:06,  6.30s/it]\u001b[A\u001b[A\n",
            "\n",
            "100%|██████████| 16/16 [01:31<00:00,  5.74s/it]\n",
            "\n",
            " 98%|█████████▊| 48/49 [06:29<00:46, 46.21s/it]\u001b[A\n",
            "\n",
            "  0%|          | 0/15 [00:00<?, ?it/s]\u001b[A\u001b[A\n",
            "\n",
            "  7%|▋         | 1/15 [00:09<02:08,  9.20s/it]\u001b[A\u001b[A\n",
            "\n",
            " 13%|█▎        | 2/15 [00:16<01:44,  8.01s/it]\u001b[A\u001b[A\n",
            "\n",
            " 20%|██        | 3/15 [00:18<01:04,  5.35s/it]\u001b[A\u001b[A\n",
            "\n",
            " 27%|██▋       | 4/15 [00:24<01:02,  5.68s/it]\u001b[A\u001b[A\n",
            "\n",
            " 33%|███▎      | 5/15 [00:30<00:58,  5.86s/it]\u001b[A\u001b[A\n",
            "\n",
            " 40%|████      | 6/15 [00:35<00:47,  5.29s/it]\u001b[A\u001b[A\n",
            "\n",
            " 47%|████▋     | 7/15 [00:40<00:42,  5.25s/it]\u001b[A\u001b[A\n",
            "\n",
            " 53%|█████▎    | 8/15 [00:46<00:38,  5.55s/it]\u001b[A\u001b[A\n",
            "\n",
            " 60%|██████    | 9/15 [00:50<00:30,  5.13s/it]\u001b[A\u001b[A\n",
            "\n",
            " 67%|██████▋   | 10/15 [00:53<00:22,  4.53s/it]\u001b[A\u001b[A\n",
            "\n",
            " 73%|███████▎  | 11/15 [01:00<00:20,  5.04s/it]\u001b[A\u001b[A\n",
            "\n",
            " 80%|████████  | 12/15 [01:02<00:12,  4.17s/it]\u001b[A\u001b[A\n",
            "\n",
            " 87%|████████▋ | 13/15 [01:05<00:07,  3.87s/it]\u001b[A\u001b[A\n",
            "\n",
            " 93%|█████████▎| 14/15 [01:10<00:04,  4.27s/it]\u001b[A\u001b[A\n",
            "\n",
            "100%|██████████| 15/15 [01:13<00:00,  4.92s/it]\n",
            "\n",
            "100%|██████████| 49/49 [07:51<00:00,  9.62s/it]\n",
            "100%|██████████| 17/17 [10:00<00:00, 35.35s/it] \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here is a random page selected from category_data:\n"
      ],
      "metadata": {
        "id": "Nm9ZURftdYh2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sample = random.choice(category_data)\n",
        "parsed = json.loads(sample)\n",
        "print(json.dumps(parsed, indent = 4,ensure_ascii=False))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c0_5yC8-dYyM",
        "outputId": "7bc51d2b-68eb-414e-846e-b8144425326f"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{\n",
            "    \"title\": \"منشآت قائم مقام فراهانی\",\n",
            "    \"main category\": \"نثر فارسی\",\n",
            "    \"all categories\": [\n",
            "        \"رده:صفحه‌های دارای ارجاع با پارامتر پشتیبانی‌نشده\",\n",
            "        \"رده:مقاله‌های خرد ادبیات\",\n",
            "        \"رده:نثر فارسی\",\n",
            "        \"رده:همه مقاله‌های خرد\",\n",
            "        \"رده:کتاب‌های ادبی\",\n",
            "        \"رده:کتاب‌های سده ۱۳\"\n",
            "    ],\n",
            "    \"content\": \"منشآت قائم مقام فراهانی عنوان کتابی است مشتمل بر برخی نوشته‌های قائم مقام فراهانی که به دستور و تدبیر شاگرد او، حاج فرهاد میرزا معتمدالدوله قاجار در سال ۱۲۸۰ گردآوری و در سال ۱۲۹۴ برای نخستین بار چاپ شد. این اثر دارای نثری زیبا و ادبی است و در دوره‌ای منتشر شد که نهضت بازگشت ادبی ظهور کرده بود و نویسندگان و شاعران شیوه اطناب و مصنوع‌نویسی را که از عهد صفوی متداول شده بود، کنار می‌گذاشتند.نوشته‌هایی که در این کتاب آمده، مجموعه‌ای از متن‌ها و نامه‌نگاری‌هایی است که قائم مقام به واسطه شغل دیوانی خود انشاء کرده‌بود. اغلب این انشاها به نثر ساده و مرسل و به زبانی نزدیک به زبان محاوره نوشته شده‌اند و دستمایه‌های طنز نیز در آن‌ها به چشم می‌خورد.منشآت، از جمله آثاری است که به عقیده کارشناسان ادبیات فارسی با الهام گرفتن از گلستان سعدی نوشته شده‌است. \\n\\n\\n== پانویس ==\",\n",
            "    \"url\": \"https://fa.wikipedia.org/wiki/%D9%85%D9%86%D8%B4%D8%A2%D8%AA_%D9%82%D8%A7%D8%A6%D9%85_%D9%85%D9%82%D8%A7%D9%85_%D9%81%D8%B1%D8%A7%D9%87%D8%A7%D9%86%DB%8C\"\n",
            "}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As you see, there is a trade-off between speed and not getting errors caused by abundance of requests. You can set both min and max delay equal to zero to speed up the code, but it is more probable to get some errors."
      ],
      "metadata": {
        "id": "aJj1JBRTdm14"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "## create_category_file\n",
        "This function wrapps the previous ones up into one function and is the most important part of wikicat.The main purpose of this function is to create a json file containing all data of a given category under the name of the category.\n",
        "\n",
        "Like **get_category_data** function this function gets:\n",
        "*   category_name: name of a valid category in a given language\n",
        "*   categorymembers: list of members of this category\n",
        "*   min_delay: (default value: 1s) minimum delay in seconds to wait in between sending requests to wikipedia\n",
        "*   max_delay: (default value: 5s) maximum delay in seconds to wait in between sending requests to wikipedia\n",
        "*  level: (default value: 0) level of the page being processed\n",
        "*  max_level: (default value: 20) maximum number of levels to be traversed\n",
        "\n",
        "then it creates the file and returns:\n",
        "*  list of deduplicated json objects where all its elements are unique.\n",
        "*  list of all json objects including repetitive ones.\n",
        "\n",
        "We continue our \"نثر فارسی\" example. The code below writes all unique data to a file named نثرفارسی.json \n",
        "All the arguements passed, are initialized in the previous part.\n",
        "\n"
      ],
      "metadata": {
        "id": "a8ksk-wZd0Bd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "deduplicated_data, data = wikicat.create_category_file(lang_code, category_name, min_delay, max_delay, max_level)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-jPYy97yeAYY",
        "outputId": "6e3d3031-d6f3-4b49-f1d5-1e8a8bbd37f0"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            " 88%|████████▊ | 15/17 [01:06<00:08,  4.19s/it]\n",
            "  0%|          | 0/8 [00:00<?, ?it/s]\u001b[A\n",
            " 12%|█▎        | 1/8 [00:07<00:50,  7.17s/it]\u001b[A\n",
            " 25%|██▌       | 2/8 [00:15<00:46,  7.76s/it]\u001b[A\n",
            " 38%|███▊      | 3/8 [00:17<00:26,  5.21s/it]\u001b[A\n",
            " 50%|█████     | 4/8 [00:19<00:16,  4.01s/it]\u001b[A\n",
            " 62%|██████▎   | 5/8 [00:29<00:18,  6.23s/it]\u001b[A\n",
            " 75%|███████▌  | 6/8 [00:35<00:11,  5.87s/it]\u001b[A\n",
            " 88%|████████▊ | 7/8 [00:45<00:07,  7.28s/it]\u001b[A\n",
            "100%|██████████| 8/8 [00:47<00:00,  5.93s/it]\n",
            " 94%|█████████▍| 16/17 [01:55<00:17, 17.55s/it]\n",
            "  0%|          | 0/49 [00:00<?, ?it/s]\u001b[A\n",
            "  2%|▏         | 1/49 [00:07<05:44,  7.17s/it]\u001b[A\n",
            "  4%|▍         | 2/49 [00:09<03:19,  4.23s/it]\u001b[A\n",
            "  6%|▌         | 3/49 [00:17<04:37,  6.03s/it]\u001b[A\n",
            "  8%|▊         | 4/49 [00:27<05:45,  7.68s/it]\u001b[A\n",
            " 10%|█         | 5/49 [00:33<05:13,  7.13s/it]\u001b[A\n",
            " 12%|█▏        | 6/49 [00:39<04:38,  6.47s/it]\u001b[A\n",
            " 14%|█▍        | 7/49 [00:44<04:13,  6.04s/it]\u001b[A\n",
            " 16%|█▋        | 8/49 [00:51<04:22,  6.41s/it]\u001b[A\n",
            " 18%|█▊        | 9/49 [01:00<04:50,  7.27s/it]\u001b[A\n",
            " 20%|██        | 10/49 [01:07<04:42,  7.25s/it]\u001b[A\n",
            " 22%|██▏       | 11/49 [01:08<03:24,  5.39s/it]\u001b[A\n",
            " 24%|██▍       | 12/49 [01:18<04:02,  6.55s/it]\u001b[A\n",
            " 27%|██▋       | 13/49 [01:27<04:24,  7.35s/it]\u001b[A\n",
            " 29%|██▊       | 14/49 [01:31<03:43,  6.39s/it]\u001b[A\n",
            " 31%|███       | 15/49 [01:38<03:45,  6.63s/it]\u001b[A\n",
            " 33%|███▎      | 16/49 [01:42<03:14,  5.89s/it]\u001b[A\n",
            " 35%|███▍      | 17/49 [01:47<02:51,  5.37s/it]\u001b[A\n",
            " 37%|███▋      | 18/49 [01:51<02:35,  5.02s/it]\u001b[A\n",
            " 39%|███▉      | 19/49 [02:00<03:08,  6.27s/it]\u001b[A\n",
            " 41%|████      | 20/49 [02:04<02:43,  5.64s/it]\u001b[A\n",
            " 43%|████▎     | 21/49 [02:12<02:59,  6.40s/it]\u001b[A\n",
            " 45%|████▍     | 22/49 [02:19<02:59,  6.64s/it]\u001b[A\n",
            " 47%|████▋     | 23/49 [02:24<02:33,  5.90s/it]\u001b[A\n",
            " 49%|████▉     | 24/49 [02:31<02:37,  6.29s/it]\u001b[A\n",
            " 51%|█████     | 25/49 [02:40<02:51,  7.16s/it]\u001b[A\n",
            " 53%|█████▎    | 26/49 [02:44<02:24,  6.27s/it]\u001b[A\n",
            " 55%|█████▌    | 27/49 [02:51<02:23,  6.54s/it]\u001b[A\n",
            " 57%|█████▋    | 28/49 [02:56<02:02,  5.83s/it]\u001b[A\n",
            " 59%|█████▉    | 29/49 [02:58<01:34,  4.74s/it]\u001b[A\n",
            " 61%|██████    | 30/49 [03:00<01:15,  3.97s/it]\u001b[A\n",
            " 63%|██████▎   | 31/49 [03:07<01:28,  4.94s/it]\u001b[A\n",
            " 65%|██████▌   | 32/49 [03:12<01:25,  5.02s/it]\u001b[A\n",
            " 67%|██████▋   | 33/49 [03:15<01:06,  4.16s/it]\u001b[A\n",
            " 69%|██████▉   | 34/49 [03:24<01:25,  5.68s/it]\u001b[A\n",
            " 71%|███████▏  | 35/49 [03:31<01:25,  6.13s/it]\u001b[A\n",
            " 73%|███████▎  | 36/49 [03:36<01:16,  5.85s/it]\u001b[A\n",
            " 76%|███████▌  | 37/49 [03:43<01:15,  6.25s/it]\u001b[A\n",
            " 78%|███████▊  | 38/49 [03:53<01:21,  7.43s/it]\u001b[A\n",
            " 80%|███████▉  | 39/49 [03:58<01:04,  6.45s/it]\u001b[A\n",
            " 82%|████████▏ | 40/49 [04:07<01:05,  7.27s/it]\u001b[A\n",
            " 84%|████████▎ | 41/49 [04:13<00:55,  6.95s/it]\u001b[A\n",
            " 86%|████████▌ | 42/49 [04:20<00:49,  7.02s/it]\u001b[A\n",
            " 88%|████████▊ | 43/49 [04:29<00:46,  7.67s/it]\u001b[A\n",
            " 90%|████████▉ | 44/49 [04:32<00:30,  6.02s/it]\u001b[A\n",
            " 92%|█████████▏| 45/49 [04:41<00:27,  6.97s/it]\u001b[A\n",
            "\n",
            "  0%|          | 0/4 [00:00<?, ?it/s]\u001b[A\u001b[A\n",
            "\n",
            " 25%|██▌       | 1/4 [00:05<00:15,  5.21s/it]\u001b[A\u001b[A\n",
            "\n",
            " 50%|█████     | 2/4 [00:15<00:16,  8.13s/it]\u001b[A\u001b[A\n",
            "\n",
            " 75%|███████▌  | 3/4 [00:16<00:04,  4.95s/it]\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "  0%|          | 0/3 [00:00<?, ?it/s]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 33%|███▎      | 1/3 [00:03<00:06,  3.18s/it]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            " 67%|██████▋   | 2/3 [00:08<00:04,  4.35s/it]\u001b[A\u001b[A\u001b[A\n",
            "\n",
            "\n",
            "100%|██████████| 3/3 [00:11<00:00,  3.85s/it]\n",
            "\n",
            "\n",
            "100%|██████████| 4/4 [00:34<00:00,  8.57s/it]\n",
            "\n",
            " 94%|█████████▍| 46/49 [05:24<00:53, 17.92s/it]\u001b[A\n",
            "\n",
            "  0%|          | 0/3 [00:00<?, ?it/s]\u001b[A\u001b[A\n",
            "\n",
            " 33%|███▎      | 1/3 [00:05<00:10,  5.18s/it]\u001b[A\u001b[A\n",
            "\n",
            " 67%|██████▋   | 2/3 [00:11<00:05,  5.77s/it]\u001b[A\u001b[A\n",
            "\n",
            "100%|██████████| 3/3 [00:18<00:00,  6.20s/it]\n",
            "\n",
            " 96%|█████████▌| 47/49 [05:53<00:42, 21.17s/it]\u001b[A\n",
            "\n",
            "  0%|          | 0/16 [00:00<?, ?it/s]\u001b[A\u001b[A\n",
            "\n",
            "  6%|▋         | 1/16 [00:05<01:17,  5.17s/it]\u001b[A\u001b[A\n",
            "\n",
            " 12%|█▎        | 2/16 [00:14<01:45,  7.53s/it]\u001b[A\u001b[A\n",
            "\n",
            " 19%|█▉        | 3/16 [00:23<01:47,  8.28s/it]\u001b[A\u001b[A\n",
            "\n",
            " 25%|██▌       | 4/16 [00:25<01:10,  5.87s/it]\u001b[A\u001b[A\n",
            "\n",
            " 31%|███▏      | 5/16 [00:32<01:09,  6.34s/it]\u001b[A\u001b[A\n",
            "\n",
            " 38%|███▊      | 6/16 [00:43<01:16,  7.65s/it]\u001b[A\u001b[A\n",
            "\n",
            " 44%|████▍     | 7/16 [00:53<01:16,  8.48s/it]\u001b[A\u001b[A\n",
            "\n",
            " 50%|█████     | 8/16 [00:54<00:49,  6.16s/it]\u001b[A\u001b[A\n",
            "\n",
            " 56%|█████▋    | 9/16 [00:56<00:34,  4.91s/it]\u001b[A\u001b[A\n",
            "\n",
            " 62%|██████▎   | 10/16 [01:05<00:37,  6.23s/it]\u001b[A\u001b[A\n",
            "\n",
            " 69%|██████▉   | 11/16 [01:16<00:37,  7.46s/it]\u001b[A\u001b[A\n",
            "\n",
            " 75%|███████▌  | 12/16 [01:22<00:28,  7.07s/it]\u001b[A\u001b[A\n",
            "\n",
            " 81%|████████▏ | 13/16 [01:29<00:21,  7.10s/it]\u001b[A\u001b[A\n",
            "\n",
            " 88%|████████▊ | 14/16 [01:36<00:14,  7.13s/it]\u001b[A\u001b[A\n",
            "\n",
            " 94%|█████████▍| 15/16 [01:43<00:07,  7.14s/it]\u001b[A\u001b[A\n",
            "\n",
            "100%|██████████| 16/16 [01:52<00:00,  7.06s/it]\n",
            "\n",
            " 98%|█████████▊| 48/49 [07:53<00:50, 50.85s/it]\u001b[A\n",
            "\n",
            "  0%|          | 0/15 [00:00<?, ?it/s]\u001b[A\u001b[A\n",
            "\n",
            "  7%|▋         | 1/15 [00:06<01:26,  6.20s/it]\u001b[A\u001b[A\n",
            "\n",
            " 13%|█▎        | 2/15 [00:15<01:43,  7.96s/it]\u001b[A\u001b[A\n",
            "\n",
            " 20%|██        | 3/15 [00:25<01:47,  8.98s/it]\u001b[A\u001b[A\n",
            "\n",
            " 27%|██▋       | 4/15 [00:27<01:09,  6.30s/it]\u001b[A\u001b[A\n",
            "\n",
            " 33%|███▎      | 5/15 [00:30<00:51,  5.17s/it]\u001b[A\u001b[A\n",
            "\n",
            " 40%|████      | 6/15 [00:40<00:58,  6.53s/it]\u001b[A\u001b[A\n",
            "\n",
            " 47%|████▋     | 7/15 [00:48<00:56,  7.07s/it]\u001b[A\u001b[A\n",
            "\n",
            " 53%|█████▎    | 8/15 [00:49<00:36,  5.20s/it]\u001b[A\u001b[A\n",
            "\n",
            " 60%|██████    | 9/15 [00:53<00:29,  4.89s/it]\u001b[A\u001b[A\n",
            "\n",
            " 67%|██████▋   | 10/15 [00:57<00:23,  4.67s/it]\u001b[A\u001b[A\n",
            "\n",
            " 73%|███████▎  | 11/15 [01:08<00:25,  6.36s/it]\u001b[A\u001b[A\n",
            "\n",
            " 80%|████████  | 12/15 [01:09<00:14,  4.79s/it]\u001b[A\u001b[A\n",
            "\n",
            " 87%|████████▋ | 13/15 [01:18<00:12,  6.12s/it]\u001b[A\u001b[A\n",
            "\n",
            " 93%|█████████▎| 14/15 [01:25<00:06,  6.44s/it]\u001b[A\u001b[A\n",
            "\n",
            "100%|██████████| 15/15 [01:26<00:00,  5.79s/it]\n",
            "\n",
            "100%|██████████| 49/49 [09:29<00:00, 11.62s/it]\n",
            "100%|██████████| 17/17 [11:29<00:00, 40.58s/it] \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here is a random page selected from deduplicated_data:"
      ],
      "metadata": {
        "id": "4CY8zm39eEv2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sample = random.choice(list(deduplicated_data))\n",
        "parsed = json.loads(sample)\n",
        "print(json.dumps(parsed, indent = 4, ensure_ascii=False))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hHFtgkQveHsG",
        "outputId": "0e514436-6921-4abc-eea5-17043cbea35c"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{\n",
            "    \"title\": \"بدایع‌الوقایع\",\n",
            "    \"main category\": \"نثر فارسی\",\n",
            "    \"all categories\": [\n",
            "        \"رده:ادبیات\",\n",
            "        \"رده:ادبیات ایران\",\n",
            "        \"رده:ادبیات فارسی\",\n",
            "        \"رده:مقاله‌های خرد ادبیات\",\n",
            "        \"رده:مقاله‌هایی که تجمیع ارجاع در آن‌ها ممنوع است\",\n",
            "        \"رده:نثر فارسی\",\n",
            "        \"رده:همه مقاله‌های خرد\"\n",
            "    ],\n",
            "    \"content\": \"بدایع‌الوقایع یا وديعةالحقايق اثر زین‌الدین واصفی هروی شاعر روزگار تیموری و صفوی است. این شرح رویدادهایی از گذشته و مطالبی از دیده‌ها و شنیده‌های خود او را شامل می‌شود. واصفی هروی بخشی از قصیده‌های خود در مدح ازبکان را در این کتاب آورده‌است. این کتاب در سال ۱۹۶۱ م در دو جلد در مسکو به‌چاپ رسید. در این کتاب برخی وقایع روزگار سلطان حسین بایقرا و فرزندانش، رویدادهای روزگار شیبک‌خان و برخی وقایع دیگر آمده‌است.\\n\\nپانویس\\nمنابع\\nصفا، ذبیح‌الله (۱۳۷۰). تاریخ ادبیات در ایران. ج. پنجم. تهران: فردوس.\",\n",
            "    \"url\": \"https://fa.wikipedia.org/wiki/%D8%A8%D8%AF%D8%A7%DB%8C%D8%B9%E2%80%8C%D8%A7%D9%84%D9%88%D9%82%D8%A7%DB%8C%D8%B9\"\n",
            "}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## get_duplicate_elements\n",
        "Our last function gets a list and returns duplicate elements.\n",
        "This function gets:\n",
        "*  cat_data: any list (in this case pages of a category )\n",
        "\n",
        "and  returns:\n",
        "*  duplicate elements of the list (in this case, duplicate pages)\n",
        "\n",
        "As we said, the second output create_category_file function may contain duplicate values. The code below extracts these repetitive pages.\n"
      ],
      "metadata": {
        "id": "pYbHW98oeI_k"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "duplicate_pages = wikicat.get_duplicate_elements(data)"
      ],
      "metadata": {
        "id": "f27ZVrf_eSIs"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here is a random page selected from duplicate_pages:"
      ],
      "metadata": {
        "id": "2xPdREzreVxC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sample = random.choice(duplicate_pages)\n",
        "parsed = json.loads(sample)\n",
        "print(json.dumps(parsed, indent = 4,ensure_ascii=False))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yIBFFaDZeX42",
        "outputId": "ae8d3ee1-e58a-486f-86af-5ad6996effa6"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{\n",
            "    \"title\": \"تذکرةالاولیاء\",\n",
            "    \"main category\": \"نثر فارسی\",\n",
            "    \"all categories\": [\n",
            "        \"رده:ادبیات تصوف\",\n",
            "        \"رده:ادبیات فارسی\",\n",
            "        \"رده:الگوهای درگاه با درگاه‌های ناموجود\",\n",
            "        \"رده:تذکره‌های فارسی\",\n",
            "        \"رده:عطار نیشابوری\",\n",
            "        \"رده:فرهنگ‌های اعلام\",\n",
            "        \"رده:نثر فارسی دوره تکوین\",\n",
            "        \"رده:پیوندهای وی‌بک الگوی بایگانی اینترنت\",\n",
            "        \"رده:کتاب‌های عرفانی\"\n",
            "    ],\n",
            "    \"content\": \"تذکرةالاولیاء کتابی عرفانی است به نثر ساده و در قسمت‌هایی مسجع، که در شرح احوال بزرگان اولیاء و مشایخ صوفیه توسط فریدالدین عطار نیشابوری به فارسی نوشته شده‌است.\\n\\nساختار و درون‌مایهٔ کتاب\\nاین کتاب مشتمل است بر مقدمه و ۷۲ باب که هریک به زندگی، حالات، اندیشه‌ها و سخنان یکی از عارفان و مشایخِ تصوف می‌پردازد، و ذکر مکارم اخلاق، مواعظ و سخنان حکمت‌آمیزشان در این کتاب آورده شده‌است.\\nنخستین باب به حالات و سخنان جعفر صادق اختصاص یافته‌است، و باب هفتاد و دوم به حسین بن منصور حلاج (متوفی ۳۰۹ هجری قمری). به نظر محمد استعلامی، دست‌نویس‌های معتبر تذکرةالاولیاء تا قرن دهم هجری، شامل همین ۷۲ باب است، امّا پس از قرن دهم فصل‌های دیگری با عنوان «ذکر متأخران از مشایخِ کِبار» بر این کتاب افزوده شده که حالات و سخنان ۲۵ تن از عارفان قرن‌های چهارم و پنجم را دربردارد.در این کتاب از کسانی چون اویس قرنی، بایزید بسطامی، ابراهیم ادهم، رابعه عدویه، ابوحنیفه، شافعی، احمد حنبل و محمد باقر یاد شده‌است.\\nعطار در دیباچهٔ کتاب نحوهٔ گزینش و تألیف مطالب را به‌روشنی بیان کرده‌است؛ بنابراین مقدمه، نقل‌ها و روایت‌های تذکرةالاولیاء التقاطی از مطالب کتاب‌های دیگر است. از آثار صوفیانهٔ مهم و مشهور مربوط به اواخر قرن چهارم تا پایان قرن ششم، چند اثر با احتمال زیاد ممکن است مأخذ نقل‌های مؤلف تذکرةالاولیاء باشند، که عبارت‌اند از: طبقات الصوفیه اثر ابوعبدالرحمن محمد سلمی نیشابوری، حلیةالاولیا ابونعیم اصفهانی، و رسالهٔ قشیریه نوشتهٔ ابوالقاسم قشیری. کشف‌المحجوب هُجویری نیز احتمالاً از منابع عطار بوده، زیرا عطار برخی از مضامین آن را با عبارات روان‌تر و رساتر آورده‌است.\\n\\nوضعیت انتشار\\nتصحیح انتقادیِ تذکرةالاولیاء اولین بار توسط رنالد نیکلسن انجام گرفته‌است (چاپ لیدن ۱۹۰۵–۱۹۰۷). از امتیازات تصحیح نیکلسن، یکی بی‌طرفی مذهبی او (از نظر شیعه و سنی)، و دیگر مقدمهٔ محمد قزوینی بر آن است که حاوی اطلاعات دقیق و سودمندی است. دومین تصحیح جدّی تذکرةالاولیاء، تصحیح محمد استعلامی است که تا سال ۱۳۹۷، ۲۸ بار چاپ شده‌است.\\nمنقح‌ترین چاپ تذکرةالاولیاء مجموعهٔ دو جلدی تصحیح «تذکرةالاولیاء» به تصحیح و توضیح محمدرضا شفیعی کدکنی است که انتشارات سخن منتشر کرده‌است.\\n\\nپانوشت‌ها\\nمنابع\\nفریدالدین عطار نیشابوری (بهار ۱۳۸۳)، تذکرةالاولیاء، به کوشش محمد استعلامی.، تهران: زوار، شابک ۹۶۴-۴۰۱-۰۳۴-۵\\nصفحات اولیهٔ تذکرةالاولیاء، تصحیح محمد استعلامی، انتشارات زَوّار، چاپ بیست‌وسوم، ۱۳۹۱. سایت خانهٔ کتاب 5-034-401-964 ISBN\\nشفیعی کدکنی و تصحیح آثار عطّار نیشابوری بانک اطلاعات نشریات کشور|روزنامهٔ ایران\\nتذکرةالاولیاء، تصحیح محمد استعلامی، انتشارات زَوّار، چاپ بیست‌وپنجم، سایت خانهٔ کتاب\\n\\nپیوند به بیرون\\n\\nتذکرة الاولیاء در ویکی‌نبشته\\nمتن تذکرةالأولیاء|وبگاه گنجور\\nتذکرةالأولیاء، بخش اول، چاپ لیدن\\nتذکرةالأولیاء، بخش دوم، چاپ لیدن\",\n",
            "    \"url\": \"https://fa.wikipedia.org/wiki/%D8%AA%D8%B0%DA%A9%D8%B1%D8%A9%D8%A7%D9%84%D8%A7%D9%88%D9%84%DB%8C%D8%A7%D8%A1\"\n",
            "}\n"
          ]
        }
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}