Pyspark_on_Colab.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JozQdM1rcSB6",
        "outputId": "9eaf41e5-a274-4577-c331-0d69499c4315"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting pyspark\n",
            "  Downloading pyspark-3.4.0.tar.gz (310.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m310.8/310.8 MB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: py4j==0.10.9.7 in /usr/local/lib/python3.10/dist-packages (from pyspark) (0.10.9.7)\n",
            "Building wheels for collected packages: pyspark\n",
            "  Building wheel for pyspark (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyspark: filename=pyspark-3.4.0-py2.py3-none-any.whl size=311317130 sha256=e199582aa5e03fd56b5fb9343f61a7114f042d15310effc6c1ee791d3144c37d\n",
            "  Stored in directory: /root/.cache/pip/wheels/7b/1b/4b/3363a1d04368e7ff0d408e57ff57966fcdf00583774e761327\n",
            "Successfully built pyspark\n",
            "Installing collected packages: pyspark\n",
            "Successfully installed pyspark-3.4.0\n"
          ]
        }
      ],
      "source": [
        "pip install pyspark\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from pyspark.sql import SparkSession"
      ],
      "metadata": {
        "id": "j0X_2G2ggD6K"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "spark = SparkSession.builder.master(\"local[*]\").getOrCreate()"
      ],
      "metadata": {
        "id": "J0NWIwd_gZDl"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "spark"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 219
        },
        "id": "03BmYUgvgzkR",
        "outputId": "8aa91282-7451-4cdc-91de-367cc4603851"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<pyspark.sql.session.SparkSession at 0x7f2f8136c1f0>"
            ],
            "text/html": [
              "\n",
              "            <div>\n",
              "                <p><b>SparkSession - in-memory</b></p>\n",
              "                \n",
              "        <div>\n",
              "            <p><b>SparkContext</b></p>\n",
              "\n",
              "            <p><a href=\"http://f69c64d31805:4040\">Spark UI</a></p>\n",
              "\n",
              "            <dl>\n",
              "              <dt>Version</dt>\n",
              "                <dd><code>v3.4.0</code></dd>\n",
              "              <dt>Master</dt>\n",
              "                <dd><code>local[*]</code></dd>\n",
              "              <dt>AppName</dt>\n",
              "                <dd><code>pyspark-shell</code></dd>\n",
              "            </dl>\n",
              "        </div>\n",
              "        \n",
              "            </div>\n",
              "        "
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from pyspark.sql.functions import col"
      ],
      "metadata": {
        "id": "PdJRHEoOg3Bc"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df = spark.read.csv(\"/content/drive/MyDrive/Spark/weatherAUS.csv\",header=True,inferSchema=True)\n"
      ],
      "metadata": {
        "id": "GwDxXwACEcGE"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sCLzfQQL2se_",
        "outputId": "47a7c344-983f-4787-853a-5f74e696c789"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "DataFrame[Date: string, Location: string, MinTemp: string, MaxTemp: string, Rainfall: string, WindGustDir: string, WindGustSpeed: string, WindDir9am: string, WindDir3pm: string, WindSpeed9am: string, WindSpeed3pm: string, Humidity9am: string, Humidity3pm: string, Pressure9am: string, Pressure3pm: string, Temp9am: string, Temp3pm: string, RainToday: string, RainTomorrow: string]"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.limit(5).show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Jh80wfEc4BZN",
        "outputId": "f42ce7b2-06fb-4dc5-fee3-bbb27cc845a5"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+---------+--------+-------+-------+--------+-----------+-------------+----------+----------+------------+------------+-----------+-----------+-----------+-----------+-------+-------+---------+------------+\n",
            "|     Date|Location|MinTemp|MaxTemp|Rainfall|WindGustDir|WindGustSpeed|WindDir9am|WindDir3pm|WindSpeed9am|WindSpeed3pm|Humidity9am|Humidity3pm|Pressure9am|Pressure3pm|Temp9am|Temp3pm|RainToday|RainTomorrow|\n",
            "+---------+--------+-------+-------+--------+-----------+-------------+----------+----------+------------+------------+-----------+-----------+-----------+-----------+-------+-------+---------+------------+\n",
            "|12/1/2008|  Albury|   13.4|   22.9|     0.6|          W|           44|         W|       WNW|          20|          24|         71|         22|     1007.7|     1007.1|   16.9|   21.8|       No|          No|\n",
            "|12/2/2008|  Albury|    7.4|   25.1|       0|        WNW|           44|       NNW|       WSW|           4|          22|         44|         25|     1010.6|     1007.8|   17.2|   24.3|       No|          No|\n",
            "|12/3/2008|  Albury|   12.9|   25.7|       0|        WSW|           46|         W|       WSW|          19|          26|         38|         30|     1007.6|     1008.7|     21|   23.2|       No|          No|\n",
            "|12/4/2008|  Albury|    9.2|     28|       0|         NE|           24|        SE|         E|          11|           9|         45|         16|     1017.6|     1012.8|   18.1|   26.5|       No|          No|\n",
            "|12/5/2008|  Albury|   17.5|   32.3|       1|          W|           41|       ENE|        NW|           7|          20|         82|         33|     1010.8|       1006|   17.8|   29.7|       No|          No|\n",
            "+---------+--------+-------+-------+--------+-----------+-------------+----------+----------+------------+------------+-----------+-----------+-----------+-----------+-------+-------+---------+------------+\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.select(\"*\").show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tgbaIBuN4pus",
        "outputId": "c14d44a4-fbd0-4bc1-e1d3-1ddcb9d4ab40"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------+--------+-------+-------+--------+-----------+-------------+----------+----------+------------+------------+-----------+-----------+-----------+-----------+-------+-------+---------+------------+\n",
            "|      Date|Location|MinTemp|MaxTemp|Rainfall|WindGustDir|WindGustSpeed|WindDir9am|WindDir3pm|WindSpeed9am|WindSpeed3pm|Humidity9am|Humidity3pm|Pressure9am|Pressure3pm|Temp9am|Temp3pm|RainToday|RainTomorrow|\n",
            "+----------+--------+-------+-------+--------+-----------+-------------+----------+----------+------------+------------+-----------+-----------+-----------+-----------+-------+-------+---------+------------+\n",
            "| 12/1/2008|  Albury|   13.4|   22.9|     0.6|          W|           44|         W|       WNW|          20|          24|         71|         22|     1007.7|     1007.1|   16.9|   21.8|       No|          No|\n",
            "| 12/2/2008|  Albury|    7.4|   25.1|       0|        WNW|           44|       NNW|       WSW|           4|          22|         44|         25|     1010.6|     1007.8|   17.2|   24.3|       No|          No|\n",
            "| 12/3/2008|  Albury|   12.9|   25.7|       0|        WSW|           46|         W|       WSW|          19|          26|         38|         30|     1007.6|     1008.7|     21|   23.2|       No|          No|\n",
            "| 12/4/2008|  Albury|    9.2|     28|       0|         NE|           24|        SE|         E|          11|           9|         45|         16|     1017.6|     1012.8|   18.1|   26.5|       No|          No|\n",
            "| 12/5/2008|  Albury|   17.5|   32.3|       1|          W|           41|       ENE|        NW|           7|          20|         82|         33|     1010.8|       1006|   17.8|   29.7|       No|          No|\n",
            "| 12/6/2008|  Albury|   14.6|   29.7|     0.2|        WNW|           56|         W|         W|          19|          24|         55|         23|     1009.2|     1005.4|   20.6|   28.9|       No|          No|\n",
            "| 12/7/2008|  Albury|   14.3|     25|       0|          W|           50|        SW|         W|          20|          24|         49|         19|     1009.6|     1008.2|   18.1|   24.6|       No|          No|\n",
            "| 12/8/2008|  Albury|    7.7|   26.7|       0|          W|           35|       SSE|         W|           6|          17|         48|         19|     1013.4|     1010.1|   16.3|   25.5|       No|          No|\n",
            "| 12/9/2008|  Albury|    9.7|   31.9|       0|        NNW|           80|        SE|        NW|           7|          28|         42|          9|     1008.9|     1003.6|   18.3|   30.2|       No|         Yes|\n",
            "|12/10/2008|  Albury|   13.1|   30.1|     1.4|          W|           28|         S|       SSE|          15|          11|         58|         27|       1007|     1005.7|   20.1|   28.2|      Yes|          No|\n",
            "|12/11/2008|  Albury|   13.4|   30.4|       0|          N|           30|       SSE|       ESE|          17|           6|         48|         22|     1011.8|     1008.7|   20.4|   28.8|       No|         Yes|\n",
            "|12/12/2008|  Albury|   15.9|   21.7|     2.2|        NNE|           31|        NE|       ENE|          15|          13|         89|         91|     1010.5|     1004.2|   15.9|     17|      Yes|         Yes|\n",
            "|12/13/2008|  Albury|   15.9|   18.6|    15.6|          W|           61|       NNW|       NNW|          28|          28|         76|         93|      994.3|        993|   17.4|   15.8|      Yes|         Yes|\n",
            "|12/14/2008|  Albury|   12.6|     21|     3.6|         SW|           44|         W|       SSW|          24|          20|         65|         43|     1001.2|     1001.8|   15.8|   19.8|      Yes|          No|\n",
            "|12/15/2008|  Albury|    8.4|   24.6|       0|         NA|           NA|         S|       WNW|           4|          30|         57|         32|     1009.7|     1008.7|   15.9|   23.5|       No|          NA|\n",
            "|12/16/2008|  Albury|    9.8|   27.7|      NA|        WNW|           50|        NA|       WNW|          NA|          22|         50|         28|     1013.4|     1010.3|   17.3|   26.2|       NA|          No|\n",
            "|12/17/2008|  Albury|   14.1|   20.9|       0|        ENE|           22|       SSW|         E|          11|           9|         69|         82|     1012.2|     1010.4|   17.2|   18.1|       No|         Yes|\n",
            "|12/18/2008|  Albury|   13.5|   22.9|    16.8|          W|           63|         N|       WNW|           6|          20|         80|         65|     1005.8|     1002.2|     18|   21.5|      Yes|         Yes|\n",
            "|12/19/2008|  Albury|   11.2|   22.5|    10.6|        SSE|           43|       WSW|        SW|          24|          17|         47|         32|     1009.4|     1009.7|   15.5|     21|      Yes|          No|\n",
            "|12/20/2008|  Albury|    9.8|   25.6|       0|        SSE|           26|        SE|       NNW|          17|           6|         45|         26|     1019.2|     1017.1|   15.8|   23.2|       No|          No|\n",
            "+----------+--------+-------+-------+--------+-----------+-------------+----------+----------+------------+------------+-----------+-----------+-----------+-----------+-------+-------+---------+------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.printSchema()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CZpFiLvg8kt0",
        "outputId": "c36a2230-6fe5-4320-a8b4-e3535dfa9f58"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "root\n",
            " |-- Date: string (nullable = true)\n",
            " |-- Location: string (nullable = true)\n",
            " |-- MinTemp: string (nullable = true)\n",
            " |-- MaxTemp: string (nullable = true)\n",
            " |-- Rainfall: string (nullable = true)\n",
            " |-- WindGustDir: string (nullable = true)\n",
            " |-- WindGustSpeed: string (nullable = true)\n",
            " |-- WindDir9am: string (nullable = true)\n",
            " |-- WindDir3pm: string (nullable = true)\n",
            " |-- WindSpeed9am: string (nullable = true)\n",
            " |-- WindSpeed3pm: string (nullable = true)\n",
            " |-- Humidity9am: string (nullable = true)\n",
            " |-- Humidity3pm: string (nullable = true)\n",
            " |-- Pressure9am: string (nullable = true)\n",
            " |-- Pressure3pm: string (nullable = true)\n",
            " |-- Temp9am: string (nullable = true)\n",
            " |-- Temp3pm: string (nullable = true)\n",
            " |-- RainToday: string (nullable = true)\n",
            " |-- RainTomorrow: string (nullable = true)\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.show(4,False)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yoZCU_vm-RXQ",
        "outputId": "c3f8e2ae-9fee-4ec6-8f7c-eb571bf98639"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+---------+--------+-------+-------+--------+-----------+-------------+----------+----------+------------+------------+-----------+-----------+-----------+-----------+-------+-------+---------+------------+\n",
            "|Date     |Location|MinTemp|MaxTemp|Rainfall|WindGustDir|WindGustSpeed|WindDir9am|WindDir3pm|WindSpeed9am|WindSpeed3pm|Humidity9am|Humidity3pm|Pressure9am|Pressure3pm|Temp9am|Temp3pm|RainToday|RainTomorrow|\n",
            "+---------+--------+-------+-------+--------+-----------+-------------+----------+----------+------------+------------+-----------+-----------+-----------+-----------+-------+-------+---------+------------+\n",
            "|12/1/2008|Albury  |13.4   |22.9   |0.6     |W          |44           |W         |WNW       |20          |24          |71         |22         |1007.7     |1007.1     |16.9   |21.8   |No       |No          |\n",
            "|12/2/2008|Albury  |7.4    |25.1   |0       |WNW        |44           |NNW       |WSW       |4           |22          |44         |25         |1010.6     |1007.8     |17.2   |24.3   |No       |No          |\n",
            "|12/3/2008|Albury  |12.9   |25.7   |0       |WSW        |46           |W         |WSW       |19          |26          |38         |30         |1007.6     |1008.7     |21     |23.2   |No       |No          |\n",
            "|12/4/2008|Albury  |9.2    |28     |0       |NE         |24           |SE        |E         |11          |9           |45         |16         |1017.6     |1012.8     |18.1   |26.5   |No       |No          |\n",
            "+---------+--------+-------+-------+--------+-----------+-------------+----------+----------+------------+------------+-----------+-----------+-----------+-----------+-------+-------+---------+------------+\n",
            "only showing top 4 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.count()\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GndHiV-6-Xvf",
        "outputId": "d1d2429d-abcd-4993-f627-42db4f1adec1"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "145460"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.select(\"Date\", \"Location\", \"Rainfall\").show(5)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZZRDu0_8IRis",
        "outputId": "8e9aea44-5f94-4be9-842c-a48d38a57d10"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+---------+--------+--------+\n",
            "|     Date|Location|Rainfall|\n",
            "+---------+--------+--------+\n",
            "|12/1/2008|  Albury|     0.6|\n",
            "|12/2/2008|  Albury|       0|\n",
            "|12/3/2008|  Albury|       0|\n",
            "|12/4/2008|  Albury|       0|\n",
            "|12/5/2008|  Albury|       1|\n",
            "+---------+--------+--------+\n",
            "only showing top 5 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.columns"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xd9AuEXQMO1y",
        "outputId": "09ff8caa-e9d0-45b0-ed14-8b2080d37bae"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Date',\n",
              " 'Location',\n",
              " 'MinTemp',\n",
              " 'MaxTemp',\n",
              " 'Rainfall',\n",
              " 'WindGustDir',\n",
              " 'WindGustSpeed',\n",
              " 'WindDir9am',\n",
              " 'WindDir3pm',\n",
              " 'WindSpeed9am',\n",
              " 'WindSpeed3pm',\n",
              " 'Humidity9am',\n",
              " 'Humidity3pm',\n",
              " 'Pressure9am',\n",
              " 'Pressure3pm',\n",
              " 'Temp9am',\n",
              " 'Temp3pm',\n",
              " 'RainToday',\n",
              " 'RainTomorrow']"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.select(\"Date\").show(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W4i5AhoVMOzN",
        "outputId": "1502f8aa-b8de-4a2f-b202-3a77737260d6"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------+\n",
            "|      Date|\n",
            "+----------+\n",
            "| 12/1/2008|\n",
            "| 12/2/2008|\n",
            "| 12/3/2008|\n",
            "| 12/4/2008|\n",
            "| 12/5/2008|\n",
            "| 12/6/2008|\n",
            "| 12/7/2008|\n",
            "| 12/8/2008|\n",
            "| 12/9/2008|\n",
            "|12/10/2008|\n",
            "+----------+\n",
            "only showing top 10 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.select(\"Location\",\"Date\").show(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PMo0YBAqMOv3",
        "outputId": "e3719690-0a7b-43b4-a371-28a6a0c41e8d"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------+----------+\n",
            "|Location|      Date|\n",
            "+--------+----------+\n",
            "|  Albury| 12/1/2008|\n",
            "|  Albury| 12/2/2008|\n",
            "|  Albury| 12/3/2008|\n",
            "|  Albury| 12/4/2008|\n",
            "|  Albury| 12/5/2008|\n",
            "|  Albury| 12/6/2008|\n",
            "|  Albury| 12/7/2008|\n",
            "|  Albury| 12/8/2008|\n",
            "|  Albury| 12/9/2008|\n",
            "|  Albury|12/10/2008|\n",
            "+--------+----------+\n",
            "only showing top 10 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.select(\"Location\",\"Date\").limit(5).show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vOPPd1QzFh7Z",
        "outputId": "f2cc97b2-62a9-4b19-b098-06b19cd8a849"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+--------+---------+\n",
            "|Location|     Date|\n",
            "+--------+---------+\n",
            "|  Albury|12/1/2008|\n",
            "|  Albury|12/2/2008|\n",
            "|  Albury|12/3/2008|\n",
            "|  Albury|12/4/2008|\n",
            "|  Albury|12/5/2008|\n",
            "+--------+---------+\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## How to filter out some data:\n",
        "\n",
        "We will do as we do in SQL we will use where"
      ],
      "metadata": {
        "id": "NuKXNpJHF01O"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.where((df.Rainfall > 25)& (df.Location==\"Albury\")).limit(5).show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EWkw97m6Fh25",
        "outputId": "5da5d643-32f9-42da-bbc5-ff693304ea96"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------+--------+-------+-------+--------+-----------+-------------+----------+----------+------------+------------+-----------+-----------+-----------+-----------+-------+-------+---------+------------+\n",
            "|      Date|Location|MinTemp|MaxTemp|Rainfall|WindGustDir|WindGustSpeed|WindDir9am|WindDir3pm|WindSpeed9am|WindSpeed3pm|Humidity9am|Humidity3pm|Pressure9am|Pressure3pm|Temp9am|Temp3pm|RainToday|RainTomorrow|\n",
            "+----------+--------+-------+-------+--------+-----------+-------------+----------+----------+------------+------------+-----------+-----------+-----------+-----------+-------+-------+---------+------------+\n",
            "| 9/23/2009|  Albury|    9.3|   16.8|    28.8|          W|           56|         W|       WSW|          26|          30|         82|         54|      997.8|     1003.8|   13.1|   15.9|      Yes|         Yes|\n",
            "|  2/5/2010|  Albury|   19.2|   26.1|    52.2|         SE|           33|         S|       SSW|           9|          13|         83|         70|     1009.9|     1009.4|   22.1|     25|      Yes|         Yes|\n",
            "|  3/8/2010|  Albury|   18.1|   25.5|      66|         NW|           56|         N|       NNW|          11|          37|         79|         38|     1008.7|     1007.5|   19.8|   24.1|      Yes|         Yes|\n",
            "| 5/29/2010|  Albury|    9.4|     15|      28|          N|           26|         E|        SE|           9|          11|         99|         89|     1005.7|     1002.8|   12.1|     14|      Yes|         Yes|\n",
            "|10/16/2010|  Albury|    4.8|   12.8|    32.8|          W|           52|       WSW|         W|          20|          28|         72|         61|     1004.6|     1008.3|    6.4|   11.2|      Yes|          No|\n",
            "+----------+--------+-------+-------+--------+-----------+-------------+----------+----------+------------+------------+-----------+-----------+-----------+-----------+-------+-------+---------+------------+\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## How to aggregate the data:\n"
      ],
      "metadata": {
        "id": "tALdMCZOGgz5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#df.groupBy(\"MinTemp\").agg({\"WindDir9am\":\"WSW\"})"
      ],
      "metadata": {
        "id": "QCDOtRV3FhqC"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## How to create a new temp view :\n",
        "\n",
        "If we want to create a new temp view and load the data frame on it"
      ],
      "metadata": {
        "id": "IwtLlHjAH0m9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.createOrReplaceTempView(\"WeatherAUS\")"
      ],
      "metadata": {
        "id": "DQ_ZkKZfFhmv"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## A good feature of Spark is that we can use SQL syntax by using Spark.sql"
      ],
      "metadata": {
        "id": "ryRnWQEZH7JR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "spark.sql(\"select * from WeatherAUS\").show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i4lKJAMkICxX",
        "outputId": "9ec5c781-1fd2-4da0-dc96-4e968618a145"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "+----------+--------+-------+-------+--------+-----------+-------------+----------+----------+------------+------------+-----------+-----------+-----------+-----------+-------+-------+---------+------------+\n",
            "|      Date|Location|MinTemp|MaxTemp|Rainfall|WindGustDir|WindGustSpeed|WindDir9am|WindDir3pm|WindSpeed9am|WindSpeed3pm|Humidity9am|Humidity3pm|Pressure9am|Pressure3pm|Temp9am|Temp3pm|RainToday|RainTomorrow|\n",
            "+----------+--------+-------+-------+--------+-----------+-------------+----------+----------+------------+------------+-----------+-----------+-----------+-----------+-------+-------+---------+------------+\n",
            "| 12/1/2008|  Albury|   13.4|   22.9|     0.6|          W|           44|         W|       WNW|          20|          24|         71|         22|     1007.7|     1007.1|   16.9|   21.8|       No|          No|\n",
            "| 12/2/2008|  Albury|    7.4|   25.1|       0|        WNW|           44|       NNW|       WSW|           4|          22|         44|         25|     1010.6|     1007.8|   17.2|   24.3|       No|          No|\n",
            "| 12/3/2008|  Albury|   12.9|   25.7|       0|        WSW|           46|         W|       WSW|          19|          26|         38|         30|     1007.6|     1008.7|     21|   23.2|       No|          No|\n",
            "| 12/4/2008|  Albury|    9.2|     28|       0|         NE|           24|        SE|         E|          11|           9|         45|         16|     1017.6|     1012.8|   18.1|   26.5|       No|          No|\n",
            "| 12/5/2008|  Albury|   17.5|   32.3|       1|          W|           41|       ENE|        NW|           7|          20|         82|         33|     1010.8|       1006|   17.8|   29.7|       No|          No|\n",
            "| 12/6/2008|  Albury|   14.6|   29.7|     0.2|        WNW|           56|         W|         W|          19|          24|         55|         23|     1009.2|     1005.4|   20.6|   28.9|       No|          No|\n",
            "| 12/7/2008|  Albury|   14.3|     25|       0|          W|           50|        SW|         W|          20|          24|         49|         19|     1009.6|     1008.2|   18.1|   24.6|       No|          No|\n",
            "| 12/8/2008|  Albury|    7.7|   26.7|       0|          W|           35|       SSE|         W|           6|          17|         48|         19|     1013.4|     1010.1|   16.3|   25.5|       No|          No|\n",
            "| 12/9/2008|  Albury|    9.7|   31.9|       0|        NNW|           80|        SE|        NW|           7|          28|         42|          9|     1008.9|     1003.6|   18.3|   30.2|       No|         Yes|\n",
            "|12/10/2008|  Albury|   13.1|   30.1|     1.4|          W|           28|         S|       SSE|          15|          11|         58|         27|       1007|     1005.7|   20.1|   28.2|      Yes|          No|\n",
            "|12/11/2008|  Albury|   13.4|   30.4|       0|          N|           30|       SSE|       ESE|          17|           6|         48|         22|     1011.8|     1008.7|   20.4|   28.8|       No|         Yes|\n",
            "|12/12/2008|  Albury|   15.9|   21.7|     2.2|        NNE|           31|        NE|       ENE|          15|          13|         89|         91|     1010.5|     1004.2|   15.9|     17|      Yes|         Yes|\n",
            "|12/13/2008|  Albury|   15.9|   18.6|    15.6|          W|           61|       NNW|       NNW|          28|          28|         76|         93|      994.3|        993|   17.4|   15.8|      Yes|         Yes|\n",
            "|12/14/2008|  Albury|   12.6|     21|     3.6|         SW|           44|         W|       SSW|          24|          20|         65|         43|     1001.2|     1001.8|   15.8|   19.8|      Yes|          No|\n",
            "|12/15/2008|  Albury|    8.4|   24.6|       0|         NA|           NA|         S|       WNW|           4|          30|         57|         32|     1009.7|     1008.7|   15.9|   23.5|       No|          NA|\n",
            "|12/16/2008|  Albury|    9.8|   27.7|      NA|        WNW|           50|        NA|       WNW|          NA|          22|         50|         28|     1013.4|     1010.3|   17.3|   26.2|       NA|          No|\n",
            "|12/17/2008|  Albury|   14.1|   20.9|       0|        ENE|           22|       SSW|         E|          11|           9|         69|         82|     1012.2|     1010.4|   17.2|   18.1|       No|         Yes|\n",
            "|12/18/2008|  Albury|   13.5|   22.9|    16.8|          W|           63|         N|       WNW|           6|          20|         80|         65|     1005.8|     1002.2|     18|   21.5|      Yes|         Yes|\n",
            "|12/19/2008|  Albury|   11.2|   22.5|    10.6|        SSE|           43|       WSW|        SW|          24|          17|         47|         32|     1009.4|     1009.7|   15.5|     21|      Yes|          No|\n",
            "|12/20/2008|  Albury|    9.8|   25.6|       0|        SSE|           26|        SE|       NNW|          17|           6|         45|         26|     1019.2|     1017.1|   15.8|   23.2|       No|          No|\n",
            "+----------+--------+-------+-------+--------+-----------+-------------+----------+----------+------------+------------+-----------+-----------+-----------+-----------+-------+-------+---------+------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1-  Use !wget to download the dataset to the server"
      ],
      "metadata": {
        "id": "KpGIYZg4-1AL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Colab is actually a Centos virtual machine with GPU. You can directly use the linux wget command to download the dataset to the server. The default is to download to the /content path\n",
        "\n",
        "Download and unzip the dataset command:"
      ],
      "metadata": {
        "id": "whslpZAP_hCn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#!wget https://download.pytorch.org/tutorial/hymenoptera_data.zip\n",
        "#!unzip hymenoptera_data.zip -d ./"
      ],
      "metadata": {
        "id": "Uy0pTv-z-0te"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Load dataset command:\n",
        "\n"
      ],
      "metadata": {
        "id": "_PkS_L31ARzm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the dataset using ImageFolder\n",
        "# define data preprocessing\n",
        "#train_tf = tfs.Compose([\n",
        " #  tfs.RandomResizedCrop(224),\n",
        " #  tfs.RandomHorizontalFlip(),\n",
        " #  tfs.ToTensor(),\n",
        " #  tfs.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) # Use ImageNet mean and variance\n",
        "#])\n",
        "#train_set = ImageFolder('./hymenoptera_data/train/', train_tf)"
      ],
      "metadata": {
        "id": "HGNPaqwGATP3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2- Loading data into PySpark from github\n",
        "Spark has a variety of modules to read data of different formats. It also automatically determines the data type for each column, but it has to go over it once.\n",
        "\n",
        "For this article, I have created a sample JSON dataset in Github. You can download the file directly into Colab using the ‘wget’ command like this:"
      ],
      "metadata": {
        "id": "9DsKr860AhrV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#!wget --continue https://raw.githubusercontent.com/GarvitArya/pyspark-\n",
        "#demo/main/sample_books.json -O /tmp/sample_books.json"
      ],
      "metadata": {
        "id": "V8MlCkAhAqB1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now read this file into a Spark dataframe using the read module."
      ],
      "metadata": {
        "id": "EGarK0mhA3Vz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#df = spark.read.json(\"/tmp/sample_books.json\")\n"
      ],
      "metadata": {
        "id": "1en2jBVmA4gj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 3- Use Google Cloud Disk to load datasets\n",
        "\n"
      ],
      "metadata": {
        "id": "CggfR44a_Hih"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "First, the command to mount Google Cloud Disk in Colab is as follows. After execution, you will be asked to enter the key of your Google account to mount"
      ],
      "metadata": {
        "id": "IYRrCoL4BPdO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#from google.colab import drive\n",
        "#drive.mount('/content/drive/')"
      ],
      "metadata": {
        "id": "E_rSlpk4-z6t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Upload the file to Google Drive, such as data/data.csv. One way to upload is to upload manually, the other is to download to Google Cloud Disk through the wget command, and then load it for use\n",
        "\n",
        "The advantage of storing in Google Cloud Disk is that the data will not be lost the next time you connect like the first method. The disadvantage is that The Google cloud disk is only 15g, which is not suitable for large data sets. The command to download the data set to the Google cloud disk is as follows:"
      ],
      "metadata": {
        "id": "9zhzZtYhBdox"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#import os\n",
        "#Change the current working directory to the path of Google Cloud Drive\n",
        "#path=\"/content/drive/My Drive/Colab Notebooks/\"\n",
        "#os.chdir(path)\n",
        "#os.listdir(path)\n",
        "#Use the wget command to download the dataset to this path\n",
        "#!wget https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/data.csv"
      ],
      "metadata": {
        "id": "s-Ztn2y4-z3s"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Load dataset\n",
        "\n"
      ],
      "metadata": {
        "id": "eLvumcueBzhk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#train = pd.read_csv('/content/drive/My Drive/Colab Notebooks/data/data.csv')\n"
      ],
      "metadata": {
        "id": "na1COwe8-z0a"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4. Load dataset from kaggle\n",
        "\n",
        "If you are playing a game on kaggle, the data set you need is prepared on it, and you can download it directly using the kaggle command. You need to choose to create an api token in the my profile of kaggle, and then generate the username and key locally"
      ],
      "metadata": {
        "id": "cF804uObB5EI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#{\"username\":\"gongenbo\",\"key\":\"f26dfa65d06321a37f6b8502cd6b8XXX\"}\n"
      ],
      "metadata": {
        "id": "hnNAI_-fB-io"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Command to download data via kaggle"
      ],
      "metadata": {
        "id": "Lic2Bum0CTXp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#!pip install -U -q kaggle\n",
        "#!mkdir -p ~/.kaggle\n",
        "#!echo '{\"username\":\"gongenbo\",\"key\":\"f26dfa65d06321a37f6b8502cd6b8XXX\"}' > ~/.kaggle/kaggle.json\n",
        "#!chmod 600 ~/.kaggle/kaggle.json\n",
        "#!kaggle competitions download -c state-farm-distracted-driver-detection"
      ],
      "metadata": {
        "id": "wAdHyocRCT7V"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Command to submit scores to kaggle after training\n",
        "\n"
      ],
      "metadata": {
        "id": "zkMvfTiYCbXB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#!kaggle competitions submit -c state-farm-distracted-driver-detection -f submission.csv -m \"Message\"\n"
      ],
      "metadata": {
        "id": "jzvfoirYCiC7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 5. Upload to disk using the upload button\n",
        "\n",
        "Google provides 67G of disk space. Use the upload button to upload the image below. This method is suitable for small datasets or own datasets:\n",
        "\n",
        "![1111.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAUUAAAEYCAYAAADLZOR0AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAACDESURBVHhe7d0PfBTVvffxH1K0LiDaohjFiIa2ohItqPyN1liJtfgHtVjFirzwiU+haBtU7lWEgrb3ARVFX3CvKF7BplpuLRbRAi1oCyi0QgtcwVpSNFUBpdUoiVWqPvM9mcHDsrvZJEvYzX7efU13Z3Zmd7Ky3/2dObNz2vTt2/dTAwA4bY477jhCEQBCVIoA4DkgvAUABAhFAPAQigDgIRQBwEMoAoCHUAQAD6EIAB5CEQA8hCIAeAhFAPAQigDgIRQBwEMoAoCHUAQAD6EIAB6upwigWbp37279+/cP55rm+eeft82bN4dz+xehCKBZFixYYO3btw/nmmbnzp120UUXhXMNUxCPGjXKJkyY4LZNpUOHDjZ58mSbOXNmWsFLKAJoslNOOcWmTZtmS5YssUWLFoVL9xaFWCoVFRW2bt26cC41vaZeWyE3duzYpMGoQLz77rvd6+u59RoN4ZgigGbbtm2bC51kU6abxqoQq6qqXNgp9BR+8fxA1LraJh2EIoCco8pQVV+yYIwPRK3bUDM70rZr164/DO8DQKMceeSRVlZWtrsiTKZNmzZWW1u7R/Xob6PnWbx4sW3fvt3Np+Ojjz6yZ5991s444wwXfqeffro999xzduCBBzY5ECUnjikWDxpmPTuHM3uos+rfzrfl75TYNdeX2lHbl9m9/73c6o4usSFnFVpsxwarXLI+XBdApkXHFOfOnWtz5swJlzbO8OHD7eqrr27UMUWfqkLtQ1FR0e5melMDUVKGonps9EI6kNmlSxf3BqRDaa/Uz5ShE2dbWWE4s4c6W//QGJteONFmD9IK1bZ45CSb1+8Gu//aYotVL7aRk+bVrwog47IhFMUPRmlqIErKY4qnnnqqK2sViLqvnU9nUjmdeVtt+d132V17TPfaPL2HTz1lyzZW26alT9nC+pUB5JlPP/2svvPvN1bKSlGpqzI0k1VfU9RXimEVGC7bUycrPPEoa1/3pm16tcYsSaXYqVuJlfQvDNbeZTs2LLbFG4J19xA8z5nBOl07me3aYZtWLLe1W+vCxwDEy5bms38MUaKmdKrTdZJJWSnqBfZ3IKanzEaMvdFuHJ6sQi2ysrHTbNpt19iQc0qt9JwyG/r9aXb/rUOCR0KxEht91zSbOHxI8HiwznlDbfQdwfzlu9cAkGUS9TJr0n0ti++VTkfKUNRZ6mo+Z4egirtymA3zp0HF4WOpFZeX29ATO1ndq8vskfFjbMz4GTb/5RqLHT/YysvrnyN2Qan1OsysbmOljR8z0irunW+b3tllscLe1sOtASCbJApEVYWamhOMKUPxwQcftMrKSleK6jihyuR0pobOXG+aTtbDVXne1O+E8LFUSm2Quq7rNtn82yttedAcrtu61hbeudg2fWjWuaivC726T+rXtgNidvhBZjUbFtpdN46xcXfOs03hQwCyQ7JAjDQnGFOGYmPb4vtWgo6WOWk07WMFdmhMtz1s2OzZNnv3NNR6BOFnnQusp9Z7epmtfiuoDE8YYjfcNdv+a/oUm/jdwdZD2wJIKTo7pSmTtm0snRmTLBAj8cGobdKRsqNFzeeOHTu6n/DsTw13tAy1ibPLrDDqWPE7Wu5sZ3fcX2oFW5fbXT9dHa7vq7U3N1ZbfZdLzAoGllpZr2Lr3q3ICjoFi2rW2qyKGZZoSwCZuSCETuy+8MILw7mGKeTy+oIQzQrFSa/ZDdPLrfigYPvbg+3fCDdR58tVve3dp+fZ6neC2cOKbfDFPa3mZ0ET23U4x2zIbffb4G7huZAvaBmAeNGvWppDHbr7u/iKNHhKjiZdAWN/al4ozrOiyyfaTYMKrV1dta3+3SrbVNPZ+pxdYj2OaGc16x6x8fctt75j77dhJ8Zs145NtnzpMqs+qJ+VfbOXFbTbassmjLfK3WEKoDVLGYpRWax2uU7e1rlE6dC5RtomU5obiqr6el15k404u9Bi3lHUmo3zbMbdi82d2RTrZcPGjrDSbt5BxE/qrOrpe+3HT9af+wSg9UsZiuPGjXMHQhVw0a9a0qEyOCvPb4wVWFG3Q+3A4G7t1k1WrWZznFhBkR17mFvDO9YIIF/kxDFFAGgpKU/JAYB8QygCgIdQBAAPoQgAHkIRADyEIgB4CEUA8BCKAOAhFAHAQygCgIdQBAAPoQgAHkIRADyEIgB4CEUA8BCKAOAhFAHAQygCgIdQBAAPoQgAnlY/cJVGI7z55pvdgN2paPTBqVOnhnMA8lXbrl27/jC83yrdc889DQaidO/e3a23cuXKcAmAfNTqK8WlS5e623POOcfdJhKtI/urYhwwYIBdffXVLpwTydZKNlf3OxX9LcXFxdahQwc3v3PnTlu/fr1t3rzZzaN1IxQDfijK/vgg//SnP7UuXbqEc4llY8Dk6n7HUytB4a6Qj8IwnsJRLYm5c+fatm3bwqVobQjFQHwoSkt/kBPtQ1Powzpz5swWOwyQq/vtGz58uAvExpgzZ44LR7Q+9D4nUVZW5jpock1U8WSjqqoqW7dunZu2b98eLq23P/ZbFeEDDzyw1+sqmCsqKtwXqSbd1z77FKTTpk1LWlUid1EpBlJVO6m2y6R09jMdmXqedDWmUrzuuut2H5dTCFZWVrr7vpbab4XZ3XffvdexULUO1EpIJFFFqb9n7NixrmmN1oFKMaAPYvyEzFJT89JLLw3n6pvLv/jFL8K5lqeAiw9EVYgKRAXmqFGjXGhrioJQTeb4ilHPoedC60EoYp9TU1mBoqpK541GtKy2tjacaznah0suuSSc+8wTTzzhbhVyCnBVs5o0P3r0aPeY9jmensv/u5Db8joUc/W4YSrZWOlOmTLF3Sp0Bg4c6O6LmpxRELWkZJVdVAXq30W8/v37u9v4SjFCtdh65G0oRoGoW79Zl0vUdFPTTgf8J0+e7CoWNf00rykbqEnqB4mazX7oqPKK73TJRm3atAnvobXLy1CMrxB1/ChRdZDNFOTqOVWFoiBUM09NPC1r6LzBlqTTbHzabwW532s7Y8aM8F7LSNQElqgJnKh6jTpfdB5jIsmeE7knrVAsKipylYh6GqNJ81qea+IDMRJVjblAAagg1yku6tEtLy+3CRMmuHCJAjIbxJ/krABXYGv/1PMbVbQtXamrck3UyRPthwJOj6uC1aS/Iwq9RPuqdZM1q5F7GgxFBZ9+P6xvSDWF9A9Et5rX8lw7wHzqqae6W//E7Oh+9Fi2i8JbQRid4qJlqhT9Cmx/809t0X5pio55Ksx1/l806W9pSQo5fan49G866oDRF8yVV17ppigQFerx/971HFSJrUvK8xTbt29vjz32mLv/gx/8YI9/RArL22+/3f1Dv+KKK/ZLL2I6Ep23p3/Y+mb3H4uWRRJtty815vWi44UKk4iqL7/ZrP8eLfFb3Wi/E/ErLAW27ic7n08tD4VSS73fon+7et34oNO/A+1r9O9BjycKRD2uMOccxdYlZaWonkIFo741479VNa/letzvUcwFfvhFEi3LJWqm6m+IppYIxIbo2KHCWh1CCo5k4aGwUSC2NO2PvlgU3j7tj754FPiadD8+ELWNtiUQW5+UlaK+HfUP+8ILL0xaCeofjQ5Mxx9QzxapKrCmPrYvpHNhBdEH8bzzzrNBgwbZsGHD9jhmJ9EHWv89WuJ0l4b2O9o/hUeyf0Nqdahq0xetjo/uDwpvHS/UYQh90Sei/dchAb2v8e87Wo+MhOKSJUt2n4uWbZoabi0diqqU9H431HmlKlCBp15mVYMTJ07c/QFVRXbTTTe520SBuS+ku98NUYeGWh46Xr2/6f3T3xN1WOl9VGBnQ/WNfS9lKEY9tcl+D6oPhI7JtFRV0hQLFixI+s3fEH1QdaA9G6mqUQ+0RE3/qImX6ve7AFJLu6NFzRq/8ohOq+jYsWNWd7QoKMaNG9foc/cUiKp+o8DJRvpvoC+m6Hic9nnRokVZvc9AtksZiqJQUS+zrFixwgWjPoxRJ4w+gIk6YgAgFzUYiqLjKzpuFFUkEoWhzlWU+FN2ACAXpRWKPgWkH36aJxgBtBaN/u1zfOhpXmEoCsfm9kICwP6UkQtCxAdjU3t7AWB/a3TzORVViTrHi9NBAOSqjIYiAOS6jDSfAaC1IBQBwEMoAoAn648p6qdrANBc6f7Ul44WAPDQfAYAD6EIAB5CEQA8hCIAeAhFAPAQigDgIRQBwEMoAoCHUAQAD6EIAB5CEQA8hCIAePLyghBlZWVuird582abOXNmOAcgH+VdKGoMmQceeCCc25vGl5k6dWo4ByDf5F0onnLKKTZt2rRwrvHWrVtnFRUV4RyA1oZjiqF0L2arUNUEoHUiFANLliyxK6+80t0CyG95H4oKwilTprj7uiUYgfyW16HoB2KEYATyW96Gok6/WbRokeuN9mley/U4gPyTt6Go8FMv9KhRo8Il9TSv5fFhmb4iK73qGhvSK1Y/G+tlQ0YMs9Lj62cBZDc6WjKuh/UdWGIlfU6on+12WjBfaiVfrZ8FkN0IxYxbaD/+vyOt4j/X1s9unGXjRo60SU/UzwLIbnkfikVFRa65HE2aB5C/8i4Ua2trw3v1OnTosPuEbE2ab0j8cwBoPfL2ghADBgxIKwB9O3fudL+NXrlyZbgEQGuTMhQVHOqN1U/g0vm977hx49w2M2bMcOEBALkmZfNZIVdVVZX2BRB04rOqqNGjR4dLACC3pAzF9u3bu1D0aZmOvfkdEmqOHnnkke7+tm3b3DoAkIsa1dGiILznnnvs9ttvt1mzZrnw0+3NN99sl1xySbgWAOSuRoWiqsby8nKbM2eOm1dA6tihvwwAclnSUEzVBI5+F6ze2CeeeMKFZfxpKjShAeSihKGo44PqLFHQrVixIlz6GT8U42l9badOmug4IwDkiqSVYnQOX6ITlYcPH+5uTz31VHcLAK1FwlBUD7JOr1ET2B/1TpWf5nUFGTWb9bh6ovXzuKi5PHDgQHdfo+LpeQAglyStFOMrxMmTJ1tlZaVrVt92221uACdRIOrcxPj1CcSWUVhYaMOGDbMxY8bY9773PRs6dKh17NgxfBRAY6X8RcvSpUtt7ty5rmdZFaGay+ptjgLv0ksv3f3Tt4ia1ldffbWdc8454ZLs06tXL/vqV/e+ltfWrVvtmWeeCeeyX5s2beyyyy5zhzpWr15tdXV1FovFbOPGjeEaABoraaUYT5WhwtGvANWEzrWf8xUUFLhzKo877ri9pv79++fU+ZZf+MIX3KGKt99+2wXhq6++SiACzZSyUlywYIFrGsePY5JK9PvnCy+8MFySGV/84hft73//ezjXdAq/kSNHhnONt2XLFps9e3Y4l54TTjjBBa4C7OOPP7bXXnvNli1bZh988IE7TnvmmWfaEUcc4dZ966237LnnnnO3ahoPGjTI/va3v7n7n//8511l/uyzz9onn3xi3/jGN+yggw5y23344Yf229/+1oL/nrZjxw5buHChqyR1DPj444+3Aw44wIWn9kHPrccB7C1lpagLOyjg9IuVdOj4oprZ2i7Tzj33XBeM+1tUVaarW7durvNJgfTQQw/Z8uXL7eijj3bhdfDBB7vDDAceeKA9+eSTbtIy/a26lbZt29pRRx3lglJfUnLaaadZdXW1/fznP7f33nvP/vrXv9oDDzzgXsOn19C+vvTSS/b444/bO++845rXAJJLGYpqGqvimzp1argkNV04QuMn76sm9Xe+852sCMbGUJWmqu73v/+9qww3bNhgmzZtss997nP2la98xVVuf/zjH+31119304svvuiW6THRttrmL3/5i6swo2rvsMMOc48noyrx2GOPdUH4u9/9zlWPa9asSXhuKYDPpH1MMRt07tw554JRldk///lP12SNqFr89a9/7TpI/vWvf7nAiijEFITReaKffvqpW8enwFNzOBUdb4ya23oOAOnJqVCUXAzGxlBzuaHAA7Dv8Onbx3SajCq2qCNFTj75ZHdKkKo4NaMV9JFDDjnE3Ta3matOKVWoOjapyhJAenIuFNXUfPTRRzPSE70vKITUOfXlL3/ZzasTRJXfGWec4R7r2bOn64k+9NBD7c9//rM76V2dUzpVqGvXru4cSi3TY82lY5CqqPV6Ct7evXs3eggGIN/kVChmeyCKwk6n4EQX4dW5g7pIxuGHH27XXnutC8xomTpedIK86ET4iy++2J1ao2V6rLlWrVrlXkuh++1vf9t1zmTieYHWLGcGrtKHWp0TzQ3E5p6nKDpPUecrZjs1m9U837Vrl5vv0qWLffOb37RXXnkl4dWPAORQpZiJQMw3ajbrFKkvfelLrumuJrw6ct54441wDQDxciYUMxWI6nxorkw8R0tYu3atO+Tw9a9/3TXdVSn+4Q9/yIkqF9hf8nLcZ3Vm9OjRw/UKN4bCUEGjk68BtE55GYoAkAznKQKAh1AEAA+hCAAeQhEAPIQiAHgIRQDwEIoA4CEUAcDT9sVv/eOH3z5mu2l6/PUu4WJkiq5Mo6FhdTmwffnzQF0wQ+PB6GIPmaL9Hjx4sPupYE1NTbg0M/bF/op+461fLOlq4++//364FEhfXlaK+tDoSjnx0/nnnx+ukTm6RqIu3aWLMmDf0/Uj9X6fdNJJ4ZLUNKjYdddd50ZLBCTvQrGlx33WoFMaaU8DV2Hf0+BfDz74oLuqEtAUbbb9ILb7t88Xv1Ac3ss+qgAycaUchZ+qwqZq7LjPaoL26dPHVq9ebf/4xz/cOM4akU/h3K5dOzdEqcZxVvP6ggsucANX/fKXv3TbaihTXS1b1z7UfusCthreQEOeanArXdVbIyeqqajmqMaU1pW1Nekaigpkjdudiq65eNZZZ7kL42qoVV2E9k9/+pO7mo72Xfugcaf1+tpf/Tf4zW9+s3sgrn79+rmrieviGtpW22l7STXetfZXQy5o/GkN1aAxrHWB3WeeecZV7P7f8tFHH7nn1HsoDY2VrefSsK5639T8T/a+XX755XsME6GLfWgbXQhYQ0ZoTG3tk55L7yMDgOWHnKkUc3Xc53i6nqFC4umnn7ZFixa5oQrUhFMo6tidjkFGf6c+/AoQhZLow60gqaysdMMV6JicrvYT0ZADOkb3k5/8xN588033wW6o2a7XVnhpaFVVtLpSt8aPibZTUGp/NOa0Ak0hpbAUradAVJg8/PDDbixqhajen1TjXft0DPDss8929/X8+pIQvQ8a+vWxxx6z7du3u9eMrgvZ0FjZ8ZK9bxoLW/uu4NPzKBC1XO+b3kdVnLrV/Iknnhg+G1q7nGo+t4ZR/FQBbdy40QWdxnLWh7Rjx452zDHHuCpUH3bd1wBWGqZU1U8UFO+++64bw1nVpLZV5eKP/6ymoyoaPa6qStViqgBXdaexoRU6CkWFg7bbunXr7oDRc6j6U8iqYtK+aL9UYWpsaoWetlXoqJpTJaaKLNV41xE9hwJOf6sCyR8GVvug59MXhcar1nPpfUlnrOx4Db1vPlWHei39vdpvvbb2XV9ayA85FYqtcXhTffgUlPpQKyjVQ60KUR00+oBGVaJo/Of4Hmx/pD6/eaeA0boK2WT0mgop/7CEQu+pp56y9evXu3k9p8IyXjSutILq+uuvd5Oao6oktU+pxruO6DnU3FVztqqqKlxaz/9bFHwKJT2nnr+hsbLjNfS++fR+67UU1mriq8NG1aQqaOSHnOtoaW3BqON00QdUgaQQ0XEuNTUVRn4oZqOXX37Z7rvvvj0mVX3pUtWsajYa6CsRfTmkGgs7k2Nl6wtCTXY15RWOOjxwxRVX0DudR3IuFFubTp06uVCMmmc6LqcQULPWbzqnw69+FKyq5NRJkYxeU1WU/wWjZrOO++m4WyraVs+tLyn/CubR/VTjXUcUQBpxUH+jjkUmOyao90gVr55TzfR9NVa26Njh6aef7ipDVcy/+tWvXOgSivkj50JRzaZsHuZUH2x/3Od4+oDpgxeN86zmmUIhqgh1qwpRIbBt2za3LF16Pr22jpepx1tVaKrxWNSkVI+wxm5RKCmM1TlSXFzstk1F2+q51YlRUlLittV2GihLf1Oq8a59eh4dt1OwaZ8jOi6p51OTOBpwS6+X6bGyoyBVU17hq0nPp0nvgV5DIZzqywWtS06FYrYHouhD74/7HE/HDxV6F110kQ0ZMsTNP//887uPeelXGKoQVRU1tums90dhfNVVV7kPszoq1LGQipq6agIrFMvLy90xQlVvDW0nWk/HHtWpohOgFXoKQzWJo7GtE413HU/HEzWpoyT6MomqxxEjRuz+WxSK6vzI5FjZeo/1nivY9YWgv0nvh0L8u9/9rgvHzZs3u84m5IecOU9RB71zfdxnNcF0nqJ6S/UhT0RNYHVYqILROXzNFZ23p0ouXnReXrbxz2EEWlrOVIr5MO6zjr+pqahTdJoSuonoGKXOP4zvDGlshwiQL1KG4hEHfWQnHbJzr6l924/DNVpOpgIx/tSMpsjEcyRSWlrqDvIryNQEBdDykjafFYizer0czu1pS93BNv5/j7faj9uGS3KLjhMx7jOARJKGorucWNft4dze3vrwwGBK3EO5pTZms18tCOcAIHc0eExxS+3n7aX3Ouw1vR2EYpsE/1PT+oKCt63PYZm9/h4AtIQGK8XxLxXZ/77XPlzasJMPqbU7TqpyF6x9/G9ctBZAbsm5k7cBYF8iFAHAQygCgIdQBAAPoQgAHkIRADwNhuJx7T9I+FO/ZJPWR5zLJ9rs6TfYZxfGApCtmvQzv3SUrz3B/eolG5WVlbkpni4RNXPmzHAugxSK/d+1WTdMt/rx6FLod4Pdf2132/zQGJv+QrhsX9F+DTJbPHKSzQsXAfku5aXDFIxHHLQrnEuffv6XrYHYvXt3d9WYZDT05dSpU8O5DCEUgZyRMhSzicYseeONN8K5ptMVm6dNmxbONd66deusoqIinEvi+CF2y/cGW1Gn4H7dVltf3c6Ku765OxRjvYbZTVeVWGGndmaf7LKaqsU24//Nt87X32/lp8TcU0jdulk25r7VSdePhnoquvgWG31ekelh21VjVb9+0O59YpPV6cFYiV3zb0Os5GjtjFnNXxbag/fNt543zbYy7wr71UtG2qSfBasPLLdbLu9jBdqNT+ps69r5dud/LjN+tIl8kTMdLbowrC49v78pVDUlV2zl1w22wrrl9sj4Chv/36usXcFn44mYDbbvl5dap+p5Nn7MSBtz93Kr6TbYRlxVYKvvG2MjH1ofhFmdrX8oeCwIxFTrOyeWW/kFhVazYrpVjKyw6StqrPD8/2Mj3C4W2bB/G2Z9P15j028caRX3zrc3CwbbqGv72LxJI23kkupgneqgUqwPRPdaV/WxWFVl8FpjbPzjmy122lAbffFnQQ20djnV+/yjH/0oK4IxpX5nW8/ONbbhfx6x5Vtrgkproc18cWv4YCC2xn5234/tzlnLbGtQytW9XGmbgocLjisJV4jT0PqdDrZYUMe99vz64P9rbP1P5tv8pWts8yfBY/0GW9+ja2zVQ5W2/p2gStyw0CrX7rBYt9OsV/3WcWLWrt0ue/PlVcFrBVXi0nk2/5nltmlHfZUJ5IOcCkUFYtYHY+GhFtu1w15bF84H6j72jssGzekdHfta+R3/5YY10OSascn+SzS0/gurbMNbnazk3++3KZNvDKrGTrb+yUpbvCF4TPtina1kcv12mu44M6ha237OEl/0bbmteXmX9fjW/TbtjlvshquKrWZZpc1f4YU60MrlVChKTgRjKkcPte+PUHN4vk0KmrQ6LLBYrdhkGlx/tc3696CZPmdxUNG1t6JB19gdU26xwUeHD++qsvnBNtpu95S0w2erLbxzTNDMnmerqj+yw3sNtRv+Y4qV9wsfBvJAzoVi1qt+1+radbZjvcOOsbZeXVZYYJ3bVduqexdbddCkdU3WVBcwb2D9goFDbNilvc1WLLRH7ptk48YttKp2RdZ7YPDgW7XBvhTYCWfXr+vEYsEzJNGzzIZdWWbHblhs82bdZeMrptvanZ2t5+mcYYn8kXOh+Prrr9utt97qbrPSC8/ahh2drOe3rrGSgk5WEFRb3+/nXYW85gOrswLrfW2JFXYrtJIRt1hJVNXJuh1WEzRuDz2qwAqC7Rtav+awHlZyfhCM5/cIwi5mBf2KrKDdLtv1fvDgs0uCfYlZjwtutMEnBFF4WLENHn2HTbl1cNCoDrwRBLh1ssNP02sFjx9wrBWfM8SGjugTLDXr1LOXFXQIWvB16Q/ID+S6tjf2a/fD8L67MGy2Ouuss9yg5M0NxCOPPDLhiduNoXMZt29PNlTDdlvz17ZWPOBcKz2/zEpPbm+vV39oXQ5539YsWm1vvL3GthxQbH3PLrWyr33Nvtxui238+xes4ODt9oelf7Kdu3ZY7CsDrG+/YNvjzRY8MTfl+u+8ssbe7NDbBgz6hg0Zcr6VnnSg7Vg5z+78n2rbpX1Zs9OOOb2vlZZdZBcN6mvH7nrFFj76qG1Q1Rk0w4/pd4ad/rVzrfehW2zxUwtty0dF9tWzzrfLLrvIys443D58ZbHNmrHcXJEK5IGcOU9xwoQJ9vDDDze7QmzueYqi8xR1viKA1idnms/qOc3aJjOAViNnQjETv2aR2tra8F7TZeI5AGSnnGk+Z5KOKQ4YMMA6dOgQLknPzp073fHElStXhksAtDZ5GYoAkEybvn377g5FAMh3nLwNAB5CEQA8hCIAeAhFAPAQigDgIRQBwEMoAoCHUAQAD6EIAB5CEQA8hCIAeAhFAPAQigDgIRQBwEMoAoCHUAQAD6EIAB5CEQA8hCIAeAhFAPAQigDgIRQBwEMoAoCHUAQAD6EIAB5CEQA8hCIAeAhFAPAQigDgIRQBwEMoAoCHUAQAD6EIAB5CEQA8hCIAeAhFAPAQigDgIRQBwEMoAoCHUAQAD6EIAB5CEQA8hCIAeAhFAPAQigDgIRQBwEMoAoCHUAQAD6EIAB5CEQA8hCIAeAhFAPAQigDgIRQBwEMoAoCHUAQAD6EIAB5CEQA8hCIAeAhFAPAQigDgIRQBwEMoAoCHUAQAD6EIAB5CEQA8hCIAeAhFAPAQigDgIRQBwEMoAoCHUAQAD6EIAB5CEQA8hCIAeAhFAPAQigDgIRQBwEMoAoCHUAQAD6EIAB5CEQA8hCIAeAhFAPAQigDgIRQBwEMoAoCHUAQAD6EIAB5CEQB2M/v/iR9vPQ6IdggAAAAASUVORK5CYII=)"
      ],
      "metadata": {
        "id": "mXp-1ikHClNk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 6-There is a library in jovian called open datasets.\n",
        "\n",
        "First, install it into colab using-\n",
        "The URL can be any link be it google or kaggle links.\n",
        "\n"
      ],
      "metadata": {
        "id": "O-FlUPlnDZB1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#!pip install opendatasets --upgrade\n",
        "#import opendatasets as od\n",
        "#dataset_url = 'https://www.kaggle.com/tunguz/us-elections-dataset'\n",
        "#od.download(dataset_url)"
      ],
      "metadata": {
        "id": "UEl4Uj0rC2m4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Reference:\n",
        "\n",
        "https://stackoverflow.com/questions/71619540/how-to-upload-a-62-gb-datasets-to-google-colab\n",
        "https://blog.devgenius.io/get-started-with-pyspark-and-google-colab-notebook-in-4-minutes-94b4e2cecad7"
      ],
      "metadata": {
        "id": "3irGze45DxV1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "If you have any question feel free to ask, stay tuned to next works\n",
        "\n",
        "\n",
        "**Author :Parissan Ahmadi**\n",
        "\n",
        "Linkdin : https://www.linkedin.com/in/parisan-ahmadi-1410a0a9/\n",
        "\n",
        "Github : https://github.com/parisa-ahmadi\n",
        "\n",
        "TelegramChannel : https://t.me/AIwithParissan\n",
        "\n",
        "Happy Learning!\n",
        "\n",
        "Stick To The Plan ✌"
      ],
      "metadata": {
        "id": "OhKrERZ3UxJW"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "NOXrPON6DxBe"
      },
      "execution_count": 33,
      "outputs": []
    }
  ]
}