Collecting Data Using APIs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Collecting Job Data Using APIs**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Instructions\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run the actual lab, firstly you need to click on the [Jobs_API](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/Jobs_API.ipynb) notebook link. The file contains flask code which is required to run the Jobs API data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset Used in this Assignment\n",
    "\n",
    "The dataset used in this lab comes from the following source: https://www.kaggle.com/promptcloud/jobs-on-naukricom under the under a **Public Domain license**.\n",
    "\n",
    "> Note: Using a modified subset of that dataset.\n",
    "\n",
    "The original dataset is a csv. I have converted the csv to json as per the requirement.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests # need this module to make an API call\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api_url = \"http://api.open-notify.org/astros.json\" # this url gives use the astronaut data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(api_url) # Call the API using the get method and store the\n",
    "                                # output of the API call in a variable called response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if response.ok:             # if all is well() no errors, no network timeouts)\n",
    "    data = response.json()  # store the result in json format in a variable called data\n",
    "                            # the variable data is of type dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data)   # print the data just to check the output or for debugging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of astronauts currently on ISS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data.get('number'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The names of the astronauts currently on ISS.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "astronauts = data.get('people')\n",
    "print(\"There are {} astronauts on ISS\".format(len(astronauts)))\n",
    "print(\"And their names are :\")\n",
    "for astronaut in astronauts:\n",
    "    print(astronaut.get('name'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hope the warmup was helpful. Good luck with your next lab!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collecting Jobs Data using Jobs API\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Objective: Determining the number of jobs currently open for various technologies  and for various locations\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Collecting the number of job postings for the following locations using the API:\n",
    "\n",
    "* Los Angeles\n",
    "* New York\n",
    "* San Francisco\n",
    "* Washington DC\n",
    "* Seattle\n",
    "* Austin\n",
    "* Detroit\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Required libraries\n",
    "import pandas as pd\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api_url=\"http://127.0.0.1:5000/data/all\"\n",
    "def get_number_of_jobs_T(technology):\n",
    "    \n",
    "    response_api = requests.get(api_url)\n",
    "\n",
    "    number_of_jobs = 0\n",
    "\n",
    "    if response_api.ok:            \n",
    "        jobs = response_api.json()\n",
    "\n",
    "    for job in jobs:\n",
    "        key = job.get('Key Skills')\n",
    "\n",
    "        if key.find(technology) > -1 :\n",
    "            number_of_jobs = number_of_jobs + 1\n",
    "\n",
    "    number_of_jobs\n",
    "    \n",
    "    return technology,number_of_jobs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling the function for Python and checking if it works.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_number_of_jobs_T(\"Python\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Function to find number of jobs in US for a location of your choice\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_number_of_jobs_L(location):\n",
    "    \n",
    "    response_api = requests.get(api_url)\n",
    "\n",
    "    number_of_jobs = 0\n",
    "\n",
    "    if response_api.ok:            \n",
    "        jobs = response_api.json()\n",
    "\n",
    "    for job in jobs:\n",
    "        loc = job.get('Location')\n",
    "\n",
    "        if loc.find(location) > -1 :\n",
    "            number_of_jobs = number_of_jobs + 1\n",
    "\n",
    "    number_of_jobs\n",
    "    return location,number_of_jobs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling function for Los Angeles and check if it is working.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_number_of_jobs_L(\"Los Angeles\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Storing the results in an excel file\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling the API for all the given technologies above and write the results in an excel spreadsheet.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "locations = ['Los Angeles','New York','San Francisco','Washington DC','Seattle','Austin','Detroit']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "libraries required to create excel spreadsheet\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openpyxl import Workbook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a workbook and select the active worksheet\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wb1 = Workbook()\n",
    "ws1 = wb1.active"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finding the number of jobs postings for each of the location in the above list.\n",
    "Writing the Location name and the number of jobs postings into the excel spreadsheet.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws1.append(['Location','Number of Jobs'])\n",
    "\n",
    "for i in range(len(locations)):\n",
    "    ws1.append(get_number_of_jobs_L(locations[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving into an excel spreadsheet named 'job-postings.xlsx'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wb1.save('2.a-job-postings (Collected from API).xlsx')\n",
    "wb1.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In the similar way, we can try for below given technologies and results  can be stored in an excel sheet.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Collecting the number of job postings for the following languages using the API:\n",
    "\n",
    "*   C\n",
    "*   C#\n",
    "*   C++\n",
    "*   Java\n",
    "*   JavaScript\n",
    "*   Python\n",
    "*   Scala\n",
    "*   Oracle\n",
    "*   SQL Server\n",
    "*   MySQL Server\n",
    "*   PostgreSQL\n",
    "*   MongoDB\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wb2 = Workbook()\n",
    "ws2 = wb2.active\n",
    "\n",
    "languages = ['C','C#','C++','Java','JavaScript','Python','Scala','Oracle','SQL Server','MySQL Server','PostgreSQL','MongoDB']\n",
    "\n",
    "ws2.append(['Languages','Number of Jobs'])\n",
    "\n",
    "for i in range(len(languages)):\n",
    "    ws2.append(get_number_of_jobs_T(languages[i]))\n",
    "\n",
    "wb2.save('2.a-job-postings-languages (Collected from API).xlsx')\n",
    "wb2.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}