Project Scope.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DSCI 521 - Data Analysis and Interpretation\n",
    "#### Term Project Phase 1: Scoping an analytics project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Team Details\n",
    "\n",
    "1. Abhinav Mani Tripathi\n",
    "\n",
    "    Q. What areas/skills/domains does the team member presently identify with?\n",
    "    \n",
    "    A. Analytical Skill – Data Analysis, Data Manipulation, Research and Testing Hypotheses. <br> Mathematics - Creating and Maintaining Algorithms and Statistics. <br> Programming Skill – Python and R.\n",
    "       \n",
    "    Q. Into which areas/skills/domains would the team member like to grow?\n",
    "    \n",
    "    A. Data Analysis and Machine Learning using Python.\n",
    "    \n",
    "2. Aishwarya Ravi\n",
    "\n",
    "    Q. What areas/skills/domains does the team member presently identify with?\n",
    "    \n",
    "    A. Data Acquisition and data preprocessing \n",
    "    \n",
    "    Q. Into which areas/skills/domains would the team member like to grow?\n",
    "    \n",
    "    A. Methods to Observe, explore and interpret data for better analysis.\n",
    "    \n",
    "3. Awanish Tiwari\n",
    "\n",
    "    Q. What areas/skills/domains does the team member presently identify with?\n",
    "    \n",
    "    A. Having a good command in Banking and Insurance domain. Good at applying logics and following timelines. Earlier worked with DB2(Mainframe Database) so having good SQL command as well. Data Analysis and Testing hypothesis is strongest among other skill set.\n",
    "    \n",
    "    Q. Into which areas/skills/domains would the team member like to grow?\n",
    "    \n",
    "    A.  Looking forward to command over Financial domain and digital skills for application development.\n",
    "    \n",
    "4. Nupur Roy Chaudhary\n",
    "\n",
    "    Q. What areas/skills/domains does the team member presently identify with?\n",
    "    \n",
    "    A. Being a software developer and while studying for the Masters in DataScience the areas/skills  which I presently identify with includes : Software Engineering , programming , Data Analytics , Data pre-processing and programming tools.\n",
    "    \n",
    "    Q. Into which areas/skills/domains would the team member like to grow?\n",
    "    \n",
    "    A. The areas which I would like to grow myself and build a firm foundation includes : Data Visualization and Communication, Data Intuition, Data Wrangling, Machine learning and the current subject of Multivariable Calculus and Linear Algebra."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TOPIC : WoRLD of World"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ● Discussion of what you would like to your analysis to do, who/what it will support\n",
    "\n",
    "WoRLD or World Revenue Longitudinal Data is compilation of government tax and non-tax revenuesfrom IMF's Government Financial Statistics. It's strength lies primarily in the breadth of its countries and time. This dataset can be used to encourage the analytical work on revenue, policy and administration, and macro fiscal policy more generally, particularly in relation to developing economies. \n",
    "\n",
    "We are also including the international trade data to cover more robust analysis of GDP variation of different countries. The growing tradability of services is critical for competitiveness of advanced economies.The rise of services trade is a particular crucial engine of trade, growth and jobs in country thus play a vital role in GDP.\n",
    "\n",
    "Under this project we are identifying the benefic and malefic indicators which lays strong impact in National GDP. Also we are using the time series prediction of final values as our end result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ● Exhibition of analyses from dataset(s) explored, including visual analyses, captions, and useful descripitions\n",
    "\n",
    "We are extracting data from different databanks and merging them in one for the identified indicators for all the 190 countries of World. We are doing the acquisition and preprocessing as well because this data is not available in single. Currently we have 65+ indicators which we will be clubbing to have some new dataframes and performing analysis on it. Once we find our main indicators we will be predicting the values of 2030 for them. \n",
    "\n",
    "Below we are using test data created just to show basic EDA of the dataset we have, its just limited to India in testdata but we are covering whole world in our project. The data extracted is from 1960 till 2017 but it needs to be cleaned. We will be performing detail analysis with clean dataset during our project.\n",
    "\n",
    "Showing some glimpse of data and its attributes below for India:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CountryName</th>\n",
       "      <th>CountryCode</th>\n",
       "      <th>SeriesName</th>\n",
       "      <th>SeriesCode</th>\n",
       "      <th>1960[YR1960]</th>\n",
       "      <th>1961[YR1961]</th>\n",
       "      <th>1962[YR1962]</th>\n",
       "      <th>1963[YR1963]</th>\n",
       "      <th>1964[YR1964]</th>\n",
       "      <th>1965[YR1965]</th>\n",
       "      <th>...</th>\n",
       "      <th>2010[YR2010]</th>\n",
       "      <th>2011[YR2011]</th>\n",
       "      <th>2012[YR2012]</th>\n",
       "      <th>2013[YR2013]</th>\n",
       "      <th>2014[YR2014]</th>\n",
       "      <th>2015[YR2015]</th>\n",
       "      <th>2016[YR2016]</th>\n",
       "      <th>2017[YR2017]</th>\n",
       "      <th>2018[YR2018]</th>\n",
       "      <th>2019[YR2019]</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>India</td>\n",
       "      <td>IND</td>\n",
       "      <td>Adjusted net national income (annual % growth)</td>\n",
       "      <td>NY.ADJ.NNTY.KD.ZG</td>\n",
       "      <td>..</td>\n",
       "      <td>..</td>\n",
       "      <td>..</td>\n",
       "      <td>..</td>\n",
       "      <td>..</td>\n",
       "      <td>..</td>\n",
       "      <td>...</td>\n",
       "      <td>10.57214424</td>\n",
       "      <td>4.999235888</td>\n",
       "      <td>6.125368194</td>\n",
       "      <td>5.72454801</td>\n",
       "      <td>6.991054566</td>\n",
       "      <td>8.816296268</td>\n",
       "      <td>7.570456273</td>\n",
       "      <td>5.13696452</td>\n",
       "      <td>..</td>\n",
       "      <td>..</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>India</td>\n",
       "      <td>IND</td>\n",
       "      <td>Adjusted net national income (current US$)</td>\n",
       "      <td>NY.ADJ.NNTY.CD</td>\n",
       "      <td>..</td>\n",
       "      <td>..</td>\n",
       "      <td>..</td>\n",
       "      <td>..</td>\n",
       "      <td>..</td>\n",
       "      <td>..</td>\n",
       "      <td>...</td>\n",
       "      <td>1.45E+12</td>\n",
       "      <td>1.57E+12</td>\n",
       "      <td>1.58E+12</td>\n",
       "      <td>1.60E+12</td>\n",
       "      <td>1.77E+12</td>\n",
       "      <td>1.84E+12</td>\n",
       "      <td>1.99E+12</td>\n",
       "      <td>2.25E+12</td>\n",
       "      <td>..</td>\n",
       "      <td>..</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>India</td>\n",
       "      <td>IND</td>\n",
       "      <td>Agricultural raw materials imports (% of merch...</td>\n",
       "      <td>TM.VAL.AGRI.ZS.UN</td>\n",
       "      <td>..</td>\n",
       "      <td>..</td>\n",
       "      <td>9.242249308</td>\n",
       "      <td>8.603457071</td>\n",
       "      <td>8.383790138</td>\n",
       "      <td>7.394065716</td>\n",
       "      <td>...</td>\n",
       "      <td>1.776209565</td>\n",
       "      <td>1.771862</td>\n",
       "      <td>1.827499091</td>\n",
       "      <td>1.788991089</td>\n",
       "      <td>1.896625912</td>\n",
       "      <td>2.005692719</td>\n",
       "      <td>2.166929344</td>\n",
       "      <td>1.990886193</td>\n",
       "      <td>1.858205476</td>\n",
       "      <td>..</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>India</td>\n",
       "      <td>IND</td>\n",
       "      <td>Agricultural raw materials exports (% of merch...</td>\n",
       "      <td>TX.VAL.AGRI.ZS.UN</td>\n",
       "      <td>..</td>\n",
       "      <td>..</td>\n",
       "      <td>9.257298841</td>\n",
       "      <td>8.252923511</td>\n",
       "      <td>7.990264648</td>\n",
       "      <td>7.711348153</td>\n",
       "      <td>...</td>\n",
       "      <td>2.006039429</td>\n",
       "      <td>1.829675587</td>\n",
       "      <td>1.954673746</td>\n",
       "      <td>2.061501155</td>\n",
       "      <td>1.57458613</td>\n",
       "      <td>1.532101146</td>\n",
       "      <td>1.37944857</td>\n",
       "      <td>1.310121602</td>\n",
       "      <td>1.360103324</td>\n",
       "      <td>..</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>India</td>\n",
       "      <td>IND</td>\n",
       "      <td>Agriculture, forestry, and fishing, value adde...</td>\n",
       "      <td>NV.AGR.TOTL.ZS</td>\n",
       "      <td>41.31283906</td>\n",
       "      <td>40.38740161</td>\n",
       "      <td>38.3713352</td>\n",
       "      <td>39.31975131</td>\n",
       "      <td>41.18773833</td>\n",
       "      <td>38.95214348</td>\n",
       "      <td>...</td>\n",
       "      <td>17.02650901</td>\n",
       "      <td>17.19197043</td>\n",
       "      <td>16.84537707</td>\n",
       "      <td>17.14842354</td>\n",
       "      <td>16.79193451</td>\n",
       "      <td>16.17450811</td>\n",
       "      <td>16.24980967</td>\n",
       "      <td>15.61945495</td>\n",
       "      <td>14.60193402</td>\n",
       "      <td>..</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 64 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  CountryName CountryCode                                         SeriesName  \\\n",
       "0       India         IND     Adjusted net national income (annual % growth)   \n",
       "1       India         IND         Adjusted net national income (current US$)   \n",
       "2       India         IND  Agricultural raw materials imports (% of merch...   \n",
       "3       India         IND  Agricultural raw materials exports (% of merch...   \n",
       "4       India         IND  Agriculture, forestry, and fishing, value adde...   \n",
       "\n",
       "          SeriesCode 1960[YR1960] 1961[YR1961] 1962[YR1962] 1963[YR1963]  \\\n",
       "0  NY.ADJ.NNTY.KD.ZG           ..           ..           ..           ..   \n",
       "1     NY.ADJ.NNTY.CD           ..           ..           ..           ..   \n",
       "2  TM.VAL.AGRI.ZS.UN           ..           ..  9.242249308  8.603457071   \n",
       "3  TX.VAL.AGRI.ZS.UN           ..           ..  9.257298841  8.252923511   \n",
       "4     NV.AGR.TOTL.ZS  41.31283906  40.38740161   38.3713352  39.31975131   \n",
       "\n",
       "  1964[YR1964] 1965[YR1965]  ... 2010[YR2010] 2011[YR2011] 2012[YR2012]  \\\n",
       "0           ..           ..  ...  10.57214424  4.999235888  6.125368194   \n",
       "1           ..           ..  ...     1.45E+12     1.57E+12     1.58E+12   \n",
       "2  8.383790138  7.394065716  ...  1.776209565     1.771862  1.827499091   \n",
       "3  7.990264648  7.711348153  ...  2.006039429  1.829675587  1.954673746   \n",
       "4  41.18773833  38.95214348  ...  17.02650901  17.19197043  16.84537707   \n",
       "\n",
       "  2013[YR2013] 2014[YR2014] 2015[YR2015] 2016[YR2016] 2017[YR2017]  \\\n",
       "0   5.72454801  6.991054566  8.816296268  7.570456273   5.13696452   \n",
       "1     1.60E+12     1.77E+12     1.84E+12     1.99E+12     2.25E+12   \n",
       "2  1.788991089  1.896625912  2.005692719  2.166929344  1.990886193   \n",
       "3  2.061501155   1.57458613  1.532101146   1.37944857  1.310121602   \n",
       "4  17.14842354  16.79193451  16.17450811  16.24980967  15.61945495   \n",
       "\n",
       "  2018[YR2018] 2019[YR2019]  \n",
       "0           ..           ..  \n",
       "1           ..           ..  \n",
       "2  1.858205476           ..  \n",
       "3  1.360103324           ..  \n",
       "4  14.60193402           ..  \n",
       "\n",
       "[5 rows x 64 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "India_data = pd.read_csv(\n",
    "    filepath_or_buffer= \"TestData.csv\", \n",
    "    sep = \",\",\n",
    "    header = 0\n",
    ")\n",
    "\n",
    "India_data.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Adjusted net national income (annual % growth)',\n",
       " 'Adjusted net national income (current US$)',\n",
       " 'Agricultural raw materials imports (% of merchandise imports)',\n",
       " 'Agricultural raw materials exports (% of merchandise exports)',\n",
       " 'Agriculture, forestry, and fishing, value added (% of GDP)',\n",
       " 'Arms exports (SIPRI trend indicator values)',\n",
       " 'Arms imports (SIPRI trend indicator values)',\n",
       " 'Bank capital to assets ratio (%)',\n",
       " 'Bank liquid reserves to bank assets ratio (%)',\n",
       " 'Broad money (% of GDP)',\n",
       " 'Central government debt, total (% of GDP)',\n",
       " 'Claims on central government, etc. (% GDP)',\n",
       " 'Claims on other sectors of the domestic economy (% of GDP)',\n",
       " 'Coal rents (% of GDP)',\n",
       " 'Commercial service exports (current US$)',\n",
       " 'Commercial service imports (current US$)',\n",
       " 'Current account balance (% of GDP)',\n",
       " 'Current health expenditure (% of GDP)',\n",
       " 'Domestic credit provided by financial sector (% of GDP)',\n",
       " 'Domestic credit to private sector (% of GDP)',\n",
       " 'Domestic credit to private sector by banks (% of GDP)',\n",
       " 'Domestic general government health expenditure (% of GDP)',\n",
       " 'Expense (% of GDP)',\n",
       " 'Exports of goods and services (% of GDP)',\n",
       " 'External balance on goods and services (% of GDP)',\n",
       " 'Final consumption expenditure (% of GDP)',\n",
       " 'Foreign direct investment, net inflows (% of GDP)',\n",
       " 'Forest rents (% of GDP)',\n",
       " 'General government final consumption expenditure (% of GDP)',\n",
       " 'Government expenditure on education, total (% of GDP)',\n",
       " 'Gross capital formation (% of GDP)',\n",
       " 'Gross domestic savings (% of GDP)',\n",
       " 'Gross fixed capital formation (% of GDP)',\n",
       " 'Gross national expenditure (% of GDP)',\n",
       " 'Gross savings (% of GDP)',\n",
       " 'Imports of goods and services (% of GDP)',\n",
       " 'Industry (including construction), value added (% of GDP)',\n",
       " 'Inflation, GDP deflator (annual %)',\n",
       " 'International tourism, receipts (% of total exports)',\n",
       " 'Manufacturing, value added (% of GDP)',\n",
       " 'Market capitalization of listed domestic companies (% of GDP)',\n",
       " 'Merchandise trade (% of GDP)',\n",
       " 'Military expenditure (% of GDP)',\n",
       " 'Mineral rents (% of GDP)',\n",
       " 'Natural gas rents (% of GDP)',\n",
       " 'Net acquisition of financial assets (% of GDP)',\n",
       " 'Net incurrence of liabilities, total (% of GDP)',\n",
       " 'Net investment in nonfinancial assets (% of GDP)',\n",
       " 'Net lending (+) / net borrowing (-) (% of GDP)',\n",
       " 'Oil rents (% of GDP)',\n",
       " 'Personal remittances, received (% of GDP)',\n",
       " 'Revenue, excluding grants (% of GDP)',\n",
       " 'Stocks traded, total value (% of GDP)',\n",
       " 'Tax revenue (% of GDP)',\n",
       " 'Total natural resources rents (% of GDP)',\n",
       " 'Trade (% of GDP)',\n",
       " 'Trade in services (% of GDP)']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "## initialize a counter for the positions\n",
    "positions = Counter()\n",
    "\n",
    "## loop over rows to count up the positions\n",
    "for ix, row in India_data.iterrows():\n",
    "    positions[row[\"Series Name\"]] += 1\n",
    "\n",
    "list(positions.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ● Discussion of who might be interested in your analysis\n",
    "\n",
    "As stated earlier this data contains the Revenue and financial data of countries combined which affect GDP and thus the growth of the country. It also tells the current position of the country amongst underdeveloped, developing or developed nations. This analysis would be useful for the finance department of any nation or the officers looking at the economic trend. This is much useful for economists who can further take this analysis to match the country's growth."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ● Discussion of how your analysis might fit into an application or investigation\n",
    "\n",
    "Under this project, we are focussing on the prediction of GDP and economic growth of the country depending on the variation of parameters. This project will also cover the different factors which can boom or burst the economy thus helpful for economists to bring superior plans to promote the nation's growth."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ● Discussion of how your analysis is limited and could be improved\n",
    "\n",
    "Currently, we are having the limited data which are made public from IMF and WorldBank so the analysis is done based on those data only. Also, the data is not rich for all parameters or indicators thus more rich data will give a better prediction. Also, this analysis is done on a very small scale as the knowledge of financial domain is limited to our members so we are not able to cover much but a max of them are in. We are learning a few economic journals to enrich our financial domain understanding and maybe we do some alterations later on while developing projects. Alteration in sense to make project rich not to change project :)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ● Selection of data for continued analysis, including justification\n",
    "\n",
    "Data is mainly captured from IMF and WorldBank databank after reading several journals and links related to finances. Some of them are presented below:\n",
    "\n",
    "IMF : http://data.imf.org/?sk=77413F1D-1525-450A-A23A-47AEED40FE78\n",
    "\n",
    "IMF : http://data.imf.org/?sk=07109577-E65D-4CE1-BB21-0CB3098FC504\n",
    "\n",
    "IMF : http://www.imf.org/en/Publications/WP/Issues/2017/03/29/World-Trade-in-Services-Evidence-from-A-New-Dataset-44776\n",
    "\n",
    "IMF : http://data.imf.org/bop\n",
    "\n",
    "WorldBank : http://datatopics.worldbank.org/world-development-indicators/\n",
    "\n",
    "IMF : https://www.imf.org/external/pubs/ft/fandd/basics/gdp.htm\n",
    "\n",
    "GDP can be viewed in three different ways:\n",
    "\n",
    "● The production approach sums the “value-added” at each stage of production, where value-added is defined as total sales less the value of intermediate inputs into the production process. For example, flour would be an intermediate input and bread the final product; or an architect’s services would be an intermediate input and the building the final product.\n",
    "\n",
    "● The expenditure approach adds up the value of purchases made by final users—for example, the consumption of food, televisions, and medical services by households; the investments in machinery by companies; and the purchases of goods and services by the government and foreigners.\n",
    "\n",
    "● The income approach sums the incomes generated by production—for example, the compensation employees receive and the operating surplus of companies (roughly sales less costs).\n",
    "\n",
    "So we selected the parameters which are related to the approach above.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ● Discussion of how your analysis might be completed and disseminated, i.e., who's the target audience?\n",
    "\n",
    "Target audience as stated earlier may be any Data scientist, economist or finance officers looking for economic data for identifying the fallback and favor indicators for Nation's growth. Any industrialist can also look for the trade trend to extract maximum benefit through a wiki page of the trend analysis generated for the companies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}