covid-19-current-situation-on-october.ipynb

{"cells":[{"metadata":{},"cell_type":"markdown","source":"![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F518134%2F485aa04e87e4e45c91815101784c6d95%2Fcorona-4930541_1280.jpg?generation=1585438527494582&alt=media)\n\n# COVID-19: current situation on October\n\nThe kernel is inspired by the great EDA kernel [COVID-19: Digging a Bit Deeper](https://www.kaggle.com/abhinand05/covid-19-digging-a-bit-deeper) by @abhinand05 in week1.<br/>\nPlease upvote his kernel as well :)<br/>\nI will write an EDA including **the recent updates, recovering country analysis & sigmoid fitting convergence date estimation**.\n\n[Note] It seems JHU has changed the data format, and stopped providing recovered cases. So I could not analyze recovered cases.\n\n`plotly` visualization is heavy used in this kernel so that we can **interactively** see the figure, map etc.<br/>\nAs a side effect, it might take a little bit more time to load the kernel, wait a minute please.\n\n### Version History\n\nThe data is updated daily-basis. **I will try to update the kernel DAILY basis too, so that you can refer latest information.**<br/>\nBelow are the version history to see the information until specified date.<br/>\n\n<details>\n    <summary>Version History (Expand by clicking here)</summary><br/>\n - Version 15: Added **Daily NEW confirmed cases** analysis & **Asia** region EDA.\n - [Version 18](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-march?scriptVersionId=31151381): Shows figure as of 2020/3/28.\n - Version 19: Added <span style=\"color:red\">sigmoid fitting to estimate when the coronavirus converge in each country</span>, jump to [When will it converge? - Estimation by sigmoid fitting](#id_converge).\n - [Version 21](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=31276251): Shows figure as of 2020/3/31.\n - [Version 23](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=31329249): Shows figure as of 2020/4/1, moved to use week3 competition data.\n - [Version 29](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=31384138): 2020/4/2\n - [Version 30](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=31436722): 2020/4/3\n - [Version 31](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=31489960): 2020/4/4\n - [Version 32](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=31547500): 2020/4/5\n - [Version 33](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=31604729): 2020/4/6\n - [Version 35](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=31670710): 2020/4/7, **Moved to week4 dataset**\n - [Version 36](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=31723129): 2020/4/8\n - [Version 38](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=31782521): 2020/4/9\n - [Version 39](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=31839132): 2020/4/10\n - [Version 40](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=31898610): 2020/4/11\n - [Version 41](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=31957894): 2020/4/12\n - [Version 42](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=32023899): 2020/4/13\n - [Version 43](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=32082933): 2020/4/14\n - [Version 49](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=32266557): 2020/4/17, **<span style=\"color:red\">Moved to use data from <a href=\"https://github.com/CSSEGISandData/COVID-19\">2019 Novel Coronavirus COVID-19 (2019-nCoV) Data Repository</a><span style=\"color:red\"> by Johns Hopkins CSSE**. Internet connection is required, since it will download latest data from the repository.\n - [Version 50](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=32321582): 2020/4/18, removed unused data.\n - [Version 51](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=32382070): 2020/4/19\n - [Version 52](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=32623228): 2020/4/23\n - [Version 53](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=32751754): 2020/4/25\n - [Version 54](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=32821067): 2020/4/26\n - [Version 55](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=32966399): 2020/4/28\n - [Version 56](https://www.kaggle.com/corochann/covid-19-eda-with-recent-update-on-april?scriptVersionId=33111789): 2020/4/30\n - [Version 57](https://www.kaggle.com/corochann/covid-19-current-situation-on-may-daily-update?scriptVersionId=33178751): 2020/5/1    \n - [Version 58](https://www.kaggle.com/corochann/covid-19-current-situation-on-may-daily-update?scriptVersionId=33246349): 2020/5/2   \n - [Version 59](https://www.kaggle.com/corochann/covid-19-current-situation-on-may-daily-update?scriptVersionId=33316049): 2020/5/3\n - [Version 60](https://www.kaggle.com/corochann/covid-19-current-situation-on-may-daily-update?scriptVersionId=33388532): 2020/5/4\n - [Version 61](https://www.kaggle.com/corochann/covid-19-current-situation-on-may-daily-update?scriptVersionId=33456431): 2020/5/5\n - [Version 62](https://www.kaggle.com/corochann/covid-19-current-situation-on-may-daily-update?scriptVersionId=33523879): 2020/5/6\n - [Version 63](https://www.kaggle.com/corochann/covid-19-current-situation-on-may-daily-update?scriptVersionId=33592887): 2020/5/7\n - [Version 64](https://www.kaggle.com/corochann/covid-19-current-situation-on-may-daily-update?scriptVersionId=33665835): 2020/5/8\n - [Version 65](https://www.kaggle.com/corochann/covid-19-current-situation-on-may-daily-update?scriptVersionId=33728654): 2020/5/9\n - [Version 66](https://www.kaggle.com/corochann/covid-19-current-situation-on-may-daily-update?scriptVersionId=33801408): 2020/5/10\n - [Version 69](https://www.kaggle.com/corochann/covid-19-current-situation-on-may-daily-update?scriptVersionId=33943023): 2020/5/12\n - [Version 70](https://www.kaggle.com/corochann/covid-19-current-situation-on-may?scriptVersionId=34085641): 2020/5/14\n - [Version 71](https://www.kaggle.com/corochann/covid-19-current-situation-on-may?scriptVersionId=34273754): 2020/5/17\n - [Version 72](https://www.kaggle.com/corochann/covid-19-current-situation-on-may?scriptVersionId=34415107): 2020/5/19\n - [Version 73](https://www.kaggle.com/corochann/covid-19-current-situation-on-may?scriptVersionId=34661387): 2020/5/22, Added growth factor\n - [Version 75](https://www.kaggle.com/corochann/covid-19-current-situation-on-may?scriptVersionId=34870894): 2020/5/25\n - [Version 76](https://www.kaggle.com/corochann/covid-19-current-situation-on-may?scriptVersionId=35075854): 2020/5/28\n - [Version 77](https://www.kaggle.com/corochann/covid-19-current-situation-on-june?scriptVersionId=35340282): 2020/6/1\n - [Version 78](https://www.kaggle.com/corochann/covid-19-current-situation-on-june?scriptVersionId=35475709): 2020/6/3\n - [Version 79](https://www.kaggle.com/corochann/covid-19-current-situation-on-june?scriptVersionId=35767313): 2020/6/7\n - [Version 80](https://www.kaggle.com/corochann/covid-19-current-situation-on-june?scriptVersionId=36013977): 2020/6/10\n - [Version 81](https://www.kaggle.com/corochann/covid-19-current-situation-on-june?scriptVersionId=37235122): 2020/6/21\n - [Version 82](https://www.kaggle.com/corochann/covid-19-current-situation-on-june?scriptVersionId=37739722): 2020/6/28\n - [Version 83](https://www.kaggle.com/corochann/covid-19-current-situation-on-july?scriptVersionId=38064445): 2020/7/2\n - [Version 84](https://www.kaggle.com/corochann/covid-19-current-situation-on-july?scriptVersionId=38768566): 2020/7/13\n - [Version 85](https://www.kaggle.com/corochann/covid-19-current-situation-on-july?scriptVersionId=39279123): 2020/7/21\n - [Version 86](https://www.kaggle.com/corochann/covid-19-current-situation-on-august?scriptVersionId=39996289): 2020/8/1\n - [Version 87](https://www.kaggle.com/corochann/covid-19-current-situation-on-august?scriptVersionId=40873956): 2020/8/15\n - [Version 88](https://www.kaggle.com/corochann/covid-19-current-situation-on-august?scriptVersionId=41227754): 2020/8/22\n - [Version 89](https://www.kaggle.com/corochann/covid-19-current-situation-on-september?scriptVersionId=41936057): 2020/9/2\n - [Version 90](https://www.kaggle.com/corochann/covid-19-current-situation-on-september?scriptVersionId=43173201): 2020/9/20\n - [Version 91](https://www.kaggle.com/corochann/covid-19-current-situation-on-october?scriptVersionId=44115812): 2020/10/4\n    \n    \n</details>\n\n - Latest Version: 2020/10/21"},{"metadata":{},"cell_type":"markdown","source":"## Table of Contents\n\n\n**[Load Data](#id_load)**<br/>\n**[Worldwide trend](#id_ww)**<br/>\n**[Country-wise growth](#id_country)**<br/>\n**[Going into province](#id_province)**<br/>\n**[Zoom up to US: what is happening in US now??](#id_province)**<br/>\n**[Europe](#id_europe)**<br/>\n**[Asia](#id_asia)**<br/>\n**[Which country is recovering now?](#id_recover)**<br/>\n**[When will it converge? - Estimation by sigmoid fitting](#id_converge)**<br/>\n**[Further reading](#id_ref)**<br/>"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import gc\nimport os\nfrom pathlib import Path\nimport random\nimport sys\n\nfrom tqdm.notebook import tqdm\nimport numpy as np\nimport pandas as pd\nimport scipy as sp\n\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom IPython.core.display import display, HTML\n\n# --- plotly ---\nfrom plotly import tools, subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.express as px\nimport plotly.figure_factory as ff\nimport plotly.io as pio\npio.templates.default = \"plotly_dark\"\n\n# --- models ---\nfrom sklearn import preprocessing\nfrom sklearn.model_selection import KFold\nimport lightgbm as lgb\nimport xgboost as xgb\nimport catboost as cb\n\n# --- setup ---\npd.set_option('max_columns', 50)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id_load\"></a>\n# Load Data\n\nDownload latest data from Johns Hopkins University github repository: [https://github.com/CSSEGISandData/COVID-19](https://github.com/CSSEGISandData/COVID-19)"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# Input data files are available in the \"../input/\" directory.\nimport os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     filenames.sort()\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Referenced: https://www.kaggle.com/benhamner/covid-19-forecasting-challenges-week-2-data-prep"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"%%time\nimport requests\n\nfor filename in ['time_series_covid19_confirmed_global.csv',\n                 'time_series_covid19_deaths_global.csv',\n                 'time_series_covid19_recovered_global.csv',\n                 'time_series_covid19_confirmed_US.csv',\n                 'time_series_covid19_deaths_US.csv']:\n    print(f'Downloading {filename}')\n    url = f'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/{filename}'\n    myfile = requests.get(url)\n    open(filename, 'wb').write(myfile.content)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from datetime import datetime\n\ndef _convert_date_str(df):\n    try:\n        df.columns = list(df.columns[:4]) + [datetime.strptime(d, \"%m/%d/%y\").date().strftime(\"%Y-%m-%d\") for d in df.columns[4:]]\n    except:\n        print('_convert_date_str failed with %y, try %Y')\n        df.columns = list(df.columns[:4]) + [datetime.strptime(d, \"%m/%d/%Y\").date().strftime(\"%Y-%m-%d\") for d in df.columns[4:]]\n\n\nconfirmed_global_df = pd.read_csv('time_series_covid19_confirmed_global.csv')\n_convert_date_str(confirmed_global_df)\n\ndeaths_global_df = pd.read_csv('time_series_covid19_deaths_global.csv')\n_convert_date_str(deaths_global_df)\n\nrecovered_global_df = pd.read_csv('time_series_covid19_recovered_global.csv')\n_convert_date_str(recovered_global_df)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Filter out problematic data points (The West Bank and Gaza had a negative value, cruise ships were associated with Canada, etc.)\nremoved_states = \"Recovered|Grand Princess|Diamond Princess\"\nremoved_countries = \"US|The West Bank and Gaza\"\n\nconfirmed_global_df.rename(columns={\"Province/State\": \"Province_State\", \"Country/Region\": \"Country_Region\"}, inplace=True)\ndeaths_global_df.rename(columns={\"Province/State\": \"Province_State\", \"Country/Region\": \"Country_Region\"}, inplace=True)\nrecovered_global_df.rename(columns={\"Province/State\": \"Province_State\", \"Country/Region\": \"Country_Region\"}, inplace=True)\n\nconfirmed_global_df = confirmed_global_df[~confirmed_global_df[\"Province_State\"].replace(np.nan, \"nan\").str.match(removed_states)]\ndeaths_global_df    = deaths_global_df[~deaths_global_df[\"Province_State\"].replace(np.nan, \"nan\").str.match(removed_states)]\nrecovered_global_df = recovered_global_df[~recovered_global_df[\"Province_State\"].replace(np.nan, \"nan\").str.match(removed_states)]\n\nconfirmed_global_df = confirmed_global_df[~confirmed_global_df[\"Country_Region\"].replace(np.nan, \"nan\").str.match(removed_countries)]\ndeaths_global_df    = deaths_global_df[~deaths_global_df[\"Country_Region\"].replace(np.nan, \"nan\").str.match(removed_countries)]\nrecovered_global_df = recovered_global_df[~recovered_global_df[\"Country_Region\"].replace(np.nan, \"nan\").str.match(removed_countries)]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"confirmed_global_melt_df = confirmed_global_df.melt(\n    id_vars=['Country_Region', 'Province_State', 'Lat', 'Long'], value_vars=confirmed_global_df.columns[4:], var_name='Date', value_name='ConfirmedCases')\ndeaths_global_melt_df = deaths_global_df.melt(\n    id_vars=['Country_Region', 'Province_State', 'Lat', 'Long'], value_vars=confirmed_global_df.columns[4:], var_name='Date', value_name='Deaths')\nrecovered_global_melt_df = deaths_global_df.melt(\n    id_vars=['Country_Region', 'Province_State', 'Lat', 'Long'], value_vars=confirmed_global_df.columns[4:], var_name='Date', value_name='Recovered')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train = confirmed_global_melt_df.merge(deaths_global_melt_df, on=['Country_Region', 'Province_State', 'Lat', 'Long', 'Date'])\ntrain = train.merge(recovered_global_melt_df, on=['Country_Region', 'Province_State', 'Lat', 'Long', 'Date'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# --- US ---\nconfirmed_us_df = pd.read_csv('time_series_covid19_confirmed_US.csv')\ndeaths_us_df = pd.read_csv('time_series_covid19_deaths_US.csv')\n\nconfirmed_us_df.drop(['UID', 'iso2', 'iso3', 'code3', 'FIPS', 'Admin2', 'Combined_Key'], inplace=True, axis=1)\ndeaths_us_df.drop(['UID', 'iso2', 'iso3', 'code3', 'FIPS', 'Admin2', 'Combined_Key', 'Population'], inplace=True, axis=1)\n\nconfirmed_us_df.rename({'Long_': 'Long'}, axis=1, inplace=True)\ndeaths_us_df.rename({'Long_': 'Long'}, axis=1, inplace=True)\n\n_convert_date_str(confirmed_us_df)\n_convert_date_str(deaths_us_df)\n\n# clean\nconfirmed_us_df = confirmed_us_df[~confirmed_us_df.Province_State.str.match(\"Diamond Princess|Grand Princess|Recovered|Northern Mariana Islands|American Samoa\")]\ndeaths_us_df = deaths_us_df[~deaths_us_df.Province_State.str.match(\"Diamond Princess|Grand Princess|Recovered|Northern Mariana Islands|American Samoa\")]\n\n# --- Aggregate by province state ---\n#confirmed_us_df.groupby(['Country_Region', 'Province_State'])\nconfirmed_us_df = confirmed_us_df.groupby(['Country_Region', 'Province_State']).sum().reset_index()\ndeaths_us_df = deaths_us_df.groupby(['Country_Region', 'Province_State']).sum().reset_index()\n\n# remove lat, long.\nconfirmed_us_df.drop(['Lat', 'Long'], inplace=True, axis=1)\ndeaths_us_df.drop(['Lat', 'Long'], inplace=True, axis=1)\n\nconfirmed_us_melt_df = confirmed_us_df.melt(\n    id_vars=['Country_Region', 'Province_State'], value_vars=confirmed_us_df.columns[2:], var_name='Date', value_name='ConfirmedCases')\ndeaths_us_melt_df = deaths_us_df.melt(\n    id_vars=['Country_Region', 'Province_State'], value_vars=deaths_us_df.columns[2:], var_name='Date', value_name='Deaths')\n\ntrain_us = confirmed_us_melt_df.merge(deaths_us_melt_df, on=['Country_Region', 'Province_State', 'Date'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train = pd.concat([train, train_us], axis=0, sort=False)\n\ntrain_us.rename({'Country_Region': 'country', 'Province_State': 'province', 'Date': 'date', 'ConfirmedCases': 'confirmed', 'Deaths': 'fatalities'}, axis=1, inplace=True)\ntrain_us['country_province'] = train_us['country'].fillna('') + '/' + train_us['province'].fillna('')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"%%time\ndatadir = Path('/kaggle/input/covid19-global-forecasting-week-4')\n\n# Read in the data CSV files\n#train = pd.read_csv(datadir/'train.csv')\n#test = pd.read_csv(datadir/'test.csv')\n#submission = pd.read_csv(datadir/'submission.csv')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"#test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"#submission","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"train.rename({'Country_Region': 'country', 'Province_State': 'province', 'Id': 'id', 'Date': 'date', 'ConfirmedCases': 'confirmed', 'Deaths': 'fatalities', 'Recovered': 'recovered'}, axis=1, inplace=True)\ntrain['country_province'] = train['country'].fillna('') + '/' + train['province'].fillna('')\n\n# test.rename({'Country_Region': 'country', 'Province_State': 'province', 'Id': 'id', 'Date': 'date', 'ConfirmedCases': 'confirmed', 'Fatalities': 'fatalities'}, axis=1, inplace=True)\n# test['country_province'] = test['country'].fillna('') + '/' + test['province'].fillna('')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id_ww\"></a>\n# Worldwide trend"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"ww_df = train.groupby('date')[['confirmed', 'fatalities']].sum().reset_index()\nww_df['new_case'] = ww_df['confirmed'] - ww_df['confirmed'].shift(1)\nww_df['growth_factor'] = ww_df['new_case'] / ww_df['new_case'].shift(1)\nww_df.tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"ww_melt_df = pd.melt(ww_df, id_vars=['date'], value_vars=['confirmed', 'fatalities', 'new_case'])\nww_melt_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When we see the confirmed cases in world wide, it just look like exponential growth curve. The number is increasing very rapidly especially recently. **the number almost doubled in last 1 week**...\n\n<span style=\"color:red\"><b>Confirmed cases reached 1M people, and 52K people already died on April 2</b></span>.<br/>\n<span style=\"color:red\"><b>Confirmed cases reached 10M people, and 500K people already died on June 28</b></span>.<br/>\n<span style=\"color:red\"><b>Confirmed cases reached 20M people, and 750K people already died on Aug 13</b></span>.\n\n<span style=\"color:red\"><b>Daily new confirmed cases started not increasing from April 4. After that flat trend continues so far.</b></span>."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.line(ww_melt_df, x=\"date\", y=\"value\", color='variable', \n              title=\"Worldwide Confirmed/Death Cases Over Time\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Moreover, when we check the growth in log-scale below figure, we can see that the speed of confirmed cases growth rate **slightly increases** when compared with the beginning of March and end of March.<br/>\nIn spite of the Lockdown policy in Europe or US, the number was increasing rapidly around the beginning of April."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.line(ww_melt_df, x=\"date\", y=\"value\", color='variable',\n              title=\"Worldwide Confirmed/Death Cases Over Time (Log scale)\",\n             log_y=True)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It looks like `fatalities` curve is just shifted the `confirmed` curve to below in log-scale, which means mortality rate is almost constant.\n\nIs it true? Let's see mortality rate in detail.<br/>\nWe see that mortality rate is kept almost 3%, however it is slightly **increasing gradually to go over 7%** at the end of April.<br/>\n\nWhy? I will show you later that Europe & US has more seriously infected by Coronavirus recently, and mortality rate is high in these regions.<br/>\nIt might be because when too many people get coronavirus, the country cannot provide enough medical treatment.\n\n**It seems mortality rate is start decreasing on May!!** Is it because enough medical care is started to reaching out to everyone or the number of inspection increased and many \"hidden\" confirmed cases are detected now?"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"ww_df['mortality'] = ww_df['fatalities'] / ww_df['confirmed']\n\nfig = px.line(ww_df, x=\"date\", y=\"mortality\", \n              title=\"Worldwide Mortality Rate Over Time\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check growth factor:\n\n> Growth factor is the factor by which a quantity multiplies itself over time. The formula used is every day's new cases / new cases on the previous day.\n\nWhen this number is more than 1. the number of confirmed cases will be increasing, and when it keeps below 1. the number of confirmed cases will decrease.\nSo it is important to check growth factor is kept below 1. or not.\n\nIn worldwide, growth factor is around 1. from May, it means that confirmed cases does not increase, but not decreasing so far...\n\nReference:\n - [Covid-19 Predictions, Growth Factor, and Calculus](https://www.kaggle.com/dferhadi/covid-19-predictions-growth-factor-and-calculus)\n - [Coronavirus Cases](https://www.worldometers.info/coronavirus/coronavirus-cases/#cases-growth-factor)\n - [The one COVID-19 number to watch](https://www.abc.net.au/news/2020-04-10/coronavirus-data-australia-growth-factor-covid-19/12132478?nw=0)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = px.line(ww_df, x=\"date\", y=\"growth_factor\", \n              title=\"Worldwide Growth Factor Over Time\")\nfig.add_trace(go.Scatter(x=[ww_df['date'].min(), ww_df['date'].max()], y=[1., 1.], name='Growth factor=1.', line=dict(dash='dash', color=('rgb(255, 0, 0)'))))\nfig.update_yaxes(range=[0., 5.])\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id_country\"></a>\n# Country-wise growth"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"country_df = train.groupby(['date', 'country'])[['confirmed', 'fatalities']].sum().reset_index()\ncountry_df.tail()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What kind of country is in the dataset? How's the distribution of number of confirmed cases by country?"},{"metadata":{"trusted":true},"cell_type":"code","source":"countries = country_df['country'].unique()\nprint(f'{len(countries)} countries are in dataset:\\n{countries}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"target_date = country_df['date'].max()\n\nprint('Date: ', target_date)\nfor i in [1, 10, 100, 1000, 10000]:\n    n_countries = len(country_df.query('(date == @target_date) & confirmed > @i'))\n    print(f'{n_countries} countries have more than {i} confirmed cases')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"ax = sns.distplot(np.log10(country_df.query('date == \"2020-03-27\"')['confirmed'] + 1))\nax.set_xlim([0, 6])\nax.set_xticks(np.arange(7))\n_ = ax.set_xticklabels(['0', '10', '100', '1k', '10k', '100k'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is difficult to see all countries so let's check top countries."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"top_country_df = country_df.query('(date == @target_date) & (confirmed > 1000)').sort_values('confirmed', ascending=False)\ntop_country_melt_df = pd.melt(top_country_df, id_vars='country', value_vars=['confirmed', 'fatalities'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now **US, Italy and Spain** has more confirmed cases than China, and we can see many Europe countries in the top.\n\nKorea also appears in relatively top despite of its population, this is because Korea execcutes inspection check aggressively."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.bar(top_country_melt_df.iloc[::-1],\n             x='value', y='country', color='variable', barmode='group',\n             title=f'Confirmed Cases/Deaths on {target_date}', text='value', height=1500, orientation='h')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check these major country's growth by date.\n\nAs we can see, Coronavirus hit **China** at first but its trend is slowing down in March which is good news.<br/>\nBad news is 2nd wave comes to **Europe (Italy, Spain, Germany, France, UK)** at March.<br/>\nBut more sadly 3rd wave now comes to **US, whose growth rate is much much faster than China, or even Europe**. Its main spread starts from middle of March and its speed is faster than Italy. Now US seems to be in the most serious situation in terms of both total number and spread speed.<br/>\n\nFrom June, **Branzil & Russia** getting increased.<br/>\nFrom July, **Peru, Chile in South America & India in South Asia** getting increased.\n\nCoronavirus spreads really the \"All\" over the world, it seems it still need a long time until converge.\n\nYou can click country legend to show/hide each country's line plot.\nIf you cannot see the line (due to plotly bug?) in the plot, you can also try clicking country legend. Line will appear."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"top30_countries = top_country_df.sort_values('confirmed', ascending=False).iloc[:30]['country'].unique()\ntop30_countries_df = country_df[country_df['country'].isin(top30_countries)]\nfig = px.line(top30_countries_df,\n              x='date', y='confirmed', color='country',\n              title=f'Confirmed Cases for top 30 country as of {target_date}')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In terms of number of fatalities, Europe & US are serious situation now.<br/>\nMany countries have more fatalities than China now, including US, Italy, Spain, France, UK, Iran Belgium, Germany, Brazil, Netherlands.\n\n**US's spread speed is the fastest, US's fatality cases become top1 on Apr 10th.**\n\nIn June, the number increases rapidly in South America, **Brazil & Mexico**."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"top30_countries = top_country_df.sort_values('fatalities', ascending=False).iloc[:30]['country'].unique()\ntop30_countries_df = country_df[country_df['country'].isin(top30_countries)]\nfig = px.line(top30_countries_df,\n              x='date', y='fatalities', color='country',\n              title=f'Fatalities for top 30 country as of {target_date}')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's see mortality rate by country"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"top_country_df = country_df.query('(date == @target_date) & (confirmed > 100)')\ntop_country_df['mortality_rate'] = top_country_df['fatalities'] / top_country_df['confirmed']\ntop_country_df = top_country_df.sort_values('mortality_rate', ascending=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Italy was the most serious situation, whose mortality rate is over 10% as of 2020/3/28.<br/>\nAs of June, we can see trend that mortality rate is high in Europe region.<br/>\nWe can also find countries from all over the world when we see top mortality rate countries.<br/>\nIran/Iraq from Middle East, Phillipines & Indonesia from tropical areas.<br/>\nSpain, Netherlands, France, and UK form Europe etc. It shows this coronavirus is really world wide pandemic.\n\n[UPDATE]: According to the comment by @elettra84, 10% of Italy is due to extreme outlier of Lombardy cluster. Except that mortality rate in Italy is comparable to other country. Refer [Lombardy cluster in wikipedia](https://en.wikipedia.org/wiki/2020_coronavirus_pandemic_in_Italy#Lombardy_cluster)."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.bar(top_country_df[:30].iloc[::-1],\n             x='mortality_rate', y='country',\n             title=f'Mortality rate HIGH: top 30 countries on {target_date}', text='mortality_rate', height=800, orientation='h')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"How about the countries whose mortality rate is low?\nMany Asian, Middle-East area is on the figure.\n\nBy investigating the difference between above & below countries, we might be able to figure out what is the cause which leads death.<br/>\nBe careful that there may be a case that these country's mortality rate is low due to these country does not report/measure fatality cases properly."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.bar(top_country_df[-30:],\n             x='mortality_rate', y='country',\n             title=f'Mortality rate LOW: top 30 countries on {target_date}', text='mortality_rate', height=800, orientation='h')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see number of confirmed cases on map. Again we can see Europe, US, MiddleEast (Turkey, Iran) and Asia (China, Korea) are red."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"all_country_df = country_df.query('date == @target_date')\nall_country_df['confirmed_log1p'] = np.log10(all_country_df['confirmed'] + 1)\nall_country_df['fatalities_log1p'] = np.log10(all_country_df['fatalities'] + 1)\nall_country_df['mortality_rate'] = all_country_df['fatalities'] / all_country_df['confirmed']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.choropleth(all_country_df, locations=\"country\", \n                    locationmode='country names', color=\"confirmed_log1p\", \n                    hover_name=\"country\", hover_data=[\"confirmed\", 'fatalities', 'mortality_rate'],\n                    range_color=[all_country_df['confirmed_log1p'].min(), all_country_df['confirmed_log1p'].max()], \n                    color_continuous_scale=\"peach\", \n                    title='Countries with Confirmed Cases')\n\n# I'd like to update colorbar to show raw values, but this does not work somehow...\n# Please let me know if you know how to do this!!\ntrace1 = list(fig.select_traces())[0]\ntrace1.colorbar = go.choropleth.ColorBar(\n    tickvals=[0, 1, 2, 3, 4, 5],\n    ticktext=['1', '10', '100', '1000','10000', '10000'])\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When we see mortality rate on map, we see Europe (especaiily Italy) is high. Also we notice MiddleEast (Iran, Iraq) is high.\n\nWhen we see tropical area, I wonder why Phillipines and Indonesia are high while other countries (Malaysia, Thai, Vietnam, as well as Australia) are low.\n\nFor Asian region, Korea's mortality rate is lower than China or Japan, I guess this is due to the fact that number of inspection is quite many in Korea.\nPlease refer these blogs for detail:\n\n - [South Korea launches 'drive-thru' coronavirus testing facilities as demand soars](https://www.japantimes.co.jp/news/2020/03/01/asia-pacific/science-health-asia-pacific/south-korea-drive-thru-coronavirus/#.XoAmw4j7RPY)\n - [Coronavirus: Why Japan tested so few people](https://asia.nikkei.com/Spotlight/Coronavirus/Coronavirus-Why-Japan-tested-so-few-people)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"\nfig = px.choropleth(all_country_df, locations=\"country\", \n                    locationmode='country names', color=\"fatalities_log1p\", \n                    hover_name=\"country\", range_color=[0, 4],\n                    hover_data=['confirmed', 'fatalities', 'mortality_rate'],\n                    color_continuous_scale=\"peach\", \n                    title='Countries with fatalities')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Mortality rate map, seems mortality rate is especially high in Europe region, compared to US or Asia."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = px.choropleth(all_country_df, locations=\"country\", \n                    locationmode='country names', color=\"mortality_rate\", \n                    hover_name=\"country\", range_color=[0, 0.12], \n                    color_continuous_scale=\"peach\", \n                    title='Countries with mortality rate')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Why mortality rate is different among country? What kind of hint is hidden in this map? Especially mortality rate is high in Europe and US, is there some reasons?\n\nThere is one interesting hypothesis that BCG vaccination<br/>\nThe below figure shows BCG vaccination policy by country: Advanced countries like Europe & US, especially Italy and US does not take BCG vaccination. We can notice this map is indeed similar to mortality rate map above. Is it just accidental?\n\n![](https://www.researchgate.net/profile/Alice_Zwerling/publication/50892386/figure/fig2/AS:277209752326147@1443103363144/Map-displaying-BCG-vaccination-policy-by-country-A-The-country-currently-has-universal.png)\n\nReference: [If I were North American/West European/Australian, I will take BCG vaccination now against the novel coronavirus pandemic.](https://www.jsatonotes.com/2020/03/if-i-were-north-americaneuropeanaustral.html)\n\n - [Australia's Trialing a TB Vaccine Against COVID-19, And Health Workers Get It First](https://www.sciencealert.com/australia-is-trialling-a-tb-vaccine-for-coronavirus-and-health-workers-get-it-first)\n\nOf course this is just one hypothesis but we can notice/find some hints to tackle Coronavirus like this by carefully analyzing/comparing the data."},{"metadata":{},"cell_type":"markdown","source":"The figure showing fatality growth since 10 deaths.\n - Ref: [COVID-19 Deaths Per Capita](https://covid19dashboards.com/covid-compare-permillion/)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"n_countries = 20\nn_start_death = 10\nfatality_top_countires = top_country_df.sort_values('fatalities', ascending=False).iloc[:n_countries]['country'].values\ncountry_df['date'] = pd.to_datetime(country_df['date'])\n\n\ndf_list = []\nfor country in fatality_top_countires:\n    this_country_df = country_df.query('country == @country')\n    start_date = this_country_df.query('fatalities > @n_start_death')['date'].min()\n    this_country_df = this_country_df.query('date >= @start_date')\n    this_country_df['date_since'] = this_country_df['date'] - start_date\n    this_country_df['fatalities_log1p'] = np.log10(this_country_df['fatalities'] + 1)\n    this_country_df['fatalities_log1p'] -= this_country_df['fatalities_log1p'].values[0]\n    df_list.append(this_country_df)\n\ntmpdf = pd.concat(df_list)\ntmpdf['date_since_days'] = tmpdf['date_since'] / pd.Timedelta('1 days')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.line(tmpdf,\n              x='date_since_days', y='fatalities_log1p', color='country',\n              title=f'Fatalities by country since 10 deaths, as of {target_date}')\nfig.add_trace(go.Scatter(x=[0, 28], y=[0, 4], name='Double by 7 days', line=dict(dash='dash', color=('rgb(200, 200, 200)'))))\nfig.add_trace(go.Scatter(x=[0, 56], y=[0, 4], name='Double by 14 days', line=dict(dash='dash', color=('rgb(200, 200, 200)'))))\nfig.add_trace(go.Scatter(x=[0, 84], y=[0, 4], name='Double by 21 days', line=dict(dash='dash', color=('rgb(200, 200, 200)'))))\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Sudden increase at China on the days 85 may be because updated reporting in Wuhan:\n\n - [Coronavirus: China outbreak city Wuhan raises death toll by 50%](https://www.bbc.com/news/world-asia-china-52321529)"},{"metadata":{},"cell_type":"markdown","source":"## Daily NEW confirmed cases trend\n\nHow about **DAILY new cases** trend?<br/>\nWe find from below figure:\n - China has finished its peak at Feb 14, new confirmed cases are surpressed now.\n - Europe&US spread starts on mid of March, after China slows down.\n - I feel effect of lock down policy in Europe (Italy, Spain, Germany, France) now comes on the figure,\n   the number of new cases are not so increasing rapidly at the end of March.\n - Current US new confirmed cases are the worst speed, recording worst speed at more than 30k people/day at peak. <b>Daily new confirmed cases start to decrease from April 4 or April 10, I hope this trend will continue.</b>.\n   - <span style=\"color:red\"><b>After that we can see a weekly trend that the confirmed cases becomes small on Monday. I think this is because people don't (or cannot) get medical care on Sunday so its reporting number is low on Sunday or Monday</b></span>\n - From August, India is the top. U.S & Brazil's numbers are start decreasing."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"country_df['prev_confirmed'] = country_df.groupby('country')['confirmed'].shift(1)\ncountry_df['new_case'] = country_df['confirmed'] - country_df['prev_confirmed']\ncountry_df['new_case'].fillna(0, inplace=True)\ntop30_country_df = country_df[country_df['country'].isin(top30_countries)]\n\nfig = px.line(top30_country_df,\n              x='date', y='new_case', color='country',\n              title=f'DAILY NEW Confirmed cases by country')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"country_df['prev_new_case'] = country_df.groupby('country')['new_case'].shift(1)\ncountry_df['growth_factor'] = country_df['new_case'] / country_df['prev_new_case']\ncountry_df['growth_factor'].fillna(0, inplace=True)\ntop30_country_df = country_df[country_df['country'].isin(top30_countries)]\n\nfig = px.line(top30_country_df,\n              x='date', y='growth_factor', color='country',\n              title=f'Growth factor by country')\nfig.add_trace(go.Scatter(x=[ww_df['date'].min(), ww_df['date'].max()], y=[1., 1.], name='Growth factor=1.', line=dict(dash='dash', color=('rgb(255, 0, 0)'))))\nfig.update_yaxes(range=[0., 5.])\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Geographical animation: spready by date\n\nYou can see animation how confirmed cases spread over time, you can see trend moving to China -> Europe -> US."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"country_df['date'] = country_df['date'].apply(str)\ncountry_df['confirmed_log1p'] = np.log1p(country_df['confirmed'])\ncountry_df['fatalities_log1p'] = np.log1p(country_df['fatalities'])\n\nfig = px.scatter_geo(country_df, locations=\"country\", locationmode='country names', \n                     color=\"confirmed\", size='confirmed', hover_name=\"country\", \n                     hover_data=['confirmed', 'fatalities'],\n                     range_color= [0, country_df['confirmed'].max()], \n                     projection=\"natural earth\", animation_frame=\"date\", \n                     title='COVID-19: Confirmed cases spread Over Time', color_continuous_scale=\"portland\")\n# fig.update(layout_coloraxis_showscale=False)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can see animation how confirmed cases spread over time, you can see trend moving to China -> Europe -> US. But Europe is worse than US for number of fatalities now."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.scatter_geo(country_df, locations=\"country\", locationmode='country names', \n                     color=\"fatalities\", size='fatalities', hover_name=\"country\", \n                     hover_data=['confirmed', 'fatalities'],\n                     range_color= [0, country_df['fatalities'].max()], \n                     projection=\"natural earth\", animation_frame=\"date\", \n                     title='COVID-19: Fatalities growth Over Time', color_continuous_scale=\"portland\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"New cases trend: it looks like China is almost converged now."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"country_df.loc[country_df['new_case'] < 0, 'new_case'] = 0.\nfig = px.scatter_geo(country_df, locations=\"country\", locationmode='country names', \n                     color=\"new_case\", size='new_case', hover_name=\"country\", \n                     hover_data=['confirmed', 'fatalities'],\n                     range_color= [0, country_df['new_case'].max()], \n                     projection=\"natural earth\", animation_frame=\"date\", \n                     title='COVID-19: Daily NEW cases over Time', color_continuous_scale=\"portland\")\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id_province\"></a>\n# Going into province"},{"metadata":{},"cell_type":"markdown","source":"How many country has precise province information?<br/>\nIt seems it's 8 countries: Australia, Canada, China, Denmark, France, Netherlands, US, and UK."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"for country in countries:\n    province = train.query('country == @country')['province'].unique()\n    if len(province) > 1:       \n        print(f'Country {country} has {len(province)} provinces: {province}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id_us\"></a>\n# Zoom up to US: what is happening in US now??\n\nAs we can see, the spread is fastest in US now, at the end of March. Let's see in detail what is going on in US."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"usa_state_code_df = pd.read_csv('/kaggle/input/usa-state-code/usa_states2.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_us","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Prepare data frame only for US. \n\n#train_us = train.query('country == \"US\"')\ntrain_us['mortality_rate'] = train_us['fatalities'] / train_us['confirmed']\n\n# Convert province column to its 2-char code name,\nstate_name_to_code = dict(zip(usa_state_code_df['state_name'], usa_state_code_df['state_code']))\ntrain_us['province_code'] = train_us['province'].map(state_name_to_code)\n\n# Only show latest days.\ntrain_us_latest = train_us.query('date == @target_date')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**[Situation in April]**<br/>\nWhen we see inside of the US, **only New York, and its neighbor New Jersey** dominated its spread and are in serious situation around April.<br/>\nNew York confirmed cases is over 50k, while other states are less than about 5k confirmed cases around April.\n\n**[Situation in July]**<br/>\nThe high speed coronavirus spread does not decrease in U.S.. Now it affected to many states, especially **California, Texas, Florida**."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.choropleth(train_us_latest, locations='province_code', locationmode=\"USA-states\",\n                    color='confirmed', scope=\"usa\", hover_data=['province', 'fatalities', 'mortality_rate'],\n                    title=f'Confirmed cases in US on {target_date}')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Mortality rate in New York seems not high, around 2% for now."},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"train_us_latest.sort_values('confirmed', ascending=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**[Situation in July]**<br/>\nThe mortality rate is high only around New York & New Jersey, this may be because first wave hit these areas and medical care was not enough in March.\n\nWhile the mortality rate is relatively low in California, Texas, Florida. These states provides proper medical care?"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.choropleth(train_us_latest, locations='province_code', locationmode=\"USA-states\",\n                    color='mortality_rate', scope=\"usa\", hover_data=['province', 'fatalities', 'mortality_rate'],\n                    title=f'Mortality rate in US on {target_date}')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Daily growth**: All state is US got affected from middle of March, and now **growing exponentially**.\nIn New York, less than 1k people are confirmed on March 16, but more than 50k people are confirmed on March 30. **50 times explosion in 2 weeks!**\n\nNow in July, we need to be carefully watch the situation in **California, Texas, Florida**, the number is increasing exponentially in these states now."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_us_march = train_us.query('date > \"2020-03-01\"')\nfig = px.line(train_us_march,\n              x='date', y='confirmed', color='province',\n              title=f'Confirmed cases by state in US, as of {target_date}')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_us_march['prev_confirmed'] = train_us_march.groupby('province')['confirmed'].shift(1)\ntrain_us_march['new_case'] = train_us_march['confirmed'] - train_us_march['prev_confirmed']\ntrain_us_march['new_case'].fillna(0, inplace=True)\n\nfig = px.line(train_us_march,\n              x='date', y='new_case', color='province',\n              title=f'DAILY NEW Confirmed cases by states in US')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_us_march['prev_new_case'] = train_us_march.groupby('province')['new_case'].shift(1)\ntrain_us_march['growth_factor'] = train_us_march['new_case'] / train_us_march['prev_new_case']\ntrain_us_march['growth_factor'].fillna(0, inplace=True)\nfig = px.line(train_us_march,\n              x='date', y='growth_factor', color='province',\n              title=f'Growth factor by state in US')\nfig.add_trace(go.Scatter(x=[train_us_march['date'].min(), train_us_march['date'].max()], y=[1., 1.],\n                         name='Growth factor=1.', line=dict(dash='dash', color=('rgb(255, 0, 0)'))))\nfig.update_yaxes(range=[0., 5.])\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id_europe\"></a>\n# Europe"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Ref: https://www.kaggle.com/abhinand05/covid-19-digging-a-bit-deeper\neurope_country_list =list([\n    'Austria','Belgium','Bulgaria','Croatia','Cyprus','Czechia','Denmark','Estonia','Finland','France','Germany','Greece','Hungary','Ireland',\n    'Italy', 'Latvia','Luxembourg','Lithuania','Malta','Norway','Netherlands','Poland','Portugal','Romania','Slovakia','Slovenia',\n    'Spain', 'Sweden', 'United Kingdom', 'Iceland', 'Russia', 'Switzerland', 'Serbia', 'Ukraine', 'Belarus',\n    'Albania', 'Bosnia and Herzegovina', 'Kosovo', 'Moldova', 'Montenegro', 'North Macedonia'])\n\ncountry_df['date'] = pd.to_datetime(country_df['date'])\ntrain_europe = country_df[country_df['country'].isin(europe_country_list)]\n#train_europe['date_str'] = pd.to_datetime(train_europe['date'])\ntrain_europe_latest = train_europe.query('date == @target_date')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When we look into the Europe, its Northern & Eastern areas are relatively better situation compared to Eastern & Southern areas."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.choropleth(train_europe_latest, locations=\"country\", \n                    locationmode='country names', color=\"confirmed\", \n                    hover_name=\"country\", range_color=[1, train_europe_latest['confirmed'].max()], \n                    color_continuous_scale='portland', \n                    title=f'European Countries with Confirmed Cases as of {target_date}', scope='europe', height=800)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Especially **Italy, Spain, German, France, UK** are in more serious situation.\n\nNumber of confirmed cases rapidly increasing in **Russia now (as of May 1)**, and **Russia** is in much dangerous situation as of July..."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_europe_march = train_europe.query('date >= \"2020-03-01\"')\nfig = px.line(train_europe_march,\n              x='date', y='confirmed', color='country',\n              title=f'Confirmed cases by country in Europe, as of {target_date}')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**UK's fatality number is growing and becomes top-1 on May 5th, followed by Italy, France and Spain.**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = px.line(train_europe_march,\n              x='date', y='fatalities', color='country',\n              title=f'Fatalities by country in Europe, as of {target_date}')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When we check daily new cases in Europe, we notice:\n\n - **UK and Russia** daily growth are more than Italy now, These countries are potentially more dangerous now.\n - Italy new cases are not increasing since March 21, I guess due to lock-down policy is started working. That is not a bad news.\n - We can see **big second wave coming to Spain & France from August**."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_europe_march['prev_confirmed'] = train_europe_march.groupby('country')['confirmed'].shift(1)\ntrain_europe_march['new_case'] = train_europe_march['confirmed'] - train_europe_march['prev_confirmed']\nfig = px.line(train_europe_march,\n              x='date', y='new_case', color='country',\n              title=f'DAILY NEW Confirmed cases by country in Europe')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_europe_march['prev_new_case'] = train_europe_march.groupby('country')['new_case'].shift(1)\ntrain_europe_march['growth_factor'] = train_europe_march['new_case'] / train_europe_march['prev_new_case']\ntrain_europe_march['growth_factor'].fillna(0, inplace=True)\nfig = px.line(train_europe_march,\n              x='date', y='growth_factor', color='country',\n              title=f'Growth factor by country in Europe')\nfig.add_trace(go.Scatter(x=[train_europe_march['date'].min(), train_europe_march['date'].max()], y=[1., 1.],\n                         name='Growth factor=1.', line=dict(dash='dash', color=('rgb(255, 0, 0)'))))\nfig.update_yaxes(range=[0., 5.])\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id_asia\"></a>\n# Asia"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"**[April]**<br/>\nIn Asia, China & Iran have many confirmed cases, followed by South Korea & Turkey. \n\n**[July]**<br/>\n\nWhen I notice, coronavirus original place China has relatively few confirmed cases compared to the other seriously affected countries (it might be because China does not report all the confirmed cases).<br/>\n**South Asia, including India, Bangladesh, Pakistan, Iran, Saudi Arabia, Turkey** was more affected now."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"country_latest = country_df.query('date == @target_date')\n\nfig = px.choropleth(country_latest, locations=\"country\", \n                    locationmode='country names', color=\"confirmed\", \n                    hover_name=\"country\", range_color=[1, 300000], \n                    color_continuous_scale='portland', \n                    title=f'Asian Countries with Confirmed Cases as of {target_date}', scope='asia', height=800)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The coronavirus hit Asia in early phase, how is the situation now?<br/>\nChina & Korea is already in decreasing phase.<br/>\n\nUnlike China or Korea, daily new confirmed cases were kept increasing on March or April, especially in Iran or Japan. But the number is started to decrease now on these country as well now.\n\nThe number increases exponentially in **India** as of July, need to be careful."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"top_asian_country_df = country_df[country_df['country'].isin([\n    'China', 'Indonesia', 'Iran', 'Japan', 'Korea, South', 'Malaysia', 'Philippines',\n    'India', 'Bangladesh', 'Pakistan', 'Saudi Arabia', 'Turkey'\n])]\n\nfig = px.line(top_asian_country_df,\n              x='date', y='new_case', color='country',\n              title=f'DAILY NEW Confirmed cases Asia')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"top_asian_country_df['prev_new_case'] = top_asian_country_df.groupby('country')['new_case'].shift(1)\ntop_asian_country_df['growth_factor'] = top_asian_country_df['new_case'] / top_asian_country_df['prev_new_case']\ntop_asian_country_df['growth_factor'].fillna(0, inplace=True)\nfig = px.line(top_asian_country_df,\n              x='date', y='growth_factor', color='country',\n              title=f'Growth factor by country in Asia')\nfig.add_trace(go.Scatter(x=[top_asian_country_df['date'].min(), top_asian_country_df['date'].max()], y=[1., 1.],\n                         name='Growth factor=1.', line=dict(dash='dash', color=('rgb(255, 0, 0)'))))\nfig.update_yaxes(range=[0., 5.])\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id_recover\"></a>\n# Which country is recovering now?"},{"metadata":{},"cell_type":"markdown","source":"We saw that Coronavirus now hits Europe & US, in serious situation. How does it converge?\n\nWe can refer other country where confirmed cases is already decreasing.<br/>\nHere I defined `new_case_peak_to_now_ratio`, as a ratio of current new case and the max new case for each country.<br/>\nIf new confirmed case is biggest now, its ratio is 1.\nIts ratio is expected to be low value for the countries where the peak has already finished.  "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"max_confirmed = country_df.groupby('country')['new_case'].max().reset_index()\ncountry_latest = pd.merge(country_latest, max_confirmed.rename({'new_case': 'max_new_case'}, axis=1))\ncountry_latest['new_case_peak_to_now_ratio'] = country_latest['new_case'] / country_latest['max_new_case']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"recovering_country = country_latest.query('new_case_peak_to_now_ratio < 0.5')\nmajor_recovering_country = recovering_country.query('confirmed > 100')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The ratio is 0 for the country with very few confirmed cases are reported.<br/>\nI choosed the countries with its confirmed cases more than 100, to see only major countries with the ratio is low.\n\nWe can see:\n - Middle East coutnries.\n - South Africa countries.\n - China & Korea from Asia."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.bar(major_recovering_country.sort_values('new_case_peak_to_now_ratio', ascending=False),\n             x='new_case_peak_to_now_ratio', y='country',\n             title=f'Mortality rate LOW: top 30 countries on {target_date}', text='new_case_peak_to_now_ratio', height=1000, orientation='h')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see by map. Yellow countries have high ratio, currently increasing countries. **Blue & purple countries** have low ratio, already decreasing countries from its peak.\n\n[At July 2]: We can see that the number is still increasing in **US, South America, South Asia & Africa**, while we can see converging trend in **Europe & East Asia (around China)**."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.choropleth(country_latest, locations=\"country\", \n                    locationmode='country names', color=\"new_case_peak_to_now_ratio\", \n                    hover_name=\"country\", range_color=[0, 1], \n                    # color_continuous_scale=\"peach\", \n                    hover_data=['confirmed', 'fatalities', 'new_case', 'max_new_case'],\n                    title='Countries with new_case_peak_to_now_ratio')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see some recovering countries.\n\n## China\n\nWhen we check each state stats, we can see Hubei, the starting place, is extremely large number of confirmed cases.<br/>\nOther states records actually few confirmed cases compared to Hubei."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"china_df = train.query('country == \"China\"')\nchina_df['prev_confirmed'] = china_df.groupby('province')['confirmed'].shift(1)\nchina_df['new_case'] = china_df['confirmed'] - china_df['prev_confirmed']\nchina_df.loc[china_df['new_case'] < 0, 'new_case'] = 0.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.line(china_df,\n              x='date', y='new_case', color='province',\n              title=f'DAILY NEW Confirmed cases in China by province')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The situation of Hubei now?\n\nHubei record its new case peak on Feb 14. And finally, new case was not found on March 19.\n\nTo become no new case found, it took **about 2month after confirmed cases occured**, and **1 month after the peak has reached.** <br/>\nThis term will be the reference for other country to how long we must lock-down the city.\n\nThe anomally number of 325 on Apr 17 is maybe due to the news that China admits the reporting number was low, so its number is not fatalities on this date, but past fatalities with not reported until now."},{"metadata":{"trusted":true},"cell_type":"code","source":"china_df.query('(province == \"Hubei\") & (date > \"2020-03-10\")')[['country_province', 'date', 'confirmed', 'fatalities', 'recovered', 'new_case']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id_converge\"></a>\n# When will it converge? - Estimation by sigmoid fitting\n\nI guess everyone is wondering when the coronavirus converges. Let's estimate it roughly using sigmoid fitting.<br/>\nI referenced below kernels for original ideas.\n\n - [Sigmoid per country](https://www.kaggle.com/group16/sigmoid-per-country-no-leakage) by @group16\n - [COVID-19 growth rates per country](https://www.kaggle.com/mikestubna/covid-19-growth-rates-per-country) by @mikestubna\n \n \n[At July]: Now I am feeling coronavirus does not converge in this year... I expand prediction term until the end of this year."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def sigmoid(t, M, beta, alpha, offset=0):\n    alpha += offset\n    return M / (1 + np.exp(-beta * (t - alpha)))\n\ndef error(x, y, params):\n    M, beta, alpha = params\n    y_pred = sigmoid(x, M, beta, alpha)\n\n    # apply weight, latest number is more important than past.\n    weight = np.arange(len(y_pred)) ** 2\n    loss_mse = np.mean((y_pred - y) ** 2 * weight)\n    return loss_mse\n\ndef gen_random_color(min_value=0, max_value=256) -> str:\n    \"\"\"Generate random color for plotly\"\"\"\n    r, g, b = np.random.randint(min_value, max_value, 3)\n    return f'rgb({r},{g},{b})'","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def fit_sigmoid(exclude_days=0):\n    target_country_df_list = []\n    pred_df_list = []\n    for target_country in top30_countries:\n        print('target_country', target_country)\n        # --- Train ---\n        target_country_df = country_df.query('country == @target_country')\n\n        #train_start_date = target_country_df['date'].min()\n        train_start_date = target_country_df.query('confirmed > 1000')['date'].min()\n        train_end_date = pd.to_datetime(target_date) - pd.Timedelta(f'{exclude_days} days')\n        target_date_df = target_country_df.query('(date >= @train_start_date) & (date <= @train_end_date)')\n        if len(target_date_df) <= 7:\n            print('WARNING: the data is not enough, use 7 more days...')\n            train_start_date -= pd.Timedelta('7 days')\n            target_date_df = target_country_df.query('(date >= @train_start_date) & (date <= @train_end_date)')\n\n        confirmed = target_date_df['confirmed'].values\n        x = np.arange(len(confirmed))\n\n        lossfun = lambda params: error(x, confirmed, params)\n        res = sp.optimize.minimize(lossfun, x0=[np.max(confirmed) * 5, 0.04, 2 * len(confirmed) / 3.], method='nelder-mead')\n        M, beta, alpha = res.x\n        # sigmoid_models[key] = (M, beta, alpha)\n        # np.clip(sigmoid(list(range(len(data), len(data) + steps)), M, beta, alpha), 0, None).astype(int)\n\n        # --- Pred ---\n        pred_start_date = target_country_df['date'].min()\n        pred_end_date = pd.to_datetime('2021-01-01')\n        days = int((pred_end_date - pred_start_date) / pd.Timedelta('1 days'))\n        # print('pred start', pred_start_date, 'end', pred_end_date, 'days', days)\n\n        x = np.arange(days)\n        offset = (train_start_date - pred_start_date) / pd.Timedelta('1 days')\n        print('train_start_date', train_start_date, 'offset', offset, 'params', M, beta, alpha)\n        y_pred = sigmoid(x, M, beta, alpha, offset=offset)\n        # target_country_df['confirmed_pred'] = y_pred\n\n        all_dates = [pred_start_date + np.timedelta64(x, 'D') for x in range(days)]\n        pred_df = pd.DataFrame({\n            'date': all_dates,\n            'country': target_country,\n            'confirmed_pred': y_pred,\n        })\n\n        target_country_df_list.append(target_country_df)\n        pred_df_list.append(pred_df)\n    return target_country_df_list, pred_df_list","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plot_sigmoid_fitting(target_country_df_list, pred_df_list, title=''):\n    n_countries = len(top30_countries)\n\n    # --- Plot ---\n    fig = go.Figure()\n\n    for i in range(n_countries):\n        target_country = top30_countries[i]\n        target_country_df = target_country_df_list[i]\n        pred_df = pred_df_list[i]\n        color = gen_random_color(min_value=20)\n        # Prediction\n        fig.add_trace(go.Scatter(\n            x=pred_df['date'], y=pred_df['confirmed_pred'],\n            name=f'{target_country}_pred',\n            line=dict(color=color, dash='dash')\n        ))\n\n        # Ground truth\n        fig.add_trace(go.Scatter(\n            x=target_country_df['date'], y=target_country_df['confirmed'],\n            mode='markers', name=f'{target_country}_actual',\n            line=dict(color=color),\n        ))\n    fig.update_layout(\n        title=title, xaxis_title='Date', yaxis_title='Confirmed cases')\n    fig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"target_country_df_list, pred_df_list = fit_sigmoid(exclude_days=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_sigmoid_fitting(target_country_df_list, pred_df_list, title='Sigmoid fitting with all latest data')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If we believe above curve, confirmed cases is slowing down now and it is almost converging in most of the country.<br/>\nBut it might take until beginning on June in US, and August in Brazil. Thus it still need long-term to converge in world wide. We have a risk of 2nd wave too.\n\nI'm not confident how this sigmoid fitting is accurate, it's just an estimation by some modeling.<br/>\nLet's try validation by excluding last 7 days data."},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"target_country_df_list, pred_df_list = fit_sigmoid(exclude_days=7)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_sigmoid_fitting(target_country_df_list, pred_df_list, title='Sigmoid fitting without last 7days data')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now I noticed that sigmoid fitting tend to **underestimate** the curve, and its actual value tend to be more than sigmoid curve estimation.<br/>\nTherefore, we need to be careful to see sigmoid curve fitting data, actual situation is likely to be worse than the previous figure trained with all data."},{"metadata":{},"cell_type":"markdown","source":"<a id=\"id_ref\"></a>\n# Further reading\n\nThat's all! Thank you for reading long kernel. I hope the world get back peace & usual daily life as soon as possible.\n\nHere are the other information for further reading.\n\nMy other kernels:\n - [COVID-19: Effect of temperature/humidity](https://www.kaggle.com/corochann/covid-19-effect-of-temperature-humidity)\n - [COVID-19: Spread situation by prefecture in Japan](https://www.kaggle.com/corochann/covid-19-spread-situation-by-prefecture-in-japan)"},{"metadata":{},"cell_type":"markdown","source":"<h3 style=\"color:red\">If this kernel helps you, please upvote to keep me motivated :)<br>Thanks!</h3>"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}