Ch5_Satellite_Cloud.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 5 - Satellite Data in the Cloud\n",
    "\n",
    "This chapter gives a short introduction to accessing satellite data in the Cloud. It is more informative than practical, so you could jump straight to the examples in the next chapters. However, you might want some background if you want to modify the examples to get other data that interests you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "Satellite data and satellite-based data are available from many sources, commercial and public. Not all data are free, specially data products from private companies. \n",
    "\n",
    "### Many data providers, including NASA, NOAA, and Copernicus share their data publicly and free. Including in the Cloud.\n",
    "\n",
    "## Today, an increasing portion of these data is stored in two accesible Cloud providers: [Amazon (AWS)](https://registry.opendata.aws/) and [Google (Earth Engine)](https://developers.google.com/earth-engine/datasets)\n",
    " \n",
    "Also, data comes in different formats.\n",
    "\n",
    "We will use data from AWS because of data is stored in a format that is easily analyzed using `Python` and `xarray`, while Google uses its own interface (worth checking out, though)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "***\n",
    "Historically, satellite data is stored as snapshot images. With time, as the satellite-era grow in length, the number of images available has grown to amazing levels. \n",
    "\n",
    "And while these data has become an incredible tool for interannual and longer analysis, accessing the data for temporal analysis requires accesing each time step file, which is cumbersome and time and resource consuming.\n",
    "\n",
    "### New formats, like `zarr` have been developed to address this issue and provide faster access the data, not only in the temporal axis. \n",
    "\n",
    "Not all data is in this format, but the number of data sets is increasing and we will take advantage of those that are available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "***\n",
    "### *In these tutorials we aim to facilitate acqusition of satellite-base data and their temporal analysis*\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## In the Ocean Example (Chapter 6), we use a NASA high resolution sea surface temperature ([MUR SST](https://registry.opendata.aws/mur/)) data product, which is stored in `zarr` format\n",
    "Acquiring the data is simple, requiring only few lines of code. It is a long process, however, given the high resolution - in space and times - of the data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## In the Atmopshere Example (Chapter 7), we use a Copernicous satellite-based reanalysis product that most earth scientists are familiar with: [ECMWF/ERA-5](https://registry.opendata.aws/ecmwf-era5/)\n",
    "These data is stored in the cloud, in `zarr` format, but due to its volume, it is saved as monthly files. So, each file has to be accessed individually for a region and time to be selected. Note that because the files are in the cloud, we are not downloading them - only the selected portion is.\n",
    "\n",
    "####  In the example, we'll analyze wind vectors, but this notebook is easily modified to analyze another variable contained in this data set. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## Finally, in the Land Example (Chapter 8), we use NASA's MODIS satellite data to examine changes in vegetation through time. \n",
    "\n",
    "This data is available through diferent names and providers, and it is available in the AWS cloud (not free to date), and Google Earth Engine. It is also available online directly through NASA or NOAA, and other private providers.\n",
    "\n",
    "*Therefore, in this chapter we exemplify how to use `python` and `xarray` to process files from an online server - in this case a thredds server. Not the cloud!*\n",
    "\n",
    "You'll notice that this is straight forward method, although a bit lenghty. However, this method needs a good and stable bandwith because it downloads every file and then select the area (instead of select in the cloud before downloading it). \n",
    "\n",
    "We expect this, and other data, to be available in the cloud in zarr format soon - for free - so its acquisition would be similar to data in previous chapters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "# Resources\n",
    "\n",
    "### Data:\n",
    "- [AWS Registry for Open Data](https://registry.opendata.aws/)\n",
    "- [Google Earth Engine Data](https://developers.google.com/earth-engine/datasets)\n",
    "- [NOAA Big Data Program](https://www.noaa.gov/organization/information-technology/list-of-big-data-program-datasets) Data list\n",
    "\n",
    "### Cloud resources:\n",
    "- [Pangeo](https://pangeo.io/cloud.html)\n",
    "- [Chameleon Cloud](https://www.chameleoncloud.org/)\n",
    "- [Datalore](https://datalore.jetbrains.com/)\n",
    "- [mybinder](https://mybinder.org/)\n",
    "- [pangeo binder](https://binder.pangeo.io/)\n",
    "\n",
    "### Libraries related to cloud access and formats used here:\n",
    "- [fsspec](https://filesystem-spec.readthedocs.io/en/latest/) filesystem interfaces for python\n",
    "- [zarr](https://zarr.readthedocs.io/en/stable/) Big data storage formatt\n",
    "- [dask](https://dask.org/) library to enable parallel processing for python\n",
    "\n",
    "### If you want to dig deeper:\n",
    "- [Pangeo tutorial for AGU OSM2020](https://github.com/pangeo-gallery/osm2020tutorial)\n",
    "- [Methods for accessing a AWS bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-bucket-intro.html). Bucket is the name of the cloud storage object. S3 stands for Amazon's Simple Storage Service.\n",
    "- [earthengine-api](https://github.com/google/earthengine-api/blob/master/python/examples/ipynb/Earth_Engine_REST_API_compute_table.ipynb) Use Python to access cloud data in the Google Earth Engine.\n",
    "- [satpy](https://github.com/pytroll/satpy) Python library to analyze satellite data\n",
    "- [pysat](https://github.com/pysat/pysat) A Python satellite data analysis toolkit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}