Merge branch 'main' into mamba-package-install

arup-group · Dec 19, 2023 · 14e428b · 14e428b
2 parents 9ca6cc2 + c57bcbc
commit 14e428b
Show file tree

Hide file tree

Showing 55 changed files with 359 additions and 568 deletions.
diff --git a/.gitignore b/.gitignore
@@ -54,6 +54,7 @@ coverage.xml
 *.cover
 .hypothesis/
 .pytest_cache/
+reports/
 
 # Translations
 *.mo
@@ -122,15 +123,5 @@ dmypy.json
 .pyre/
 .idea
 
-tests/test_outputs/
-test_chunks.csv
-reports/
-mprofile*
-benchmark-outputs/
-working-dir/
-scripts/secrets-mgr-exploration.py
-*.nbconvert.*
-example_data/pt2matsim_network/genet_output/*
-genet_output/*
-example_data/output_*/*
-example_data/api_requests_send.json
+# Project specific
+examples/example_data/outputs
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -65,14 +65,15 @@ automated `PEP8` checks and runs unit tests in a fresh environment, as well as i
 8. Update/add to or generate a new jupyter notebook in `notebooks` directory which takes the user through your new feature or
 change.
    1. Jupyter notebooks are closely linked to the [wiki pages](https://github.com/arup-group/genet/wiki).
-      1. Make sure you follow the naming convention: `number.number. Theme: Catchy Title`
-      2. Make sure you structure the notebook in a way that you would like to see it in a wiki page, with a lot of
+      1. Make sure you follow the naming convention to title your page: `number.number. Theme: Catchy Title`
+      2. Make sure to also name the notebook so that it is machine readable: `number_number_theme_catchy_title.ipynb`
+      3. Make sure you structure the notebook in a way that you would like to see it in a wiki page, with a lot of
       markdown cells containing quality descriptions. Use existing notebook as examples.
    2. After your changes have been merged, you may like to update the wiki pages. To do this in an automated way:
       1. Clone the wiki part of the repo: ```git clone https://github.com/arup-group/genet.wiki.git```
-      2. Run the `notebooks/generate_usage_wiki_from_notebooks.py` script, pointing at the folder containing the
+      2. Run the `examples/generate_usage_wiki_from_notebooks.py` script, pointing at the folder containing the
       notebooks and the wiki repo folder to receive the output.
-   3. You may use example data already in `example_data` directory of this repo, or add more (small amount of) data to
+   3. You may use example data already in `examples/example_data` directory of this repo, or add more (small amount of) data to
    it to show off your new features.
 9. Add section in the `README.md` which shows usage of your new feature. This can be paraphrased from the jupyter
 notebook in point above.

diff --git a/README.md b/README.md
@@ -160,19 +160,25 @@ We welcome community contributions to GeNet; please see our [guide to contributi
 If you are making changes to the codebase, you should use the tools described below to verify that the code still works.
 All of the following commands assume you are in the project's root directory.
 
-### Unit tests
+### Tests
+To run unit tests within genet python environment (including testing example notebooks):
 
-To run unit tests within genet python environment, run `pytest`.
-To improve test runtime you can focus only on the unit tests and not on the tests of the example jupyter notebooks: `pytest tests/`.
-To run tests in the Docker container, you will need to update the Dockerfile to include development dependencies and then you can run `docker run cml-genet pytest`.
+    pytest
+
+and within a docker container:
+
+    docker run cml-genet pytest -vv
+
+In either case, for shorter test runtimes, append 'tests/' to limit to the unit tests and ignore notebook tests.
 
 ### Generate a unit test code coverage report
 
 To generate an HTML coverage report at `reports/coverage/index.html`:
 
     pytest --cov-report=html
 
+Coverage will also be tracked in pull requests.
+
 ### Lint the python code
 
     Run `pre-commit install` to install pre-commit, which will lint and format your code whenever you commit staged changes.
-
diff --git a/example_data/api_requests_send.json b/example_data/api_requests_send.json
diff --git a/notebooks/1. Intro.ipynb → examples/1_intro_to_genet.ipynb b/notebooks/1. Intro.ipynb → examples/1_intro_to_genet.ipynb
@@ -9,12 +9,12 @@
     }
    },
    "source": [
-    "# Intro\n",
+    "# 1. Introduction to GeNet\n",
     "This section goes through basic usage examples. Available as a jupyter notebook and a wiki page.\n",
     "\n",
     "## `Network` data structure\n",
     "\n",
-    "![GeNet Network Graph diagram](https://github.com/arup-group/genet/blob/master/images/genet_network_graph.png)\n",
+    "![GeNet Network Graph diagram](../images/genet_network_graph.png)\n",
     "\n",
     "Main schema:\n",
     "* nodes: exist uniquely, hold spatial information.\n",
@@ -28,7 +28,7 @@
     "* nodes hold spatial information\n",
     "* each edge/link holds information such as mode of transport allowed, speed, capacity, length, OSM-inherited tags\n",
     "\n",
-    "![GeNet Schedule Graph diagram](https://github.com/arup-group/genet/blob/master/images/genet_schedule.png)\n",
+    "![GeNet Schedule Graph diagram](../images/genet_schedule.png)\n",
     "\n",
     "`Schedule` consists of:\n",
     "* List of `Service`s, where each `Service` has:\n",

diff --git a/notebooks/2.1. Reading Data: MATSim.ipynb → examples/2_1_reading_data_matsim.ipynb b/notebooks/2.1. Reading Data: MATSim.ipynb → examples/2_1_reading_data_matsim.ipynb
@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Reading a MATSim network\n",
+    "# 2.1. Reading data: MATSim networks\n",
     "\n",
     "This page goes through methods for reading in MATSim networks. Available as a jupyter notebook or a wiki page."
    ]
@@ -37,7 +37,7 @@
    },
    "outputs": [],
    "source": [
-    "path_to_matsim_network = \"../example_data/pt2matsim_network\"\n",
+    "path_to_matsim_network = \"example_data/pt2matsim_network\"\n",
     "\n",
     "network = os.path.join(path_to_matsim_network, \"network.xml\")\n",
     "schedule = os.path.join(path_to_matsim_network, \"schedule.xml\")\n",

diff --git a/notebooks/2.2. Reading Data: OSM.ipynb → examples/2_2_reading_data_osm.ipynb b/notebooks/2.2. Reading Data: OSM.ipynb → examples/2_2_reading_data_osm.ipynb
@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Reading OSM data\n",
+    "# 2.2. Reading data: OSM\n",
     "\n",
     "Reading OSM data can be configured. You will find example config in the configs directory. Available as a jupyter notebook or wiki page.\n",
     "\n",
@@ -158,7 +158,7 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "2022-07-14 15:27:55,774 - Building OSM graph from file ../example_data/example.osm\n",
+      "2022-07-14 15:27:55,774 - Building OSM graph from file example_data/example.osm\n",
       "2022-07-14 15:27:56,540 - Creating networkx graph from OSM data\n",
       "2022-07-14 15:27:56,541 - OSM: Extract Nodes and Paths from OSM data\n",
       "2022-07-14 15:27:56,898 - OSM: Add each OSM way (aka, path) to the OSM graph\n",
@@ -176,7 +176,7 @@
    ],
    "source": [
     "n = read_osm(\n",
-    "    \"../example_data/example.osm\",\n",
+    "    \"example_data/example.osm\",\n",
     "    \"../genet/configs/OSM/slim_config.yml\",\n",
     "    num_processes=2,\n",
     "    epsg=\"epsg:27700\",\n",

diff --git a/notebooks/2.3. Reading Data: GTFS.ipynb → examples/2_3_reading_data_gtfs.ipynb b/notebooks/2.3. Reading Data: GTFS.ipynb → examples/2_3_reading_data_gtfs.ipynb
@@ -4,11 +4,10 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Reading GTFS data\n",
+    "# 2.3. Reading data: GTFS\n",
     "\n",
-    "This page goes through methods for reading in GTFS ([General Transit Feed Specification](https://developers.google.com/transit/gtfs)). [Here](https://developers.google.com/transit/gtfs/reference) is the reference page for the schema of GTFS data. Available as a jupyter notebook or wiki page.\n",
-    "\n",
-    "Small sample of this data can be found in [tests/test_data/gtfs](https://github.com/arup-group/genet/tree/master/tests/test_data/gtfs).\n",
+    "This page goes through methods for reading in GTFS ([General Transit Feed Specification](https://developers.google.com/transit/gtfs)). \n",
+    "[Here](https://developers.google.com/transit/gtfs/reference) is the reference page for the schema of GTFS data.\n",
     "\n",
     "GeNet ingests zipped or unzipped GTFS feeds. The following files are required in the unzipped folder, or inside the zip file:\n",
     "- `calendar.txt`\n",
@@ -21,8 +20,8 @@
     "\n",
     "GeNet does support extraction of services/routes/stops in the output `genet.Schedule` object based on a geographical area (Methods: `services_on_spatial_condition`, `routes_on_spatial_condition`, `stops_on_spatial_condition`, more information in notebook on using genet Network), but you might like to use [gtfs-lib](https://github.com/conveyal/gtfs-lib) prior to ingestion in GeNet.\n",
     "\n",
-    "The user assumes responsibility for the quality of their input GTFS feed. There are various validation tools that can \n",
-    "be used with GTFS feeds before using with GeNet, see this page for a summary of [validation tools](https://gtfs.org/testing/)."
+    "The user assumes responsibility for the quality of their input GTFS feed. \n",
+    "There are various validation tools that can be used with GTFS feeds before using with GeNet, see [this page](https://gtfs.org/testing/) for a summary of validation tools."
    ]
   },
   {
@@ -60,7 +59,7 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "2022-07-14 15:27:50,737 - Reading GTFS from ../example_data/example_gtfs\n",
+      "2022-07-14 15:27:50,737 - Reading GTFS from example_data/example_gtfs\n",
       "2022-07-14 15:27:50,740 - Reading the calendar for GTFS\n",
       "2022-07-14 15:27:50,743 - Reading GTFS data into usable format\n",
       "2022-07-14 15:27:50,745 - Reading stop times\n",
@@ -71,7 +70,7 @@
     }
    ],
    "source": [
-    "s = read_gtfs(\"../example_data/example_gtfs\", \"20190603\")"
+    "s = read_gtfs(\"example_data/example_gtfs\", \"20190603\")"
    ]
   },
   {

diff --git a/notebooks/2.4. Reading Data: CSV.ipynb → examples/2_4_reading_data_csv.ipynb b/notebooks/2.4. Reading Data: CSV.ipynb → examples/2_4_reading_data_csv.ipynb
@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Reading a network from CSV\n",
+    "# 2.4. Reading data: CSV\n",
     "Available as a jupyter notebook or wiki page.\n",
     "\n",
     "You can read a network from node and link CSV files.\n",
@@ -39,8 +39,8 @@
    },
    "outputs": [],
    "source": [
-    "csv_nodes = \"../example_data/example_csv/nodes.csv\"\n",
-    "csv_links = \"../example_data/example_csv/links.csv\""
+    "csv_nodes = \"example_data/example_csv/nodes.csv\"\n",
+    "csv_links = \"example_data/example_csv/links.csv\""
    ]
   },
   {
@@ -57,8 +57,8 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "2022-07-14 15:30:48,943 - Reading nodes from ../example_data/example_csv/nodes.csv\n",
-      "2022-07-14 15:30:48,949 - Reading links from ../example_data/example_csv/nodes.csv\n",
+      "2022-07-14 15:30:48,943 - Reading nodes from example_data/example_csv/nodes.csv\n",
+      "2022-07-14 15:30:48,949 - Reading links from example_data/example_csv/nodes.csv\n",
       "2022-07-14 15:30:49,024 - Added 4 nodes\n",
       "2022-07-14 15:30:49,034 - Added 2 links\n"
      ]

diff --git a/...s/2.5. Reading Data: JSON & GeoJSON.ipynb → examples/2_5_reading_data_json_geojson.ipynb b/...s/2.5. Reading Data: JSON & GeoJSON.ipynb → examples/2_5_reading_data_json_geojson.ipynb
@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Reading JSON & GeoJSON data\n",
+    "# 2.5. Reading data: JSON & GeoJSON\n",
     "Available as a jupyter notebook or wiki page.\n",
     "\n",
     "GeNet lets you read JSON data into a Networn graph and PT Schedule, or GeoJSON into Network Graph. \n",
@@ -74,14 +74,14 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "2022-07-14 15:31:04,968 - Reading Network from ../example_data/example_json/network.json\n",
+      "2022-07-14 15:31:04,968 - Reading Network from example_data/example_json/network.json\n",
       "2022-07-14 15:31:05,058 - Added 4 nodes\n",
       "2022-07-14 15:31:05,067 - Added 2 links\n"
      ]
     }
    ],
    "source": [
-    "n = read_json_network(\"../example_data/example_json/network.json\", \"epsg:27700\")"
+    "n = read_json_network(\"example_data/example_json/network.json\", \"epsg:27700\")"
    ]
   },
   {
@@ -211,12 +211,12 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "2022-07-14 15:31:05,075 - Reading Schedule from ../example_data/example_json/schedule.json\n"
+      "2022-07-14 15:31:05,075 - Reading Schedule from example_data/example_json/schedule.json\n"
      ]
     }
    ],
    "source": [
-    "s = read_json_schedule(\"../example_data/example_json/schedule.json\", \"epsg:27700\")"
+    "s = read_json_schedule(\"example_data/example_json/schedule.json\", \"epsg:27700\")"
    ]
   },
   {
@@ -240,17 +240,17 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "2022-07-14 15:31:05,119 - Reading Network from ../example_data/example_json/network.json\n",
+      "2022-07-14 15:31:05,119 - Reading Network from example_data/example_json/network.json\n",
       "2022-07-14 15:31:05,281 - Added 4 nodes\n",
       "2022-07-14 15:31:05,354 - Added 2 links\n",
-      "2022-07-14 15:31:05,360 - Reading Schedule from ../example_data/example_json/schedule.json\n"
+      "2022-07-14 15:31:05,360 - Reading Schedule from example_data/example_json/schedule.json\n"
      ]
     }
    ],
    "source": [
     "n = read_json(\n",
-    "    network_path=\"../example_data/example_json/network.json\",\n",
-    "    schedule_path=\"../example_data/example_json/schedule.json\",\n",
+    "    network_path=\"example_data/example_json/network.json\",\n",
+    "    schedule_path=\"example_data/example_json/schedule.json\",\n",
     "    epsg=\"epsg:27700\",\n",
     ")"
    ]
@@ -307,8 +307,8 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "2022-07-14 15:31:05,419 - Reading Network nodes from ../example_data/example_geojson/network_nodes.geojson\n",
-      "2022-07-14 15:31:05,520 - Reading Network links from ../example_data/example_geojson/network_links.geojson\n",
+      "2022-07-14 15:31:05,419 - Reading Network nodes from example_data/example_geojson/network_nodes.geojson\n",
+      "2022-07-14 15:31:05,520 - Reading Network links from example_data/example_geojson/network_links.geojson\n",
       "2022-07-14 15:31:05,613 - Added 4 nodes\n",
       "2022-07-14 15:31:05,622 - Added 2 links\n"
      ]
@@ -318,8 +318,8 @@
     "from genet import read_geojson_network\n",
     "\n",
     "n = read_geojson_network(\n",
-    "    \"../example_data/example_geojson/network_nodes.geojson\",\n",
-    "    \"../example_data/example_geojson/network_links.geojson\",\n",
+    "    \"example_data/example_geojson/network_nodes.geojson\",\n",
+    "    \"example_data/example_geojson/network_links.geojson\",\n",
     "    \"epsg:27700\",\n",
     ")"
    ]

diff --git a/notebooks/3.1. Writing Data: MATSim.ipynb → examples/3_1_writing_data_matsim.ipynb b/notebooks/3.1. Writing Data: MATSim.ipynb → examples/3_1_writing_data_matsim.ipynb
@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Writing `Network` to MATSim\n",
+    "# 3.1. Writing data: MATSim\n",
     "\n",
     "This page goes through the MATSim files that can be generated with GeNet. Available as a jupyter notebook or wiki page.\n",
     "\n",
@@ -72,7 +72,8 @@
     "\n",
     "from genet import read_matsim\n",
     "\n",
-    "path_to_matsim_network = \"../example_data/pt2matsim_network\"\n",
+    "path_to_matsim_network = \"example_data/pt2matsim_network\"\n",
+    "path_to_output = \"example_data/outputs/pt2matsim_network\"\n",
     "\n",
     "network = os.path.join(path_to_matsim_network, \"network.xml\")\n",
     "schedule = os.path.join(path_to_matsim_network, \"schedule.xml\")\n",
@@ -107,14 +108,14 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "2022-07-14 15:31:43,952 - Writing ../example_data/pt2matsim_network/genet_output/network.xml\n",
-      "2022-07-14 15:31:44,529 - Writing ../example_data/pt2matsim_network/genet_output/schedule.xml\n",
-      "2022-07-14 15:31:44,570 - Writing ../example_data/pt2matsim_network/genet_output/vehicles.xml\n"
+      "2022-07-14 15:31:43,952 - Writing example_data/pt2matsim_network/genet_output/network.xml\n",
+      "2022-07-14 15:31:44,529 - Writing example_data/pt2matsim_network/genet_output/schedule.xml\n",
+      "2022-07-14 15:31:44,570 - Writing example_data/pt2matsim_network/genet_output/vehicles.xml\n"
      ]
     }
    ],
    "source": [
-    "n.write_to_matsim(os.path.join(path_to_matsim_network, \"genet_output\"))"
+    "n.write_to_matsim(path_to_output)"
    ]
   },
   {
@@ -129,7 +130,7 @@
   "kernelspec": {
    "display_name": "genet",
    "language": "python",
-   "name": "genet"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {
@@ -141,7 +142,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.7"
+   "version": "3.11.5"
   },
   "latex_envs": {
    "LaTeX_envs_menu_present": true,