Skip to content

Commit

Permalink
♻️ support subset + simplify retrieval + template
Browse files Browse the repository at this point in the history
This commit builds on 1c16800 and:
- simplifies the retrieval (via git) to dynamically identify the full list of
  deployments. Notably it no longer relies on munging command output
- supports providing a subset of the deployments via an environment variable

It also adds a new function that is lightweight and just prints out the
configured URL as a simple template for testing and adapting

Testing done:
- ran the template function above

```
./e-mission-py.bash bin/historical/migrations/all_deployments_template.py
Connecting to database URLmongodb://localhost:27017/openpath_prod_nc_transit_equity_study
Running print_connect_url for ebikegj on DB ebikegj
Config file not found, returning a copy of the environment variables instead...
Retrieved config: {'DB_HOST': 'mongodb://localhost:27017/openpath_prod_ebikegj', 'DB_RESULT_LIMIT': None}
Connecting to database URL mongodb://localhost:27017/openpath_prod_ebikegj
Connecting to database URLmongodb://localhost:27017/openpath_prod_ebikegj
Running print_connect_url for cortezebikes on DB cortezebikes
Config file not found, returning a copy of the environment variables instead...
Retrieved config: {'DB_HOST': 'mongodb://localhost:27017/openpath_prod_cortezebikes', 'DB_RESULT_LIMIT': None}
Connecting to database URL mongodb://localhost:27017/openpath_prod_cortezebikes
Connecting to database URLmongodb://localhost:27017/openpath_prod_cortezebikes
Running print_connect_url for dcebike on DB dcebike
Config file not found, returning a copy of the environment variables instead...
Retrieved config: {'DB_HOST': 'mongodb://localhost:27017/openpath_prod_dcebike', 'DB_RESULT_LIMIT': None}
Connecting to database URL mongodb://localhost:27017/openpath_prod_dcebike
Connecting to database URLmongodb://localhost:27017/openpath_prod_dcebike
Running print_connect_url for washingtoncommons on DB washingtoncommons
Config file not found, returning a copy of the environment variables instead...
Retrieved config: {'DB_HOST': 'mongodb://localhost:27017/openpath_prod_washingtoncommons', 'DB_RESULT_LIMIT': None}
```
- ran the template function above with an overridden `PROD_LIST`. Note that the
  first two entries are from final values set to the DB_HOST variable before
  the program terminated previously. We may want to unset the variable before
  the program ends

```
$ export PROD_LIST='cortezebikes,fortmorgan'
$ ./e-mission-py.bash bin/historical/migrations/all_deployments_template.py
Config file not found, returning a copy of the environment variables instead...
Retrieved config: {'DB_HOST': 'mongodb://localhost/openpath_prod_nrel_commute', 'DB_RESULT_LIMIT': None}
Connecting to database URL mongodb://localhost/openpath_prod_nrel_commute
PROD_LIST: ['cortezebikes', 'fortmorgan']
Running print_connect_url for cortezebikes on DB cortezebikes
Config file not found, returning a copy of the environment variables instead...
Retrieved config: {'DB_HOST': 'mongodb://localhost:27017/openpath_prod_cortezebikes', 'DB_RESULT_LIMIT': None}
Connecting to database URL mongodb://localhost:27017/openpath_prod_cortezebikes
Connecting to database URLmongodb://localhost:27017/openpath_prod_cortezebikes
Running print_connect_url for fortmorgan on DB fortmorgan
Config file not found, returning a copy of the environment variables instead...
Retrieved config: {'DB_HOST': 'mongodb://localhost:27017/openpath_prod_fortmorgan', 'DB_RESULT_LIMIT': None}
Connecting to database URL mongodb://localhost:27017/openpath_prod_fortmorgan
Connecting to database URLmongodb://localhost:27017/openpath_prod_fortmorgan

```
  • Loading branch information
shankari committed Jan 16, 2025
1 parent d6f7108 commit 947492d
Show file tree
Hide file tree
Showing 2 changed files with 25 additions and 16 deletions.
33 changes: 17 additions & 16 deletions bin/historical/migrations/_common.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,30 @@
import subprocess
import importlib
import logging
import tempfile
import time

import emission.core.get_database as edb

DB_HOST_TEMPLATE = "mongodb://localhost:27017/openpath_prod_REPLACEME"
DB_HOST_TEMPLATE = os.environ.get('DB_HOST_TEMPLATE', "mongodb://localhost:27017/openpath_prod_REPLACEME")

proc = subprocess.run(
'rm -rf nrel-openpath-deploy-configs && ' +
'git clone --no-checkout https://github.com/e-mission/nrel-openpath-deploy-configs.git && ' +
'cd nrel-openpath-deploy-configs && ' +
'git ls-tree -r --name-only HEAD | grep configs/',
shell=True,
capture_output=True,
text=True)
filenames = proc.stdout.replace("configs/", "").split("\n")
if 'PROD_LIST' in os.environ:
PROD_LIST=os.environ['PROD_LIST'].split(",")
else:
with tempfile.TemporaryDirectory() as tmpdirname:
print(f"created {tmpdirname=} to find list of configs")
os.chdir(tmpdirname)
proc = subprocess.run(
f"git clone https://github.com/e-mission/nrel-openpath-deploy-configs.git", shell=True)
filenames = os.listdir(f"nrel-openpath-deploy-configs/configs/")

PROD_LIST = [
fname.split(".")[0]
for fname in filenames
if fname and 'dev-' not in fname and 'stage-' not in fname
]
PROD_LIST = [
fname.split(".")[0]
for fname in filenames
if fname and 'dev-' not in fname and 'stage-' not in fname
]
print(f"PROD_LIST: {PROD_LIST}")


def run_on_all_deployments(fn_to_run):
"""
Run the given function on the database for each deployment by setting the
Expand Down
8 changes: 8 additions & 0 deletions bin/historical/migrations/all_deployments_template.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
import emission.core.get_database as edb

from _common import run_on_all_deployments

def print_connect_url():
print("Connecting to database URL"+edb.url)

run_on_all_deployments(print_connect_url)

0 comments on commit 947492d

Please sign in to comment.