From e1b3f46b6f3ab1d4dd82290af61e445228cdc66a Mon Sep 17 00:00:00 2001 From: Jenny V Medina Date: Wed, 10 Jan 2024 15:28:47 -0500 Subject: [PATCH 1/5] Update README.md --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index f7e4282..da59f94 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ This Python package provides the components to connect various third-party servi This repository includes a demonstration script called [`demo.py`](demo.py), which showcases how you can use `py-orca` to launch and monitor your workflows on Nextflow Tower. Specifically, it illustrates how to process an RNA-seq dataset using a series of workflow runs, namely `nf-synstage`, `nf-core/rnaseq`, and `nf-synindex`. `py-orca` can be used with any Python-compatible workflow management system to orchestrate each step (_e.g._ Airflow, Prefect, Dagster). The demonstration script uses [Metaflow](https://metaflow.org/) because it's easy to run locally and has an intuitive syntax. -The script assumes that the following environment variables are set. +The script assumes that the following environment variables are set. Before setting up these environment variables, ensure that you have an AWS profile configured for a role that has access to the dev/ops tower workspace you plan to launch your workflows from. Refer to [`.env.example`](.env.example) for the format of their values as well as examples. You can set these environment variables using whatever method you prefer (_e.g._ using an `.env` file, sourcing a shell script). - `NEXTFLOWTOWER_CONNECTION_URI` @@ -24,7 +24,9 @@ Refer to [`.env.example`](.env.example) for the format of their values as well a Once your environment is set, you can create a virtual environment, install the Python dependencies, and run the demonstration script (after downloading it) as follows. Note that you will need to update the `s3_prefix` parameter so that it points to an S3 bucket that is accessible to your Tower workspace. -### Manually creating a virtual environment +### Creating and setting up your py-`orca` virtual environment and executing `demo.py` + +Below are the instructions for creating and setting up your virtual environment and executing the `demo.py` If you would like to set up a developer environment with the relevant dependencies, you can execute the shell script [dev_setup](https://github.com/Sage-Bionetworks-Workflows/py-orca/blob/main/dev_setup.sh). ```bash # Create and activate a Python virtual environment (tested with Python 3.10) python3 -m venv venv/ From ab40f501892f4170dac49b6aa6c7f402b592e474 Mon Sep 17 00:00:00 2001 From: Jenny Medina Date: Fri, 12 Jan 2024 11:26:32 -0500 Subject: [PATCH 2/5] Restructuring --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index da59f94..4a662d0 100644 --- a/README.md +++ b/README.md @@ -15,8 +15,8 @@ This Python package provides the components to connect various third-party servi This repository includes a demonstration script called [`demo.py`](demo.py), which showcases how you can use `py-orca` to launch and monitor your workflows on Nextflow Tower. Specifically, it illustrates how to process an RNA-seq dataset using a series of workflow runs, namely `nf-synstage`, `nf-core/rnaseq`, and `nf-synindex`. `py-orca` can be used with any Python-compatible workflow management system to orchestrate each step (_e.g._ Airflow, Prefect, Dagster). The demonstration script uses [Metaflow](https://metaflow.org/) because it's easy to run locally and has an intuitive syntax. -The script assumes that the following environment variables are set. Before setting up these environment variables, ensure that you have an AWS profile configured for a role that has access to the dev/ops tower workspace you plan to launch your workflows from. -Refer to [`.env.example`](.env.example) for the format of their values as well as examples. You can set these environment variables using whatever method you prefer (_e.g._ using an `.env` file, sourcing a shell script). +The script assumes that the following environment variables are set. Before setting them up, ensure that you have an AWS profile configured for a role that has access to the dev/ops tower workspace you plan to launch your workflows from. You can set these environment variables using whatever method you prefer (_e.g._ using an `.env` file, sourcing a shell script). +Refer to [`.env.example`](.env.example) for the format of their values as well as examples. - `NEXTFLOWTOWER_CONNECTION_URI` - `SYNAPSE_CONNECTION_URI` From 0674d1245d86bc321c2f5f82e50c2398ba5aaee9 Mon Sep 17 00:00:00 2001 From: Jenny Medina Date: Fri, 12 Jan 2024 11:37:41 -0500 Subject: [PATCH 3/5] added new example. restructuring. --- .env.example | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/.env.example b/.env.example index 987a2b9..94d145d 100644 --- a/.env.example +++ b/.env.example @@ -47,8 +47,9 @@ # IMPORTANT: The ':' and '@' characters before and after # the access token are required. Do not omit them. # -# Examples: +# Structure: # export SEVENBRIDGES_CONNECTION_URI=sbg://:@[/?project=] +# Examples: # export SEVENBRIDGES_CONNECTION_URI=sbg://:f560[...]bf9d@cavatica-api.sbgenomics.com/v2 # export SEVENBRIDGES_CONNECTION_URI=sbg://:f560[...]bf9d@cavatica-api.sbgenomics.com/v2/?project=bgrande/sandbox @@ -68,7 +69,9 @@ # IMPORTANT: The ':' and '@' characters before and after # the access token are required. Do not omit them. # -# Examples: +# Structure: # export NEXTFLOWTOWER_CONNECTION_URI=tower://:@[/?workspace=/] +# Examples: # export NEXTFLOWTOWER_CONNECTION_URI=tower://:eyJ0[...]MA==@api.tower.nf # export NEXTFLOWTOWER_CONNECTION_URI=tower://:eyJ0[...]MA==@tower.sagebionetworks.org/api/?workspace=sage-bionetworks/example-project +# export NEXTFLOWTOWER_CONNECTION_URI=tower://:eyJ0[...]MA==@tower-dev.sagebionetworks.org/api/?workspace=sage-bionetworks/example-dev-project From 7d08382b19069057d8d005dd8a911b2274c69832 Mon Sep 17 00:00:00 2001 From: Jenny Medina Date: Fri, 12 Jan 2024 11:42:00 -0500 Subject: [PATCH 4/5] Clarification in README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4a662d0..fdd6b1b 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ Once your environment is set, you can create a virtual environment, install the ### Creating and setting up your py-`orca` virtual environment and executing `demo.py` -Below are the instructions for creating and setting up your virtual environment and executing the `demo.py` If you would like to set up a developer environment with the relevant dependencies, you can execute the shell script [dev_setup](https://github.com/Sage-Bionetworks-Workflows/py-orca/blob/main/dev_setup.sh). +Below are the instructions for creating and setting up your virtual environment and executing the `demo.py` If you would like to set up a developer environment with the relevant dependencies, you can execute the shell script [dev_setup](https://github.com/Sage-Bionetworks-Workflows/py-orca/blob/main/dev_setup.sh) in a clone of this repository stored on your machine. ```bash # Create and activate a Python virtual environment (tested with Python 3.10) python3 -m venv venv/ From 1b0ce170c3b3a506c851867aa94f6ccfaf56dd4d Mon Sep 17 00:00:00 2001 From: Jenny Medina Date: Fri, 12 Jan 2024 11:46:03 -0500 Subject: [PATCH 5/5] minor fixes --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index fdd6b1b..37c28e6 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ This Python package provides the components to connect various third-party servi This repository includes a demonstration script called [`demo.py`](demo.py), which showcases how you can use `py-orca` to launch and monitor your workflows on Nextflow Tower. Specifically, it illustrates how to process an RNA-seq dataset using a series of workflow runs, namely `nf-synstage`, `nf-core/rnaseq`, and `nf-synindex`. `py-orca` can be used with any Python-compatible workflow management system to orchestrate each step (_e.g._ Airflow, Prefect, Dagster). The demonstration script uses [Metaflow](https://metaflow.org/) because it's easy to run locally and has an intuitive syntax. -The script assumes that the following environment variables are set. Before setting them up, ensure that you have an AWS profile configured for a role that has access to the dev/ops tower workspace you plan to launch your workflows from. You can set these environment variables using whatever method you prefer (_e.g._ using an `.env` file, sourcing a shell script). +The script assumes that the following environment variables are set. Before setting them up, ensure that you have an AWS profile configured for a role that has access to the dev/ops tower workspace you plan to launch your workflows from. You can set these environment variables using whatever method you prefer (_e.g._ using an `.env` file, sourcing a shell script, etc). Refer to [`.env.example`](.env.example) for the format of their values as well as examples. - `NEXTFLOWTOWER_CONNECTION_URI` @@ -26,7 +26,7 @@ Once your environment is set, you can create a virtual environment, install the ### Creating and setting up your py-`orca` virtual environment and executing `demo.py` -Below are the instructions for creating and setting up your virtual environment and executing the `demo.py` If you would like to set up a developer environment with the relevant dependencies, you can execute the shell script [dev_setup](https://github.com/Sage-Bionetworks-Workflows/py-orca/blob/main/dev_setup.sh) in a clone of this repository stored on your machine. +Below are the instructions for creating and setting up your virtual environment and executing the `demo.py`. If you would like to set up a developer environment with the relevant dependencies, you can execute the shell script [dev_setup](https://github.com/Sage-Bionetworks-Workflows/py-orca/blob/main/dev_setup.sh) in a clone of this repository stored on your machine. ```bash # Create and activate a Python virtual environment (tested with Python 3.10) python3 -m venv venv/