Release edits for cromshell2 (#255)

* Added short option for --cromwell-url * changed cromshell2 initiation from `cromshell-beta` to `cromshell`, minor edits to README.md * Added test badges * removed "Notify" command description for README.md * rebased cromshell sh script using master branch, moved cromshell sh script to legacy_cromshell folder * moved completions folder to legacy_cromshell folder --------- Co-authored-by: bshifaw <[email protected]>
broadinstitute · Apr 28, 2023 · b1d5b6a · b1d5b6a
1 parent abd12ba
commit b1d5b6a
Show file tree

Hide file tree

Showing 8 changed files with 412 additions and 54 deletions.
diff --git a/LICENSE b/LICENSE
@@ -1,6 +1,6 @@
 BSD 3-Clause License
 
-Copyright (c) 2018, Broad Institute
+Copyright (c) 2023, Broad Institute
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without

diff --git a/README.md b/README.md
@@ -7,36 +7,32 @@
       ""     ""    ""                                                          ""    ""     ""
 ```
 
-# cromshell
- A CLI for submitting workflows to a cromwell server and monitoring / querying their results.
+# Cromshell
+[![GitHub version](https://badge.fury.io/gh/broadinstitute%2Fcromshell.svg)](https://badge.fury.io/gh/broadinstitute%2Fcromshell)
+[![Integration Test Workflow](https://github.com/broadinstitute/cromshell/actions/workflows/integration_tests.yml/badge.svg)](https://github.com/broadinstitute/cromshell/actions/workflows/integration_tests.yml/badge.svg)
+[![Unit Test Workflow](https://github.com/broadinstitute/cromshell/actions/workflows/unit_tests.yml/badge.svg)](https://github.com/broadinstitute/cromshell/actions/workflows/unit_tests.yml/badge.svg)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-Current version: 2.0.0.beta 
-
-Cromshell 2 is the next step in the evolution Cromshell. It offers many of the same
-functions as Cromshell 1 but has been rebuilt in python with many added benefits such as:
-- Automatically zip dependencies when submitting a workflow.
-- Added integration and unit tests to insure reliability 
-- Multiple ways of installation (source, brew tap, and pypi).
-- Modular architecture making adding new functionalities easier.
-- Developer documentation showing how to add new features and commands to the tool.
+Cromshell is a CLI for submitting workflows to a Cromwell server and monitoring/querying their results.
 
 ## Examples:
 
 ```
-         cromshell-beta submit workflow.wdl inputs.json options.json dependencies.zip
-         cromshell-beta status
-         cromshell-beta -t 20 metadata
-         cromshell-beta logs -2
+         cromshell submit workflow.wdl inputs.json options.json dependencies.zip
+         cromshell status
+         cromshell -t 20 metadata
+         cromshell logs -2
 ```
 
 ## Supported Options:
   * `--no_turtle` or `--I_hate_turtles`
     * Hide turtle logo
   * `--cromwell_url [TEXT]`
-    * Specify Cromwell URL used. 
+    * Specifies Cromwell URL used. 
     * `TEXT` Example: `http://65.61.654.8:8000`
+    * Note: Depending on your Cromwell server configuration, you may not need to specify the port.
   * `-t [TIMEOUT]`
-    * Specify the server connection timeout in seconds. 
+    * Specifies the server connection timeout in seconds. 
     * Default is 5 sec.
     * `TIMEOUT` must be a positive integer.
   * `--gcloud_token_email [TEXT]`
@@ -51,16 +47,16 @@ functions as Cromshell 1 but has been rebuilt in python with many added benefits
 
    ####  Start/Stop workflows
    * `submit [-w] <wdl> <inputs_json> [options_json] [included_wdl_zip_file]`
-     * Will automatically validate the WDL and JSON file.
-     * Submit a new workflow.
+     * Automatically validates the WDL and JSON file.
+     * Submit a new workflow to the Cromwell server.
      * *`-w`* [COMING SOON] Wait for workflow to transition from 'Submitted' to some other status before ${SCRIPTNAME} exits.
      * *`included_wdl_zip_file`*  Zip file containing any WDL files included in the input WDL
    * `abort [workflow-id] [[workflow-id]...]`               
      * Abort a running workflow.
    #### Workflow information:
    * `alias <workflow-id> <alias_name>`
      * Label the given workflow ID with the given alias_name.  Aliases can be used in place of workflow IDs to reference jobs.
-     * Remove alias by passing empty double quotes as `alias_name` (e.g. `alias <workflow-id> ""`)
+     * Remove an alias by passing empty double quotes as `alias_name` (e.g. `alias <workflow-id> ""`)
    #### Query workflow status:
    * `status [workflow-id] [[workflow-id]...]`                   
      * Check the status of a workflow.
@@ -72,8 +68,7 @@ functions as Cromshell 1 but has been rebuilt in python with many added benefits
      * Get the summarized status of all jobs in the workflow.
      * `-j` prints a JSON instead of a pretty summary of the execution status (compresses subworkflows)
      * `-x` compress sub-workflows for less detailed summarization
-   * `timing` *`[workflow-id] [[workflow-id]...]`*                  
-
+   * `timing` *`[workflow-id] [[workflow-id]...]`*
      * Open the timing diagram in a browser.
 
    #### Logs
@@ -83,26 +78,22 @@ functions as Cromshell 1 but has been rebuilt in python with many added benefits
      * Download all logs produced by a workflow.
 
    #### Job Outputs
-   * [COMING SOON] `list-outputs [workflow-id] [[workflow-id]...]`         
+   * `list-outputs [workflow-id] [[workflow-id]...]`         
      *  List all output files produced by a workflow.
    * [COMING SOON] `fetch-all [workflow-id] [[workflow-id]...]`        
      * Download all output files produced by a workflow.
-
-   ####  Get email notification on job completion
-   * [COMING SOON] `notify [workflow-id] [daemon-server] email [cromwell-server]`
-     * *`daemon-server`*  server to run the notification daemon on
 
    #### Display a list jobs submitted through cromshell
    * `list [-c] [-u]`                                            
-     * `-c`    Color the output by completion status
-     * `-u`    Check completion status of all unfinished jobs
+     * `-c` Color the output by completion status.
+     * `-u` Check completion status of all unfinished jobs.
 
    #### Clean up local cached list
    * [COMING SOON] `cleanup [-s STATUS]`
      * Remove completed jobs from local list.
-       Will remove all jobs from the local list that are in a completed state,
+       This command removes all jobs from the local list that are in a completed state,
        where a completed state is one of: `Succeeded`, `Failed`, `Aborted`
-     * *`-s STATUS`*     If provided, will only remove jobs with the given `STATUS` from the local list.
+     * *`-s [STATUS]`* If provided, will only remove jobs with the given `[STATUS]` from the local list.
 
    #### Update cromwell server
    * `update-server`
@@ -127,9 +118,9 @@ functions as Cromshell 1 but has been rebuilt in python with many added benefits
  You may omit the job ID of the last job submitted when running commands, or use negative numbers to reference previous jobs, e.g. "-1" will track the last job, "-2" will track the one before that, and so on.
  * You can override the default cromwell server by setting the argument `--cromwell_url` to the appropriate URL.
  * You can override the default cromshell configuration folder by setting the environmental variable `CROMSHELL_CONFIG` to the appropriate directory.
- * Most commands takes multiple workflow-ids, which you *can specify both in relative and absolute ID value* (i.e. `./cromshell status -- -1 -2 -3 c2db2989-2e09-4f2c-8a7f-c3733ae5ba7b`). 
- * Assign aliases to workflow ids using the alias command (i.e. `./cromshell alias -- -1 myAliasName`).
- * Once the Alias command is used to attach an alias to a workflow id, the alias name can be used instead of the id (i.e. `./cromshell status myAliasName`).
+ * Most commands takes multiple workflow-ids, which you *can specify both in relative and absolute ID value* (i.e. `cromshell status -- -1 -2 -3 c2db2989-2e09-4f2c-8a7f-c3733ae5ba7b`). 
+ * Assign aliases to workflow ids using the alias command (i.e. `cromshell alias -- -1 myAliasName`).
+   Once the Alias command is used to attach an alias to a workflow id, the alias name can be used instead of the id (i.e. `cromshell status myAliasName`).
 
 ## Installation
 From brew
@@ -145,10 +136,9 @@ From source
 
     git clone [email protected]:broadinstitute/cromshell.git
     cd cromshell
-    git checkout cromshell_2.0
     pip install .
 
-    cromshell-beta --help
+    cromshell --help
 
 ## Uninstallation
 From brew
@@ -157,10 +147,14 @@ From brew
 
 From pypi/source
 
-    pip uninstall cromshell-beta
+    pip uninstall cromshell
 
 ## Development
 
 See the [Developer Docs](./developer_docs/)
 
+## Legacy Cromshell
 
+The original Cromshell shell script is still available in the legacy_cromshell folder and in the `cromshell1` branch of this repository.
+It is no longer maintained, but is still available for use. The original Cromshell contains some commands not yet available in Cromshell2,
+such as `fetch-logs`, `fetch-all`, `notify`, and `cleanup`. These commands will be added to Cromshell2 in the future.
diff --git a/legacy_cromshell/README.md b/legacy_cromshell/README.md
@@ -0,0 +1,130 @@
+# Legacy Cromshell 
+
+The legacy cromshell folder contains the original cromshell script. It is no longer maintained and is only kept for historical purposes.
+
+---
+
+```
+                  __                                                            __
+       .,-;-;-,. /'_\     +-----------------------------------------------+    /_'\.,-;-;-,.
+     _/_/_/_|_\_\) /      |  CROMSHELL : run Cromwell jobs from the shell |     \ (/_/__|_\_\_
+   '-<_><_><_><_>=/\      +-----------------------------------------------+     /\=<_><_><_><_>-'
+     `/_/====/_/-'\_\                                                          /_/'-\_\====\_\'
+      ""     ""    ""                                                          ""    ""     ""
+```
+
+# Installation
+
+`cromshell` and its dependencies can be installed on OSX with `brew install broadinstitute/dsp/cromshell`
+
+or through [bioconda](https://bioconda.github.io/) with `conda install cromshell`
+
+Alternatively, download the script and put it somewhere...
+
+# cromshell
+ A script for submitting workflows to a cromwell server and monitoring / querying their results.
+
+requires `column`, `curl`, `mail`, and [jq](https://stedolan.github.io/jq/)
+
+### Examples:
+
+```
+         cromshell submit workflow.wdl inputs.json options.json dependencies.zip
+         cromshell status
+         cromshell -t 20 metadata
+         cromshell logs -2
+```
+
+### Supported Flags:
+  * `-t` `TIMEOUT`
+    * Set the curl connect timeout to `TIMEOUT` seconds.
+    * Also sets the curl max timeout to `2*TIMEOUT` seconds.
+    * `TIMEOUT` must be an integer.
+
+### Supported Subcommands:
+
+
+   ####  Start/Stop workflows
+   * `submit` `[-w]` *`<wdl>`* *`<inputs_json>`* `[options_json]` `[included_wdl_zip_file]`
+     * Submit a new workflow.
+     * Will automatically validate the WDL and JSON file if `womtool` is in your path.
+       * To add `womtool` to your path, install `cromwell` with brew:
+         * `brew install cromwell`
+     * *`-w`*                     Wait for workflow to transition from 'Submitted' to some other status
+                                  before ${SCRIPTNAME} exits
+     * *`included_wdl_zip_file`*  Zip file containing any WDL files included in the input WDL
+   * `abort` *`[workflow-id] [[workflow-id]...]`*                   
+     * Abort a running workflow.
+   #### Workflow information:
+   * `alias` *`<workflow-id>` `<alias_name>`* 
+     * Label the given workflow ID with the given alias_name.  Aliases can be used in place of workflow IDs to reference jobs.
+   #### Query workflow status:
+   * `status` *`[workflow-id] [[workflow-id]...]`*                   
+     * Check the status of a workflow.
+   * `metadata` *`[workflow-id] [[workflow-id]...]`*                
+     * Get the full metadata of a workflow.
+   * `slim-metadata` *`[workflow-id] [[workflow-id]...]`*           
+     * Get a subset of the metadata from a workflow.
+   * `execution-status-count`, `counts` *`[-p] [-x] [workflow-id] [[workflow-id]...]`*   
+     * Get the summarized status of all jobs in the workflow.
+     * `-p` prints a pretty summary of the execution status instead of JSON
+     * `-x` expands sub-workflows for more detailed summarization
+   * `timing` *`[workflow-id] [[workflow-id]...]`*                  
+     * Open the timing diagram in a browser.
+
+   #### Logs
+   * `logs` *`[workflow-id] [[workflow-id]...]`*                     
+     * List the log files produced by a workflow.
+   * `fetch-logs` *`[workflow-id] [[workflow-id]...]`*               
+     * Download all logs produced by a workflow.
+
+   #### Job Outputs
+   * `list-outputs` *`[workflow-id] [[workflow-id]...]`*           
+     *  List all output files produced by a workflow.
+   * `fetch-all` *`[workflow-id] [[workflow-id]...]`*             
+     * Download all output files produced by a workflow.
+
+   ####  Get email notification on job completion
+   * `notify` *`[workflow-id]` `[daemon-server]` `email` `[cromwell-server]`*
+     * *`daemon-server`*  server to run the notification daemon on
+
+   #### Display a list jobs submitted through cromshell
+   * `list` *`[-c]` `[-u]`*                                            
+     * *`-c`*    Color the output by completion status
+     * *`-u`*    Check completion status of all unfinished jobs
+
+   #### Clean up local cached list
+   * `cleanup` *`[-s STATUS]`*    
+     * Remove completed jobs from local list.
+       Will remove all jobs from the local list that are in a completed state,
+       where a completed state is one of: `Succeeded`, `Failed`, `Aborted`
+     * *`-s STATUS`*     If provided, will only remove jobs with the given `STATUS` from the local list.
+
+   #### Update cromwell server
+   * `update-server`
+     * Change the cromwell server that new jobs will be submitted to.
+
+   #### Get cost of a workflow
+   Costs are only available for workflows that completed more than 8 hours ago on a `GCS` backend.  
+   Requires the `~/.cromshell/gcp_bq_cost_table.config` configuration file to exist and contain the name of the BigQuery cost table for your organization.
+   Also ensure that the default project has been set using `bq init`.
+   * `cost [workflow-id] [[workflow-id]...]`
+     * Get the cost for a workflow.
+   * `cost-detailed [workflow-id] [[workflow-id]...]`
+     * Get the cost for a workflow at the task level.
+
+
+ ### Features:
+ * Running `submit` will create a new folder in the `~/.cromshell/${CROMWELL_URL}/` directory named with the cromwell job id of the newly submitted job.  
+ It will copy your wdl and json inputs into the folder for reproducibility.  
+ * It keeps track of your most recently submitted jobs by storing their ids in `./cromshell/`  
+ You may omit the job ID of the last job submitted when running commands, or use negative numbers to reference previous jobs, e.g. "-1" will track the last job, "-2" will track the one before that, and so on.
+ * You can override the default cromwell server by setting the environmental variable `CROMWELL_URL` to the appropriate URL.
+ * Most commands takes multiple workflow-ids, which you *can specify both in relative and absolute ID value* (i.e. `./cromwell status -1 -2 -3 c2db2989-2e09-4f2c-8a7f-c3733ae5ba7b`). 
+ * You can supply additional headers to Cromshell REST calls by setting the environmental variable `CROMSHELL_HEADER`. This is useful if your Cromwell server is fronted by an auth server that authenticates access using bearer tokens before forwarding requests onto the Cromwell API. For example: `CROMSHELL_HEADER="Authorization: Bearer 3e2f34f2e..."`
+
+ ### Code Conventions:
+ Please try to follow these conventions when editing cromshell.
+ * Use double brackets for tests ( `[[ ... ]]` instead of `[]`)
+ * Use `{}` when doing dereferencing variables (`${VALUE}`,`${1}` instead of `$VALUE`,`$1`)
+ * Define functions with the `function` keyword (`function doThing()` instead of `doThing()`)
diff --git a/completions/_cromshell → legacy_cromshell/completions/_cromshell b/completions/_cromshell → legacy_cromshell/completions/_cromshell
@@ -23,6 +23,9 @@ function _cromshell {
                notify\:'Get email notification on job completion'
                list\:'Display a list jobs submitted through cromshell'
                cleanup\:'Clean up local cached list'
+               cost\:'Display workflow cost'
+               cost-detailed\:'Display workflow cost broken down by task'
+               update-server\:'Change the cromshell server new jobs will be submited to'
              ))" \
         "*::arg:->args"
 
@@ -33,7 +36,7 @@ function _cromshell {
         alias)
           _alias
         ;;
-        abort|status|metadata|slim-metadata|timing|logs|fetch-logs|list-outputs|fetch-all)
+        abort|status|metadata|slim-metadata|timing|logs|fetch-logs|list-outputs|fetch-all|cost|cost-detailed)
           _id_list_function
         ;;
         list)

diff --git a/legacy_cromshell/completions/reload.sh b/legacy_cromshell/completions/reload.sh
@@ -0,0 +1,8 @@
+# Reload the _cromshell completion file for testing
+# Usage:
+# source reload.sh
+
+fpath=($PWD $fpath)
+unfunction _cromshell
+autoload -U compinit
+compinit