Genie DAG import scripts

[callachennault]

This PR adds scripts for the Genie import DAG:

• dags/import_genie_dag.py: DAG which handles the genie import

• genie-airflow-clone-db.sh: Drop tables in non-prod SQL DB, clone prod DB

• genie-airflow-setup.sh: DB check, repo fetch, CDD/Oncotree cache refresh

• genie-airflow-import-sql.sh: Import cancer types into SQL, import from genie-portal column into SQL DB

• genie-airflow-import-clickhouse.sh: Import into Clickhouse DB

• updates to refresh-cdd-oncotree-cache.sh to use correct URLs for CDD and OncoTree

• DISCUSS: Should these scripts be moved to the same repository as the genie import DAG? Should the genie import DAG be moved to this repo? - Discussed with Avery 1/29: will store import_genie_dag in cmo-pipelines

[sheridancbio]

review covers first file only, but most comments apply to all scripts
additional scripts will be reviewed in a second review.

[sheridancbio]

additional issues found in remaining files (part 2 of review)

[averyniceday]

Overall looks good for first release --

couple comments regarding logging messages, mostly just clarifying Source/Dest db.

For future work/improvements, need to make sure server is inline with what the scripts expect (esp. in this SSHOperator mode of running)

Also suggest (maybe this is already in the DAG repo) -- adding a README.md for the DAG workflow and steps. What is "wrapper" logic versus tooling logic and expected frequency of updates/tweaks.

[callachennault]

thanks @averyniceday, responding below:

For future work/improvements, need to make sure server is inline with what the scripts expect (esp. in this SSHOperator mode of running)

Can you elaborate on what you mean by this?

Also suggest (maybe this is already in the DAG repo) -- adding a README.md for the DAG workflow and steps. What is "wrapper" logic versus tooling logic and expected frequency of updates/tweaks.

I'd prefer not to document the DAG steps because Airflow already generates a DAG diagram which will automatically reflect any changes to the tasks/task order. The tasks are also documented in the DAG file and in the scripts themselves. What do you mean by tooling logic?