01_perform_dda.snakefile shuts down without error message on rule download_query_genome

[ccbaumler]

When running 01_perform_dda.snakefile and downloading a query genome for Muribaculaceae bacterium Z82 from genbank with the dominating set differential abundance rule download_query_genome of 01_perform_dda.snakefile, the snakefile would simply shut down with:

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message

The interesting thing was that there was no error message. So, I dug into the workflow and found that the genome URL below was broken with a 404 error:
https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/009/911/635/GCA_009911635.1_ASM991163v1/GCA_009911635.1_ASM991163v1_genomic.fna.gz
It turns out that the reference was removed and suppressed in GenBank.

I could use the package urllib that Taylor has used in this rule to create an error message and an if else loop that recursively removes the query genome and moves on.
https://docs.python.org/3/howto/urllib2.html#urlerror

@ctb

[ccbaumler]

A Titus hot-take solution is to use sourmash sig extract --picklist to remove all the deleted genomes from the database prior to gather.
Follow this example -> sourmash-bio/sourmash-examples#4 (comment)

[ctb]

actually, most sourmash commands take picklists, so you don't need to use extract specifically - you can use an exclusion picklist with gather directly.

picklist docs

You can also do clever things with manifests, e.g. this link

[ctb]

actually, most sourmash commands take picklists, so you don't need to use extract specifically - you can use an exclusion picklist with gather directly.

picklist docs

You can also do clever things with manifests, e.g. this link

[ctb]

problem cropping up elsewhere for us, too:

dib-lab/genome-grist#277
dib-lab/charcoal#235

[ccbaumler]

Temporary solution for this issue and the beginnings of something for this issue as well

https://github.com/dib-lab/2023-clean-gtdb