Pipeline fails when work directory is not removed before run

[AWS-crafter]

Description of the bug

nextflow run main.nf -profile docker [...] --run_ancestry fails when it was ran previously but when the work directory has not been removed. The samplesheet format is pfile. This occurred with multiple different files.

The easy workaround is to just remove the work directory before running, so this is not important/urgent.

Command used and terminal output

Here is an example:
initial nextflow run --> success
repeating the same command (same files) immediately after --> failure
rm -rf work
repeating the same command (same files) --> success

Or...
initial nextflow run --> failure
rm -rf work
running the pipeline on a different file immediately after --> success

Relevant files

No response

System information

pgscatalog/pgsc_calc v2.0.0-alpha.2
Ubuntu, on AWS EC2
Docker

[smlmbrt]

@AWS-crafter are you retrying it while adding -resume? That often solves this issue. Relevant nextflow docs:

• https://training.nextflow.io/basic_training/cache_and_resume/#race-condition-in-global-variable
• https://www.nextflow.io/blog/2019/demystifying-nextflow-resume.html

[nebfield]

This could also be related to the VCF input bug #180

We cache intermediate files (e.g. recoded genomes, summary stats, etc.) across runs on the same dataset to save space and CPU time. If a bugged workflow run gets cached then future runs of the pipeline could fail randomly.

It's worth waiting to see if this still happens for you when we release a patch sometime next week.