Bwa-mem2 mem parallelisation

[priyanka-surana]

Description of feature

For illumina and hic, I want to break the input fastq files, align them against the genome. The split files need to be aligned with the same @RG tag.

1. Split fastq using split
2. Make sure all split files have the same @RG tag
3. Run bwa-mem2 mem on them individually
4. Sort them with samtools individually
5. Merge all split alignments from the same individual with -c tag
6. Merge at the specimen level.
7. Run through the rest of the markdup_stats subworkflow

Maybe possible to combine steps (5) and (6).

[muffato]

For the record, we want to split the fastq file rather than the CRAM as it would allow reusing the workflow for PacBio and ONT, esp. as the PacBio BAM file is "weird".

[priyanka-surana]

@muffato Based on the latest discussion with Shane, are we still focusing on splitting by FASTQ. He recommended we split by BAM/CRAM to avoid excess I/O.

[muffato]

If you can avoid converting to fastq, please do so. It's a valid disk optimisation even without splitting the input file

[muffato]

We've run the pipeline quite a lot and Hi-C alignment with BWA doesn't seem to be an issue for us. The largest sample I've tested was Sambucus nigra: 6.3 billion reads (many species have around 1 billion reads) and 11.8 Gbp genome (the average genome size is < 1 Gbp). It took 2 days and 2 hours to run, which is fine in the After-Party context.