The train step needs large memory

[jiangzy26]

Hi,
When I run the RELERNN_TRAIN with default settings, the step was killed because of the large memory, how to deal with this? could you share your help? Thank you very much.

[jiangzy26]

Or, Could I use the same command line to simulate each chromosome, and Train each chromosome to get the predicted result?

[andrewkern]

how much memory are you using?

[willright28]

Hi, @andrewkern and all users,

I had the same question. The training step was fine when no demography history was set, but it took up to 3 T or more of space (no tested but killed) when demography history was set.

My code is:
ReLERNN/ReLERNN_SIMULATE
-v A.phased.vcf \ #only filter missing snp --max-missing 0.9 using vcftools
--phased
-g fasta.fai
-d ./A
-n A_two-epoch_unfold.final.summary \ #output of stairwayplot2
-l 1
-m reference.missing.bed
-t 80

Any clue on how to solve this?

Thanks in advance!

Best,
willright28

[andrewkern]

hi @willright28 -- what does your demographic history look like? I'm guessing a contracting population size moving to the present?

[willright28]

Hi @andrewkern
Thanks for your reply. The demographic history looks like this, a strong bottleneck then a rebound:

[andrewkern]

Just so I'm oriented here-- is the y-axis correct? you have Ne going down to 0.1? Are these in relative units?

[willright28]

Sorry for the misleading, the y-axis is log-transformed. The lowest Ne is ~1300.

[andrewkern]

Okay thanks for the clarification. I'm looking at your code-- it looks like your -g flag is pointing to a .fai file-- is that a indexed fasta or a bed file?

My code is: ReLERNN/ReLERNN_SIMULATE -v A.phased.vcf \ #only filter missing snp --max-missing 0.9 using vcftools --phased -g fasta.fai -d ./A -n A_two-epoch_unfold.final.summary \ #output of stairwayplot2 -l 1 -m reference.missing.bed -t 80

[willright28]

The .fai file is a bed file formated as chr1 0 100000 (length of chr)

[willright28]

Maybe I can set --maxsites to a reasonable number to avoid this problem?

[andrewkern]

this shouldn't be too big. can you provide for me your input files and I can poke around

[willright28]

I have tested to run a single chromosome, chr1, which is ~ 10% of the whole genome. The program worked fine and only took ~ 0.1 T memory.
Is running each chromosome separately and using the same demographic history in the -n parameter ok?

[andrewkern]

yes it should be fine to run each chromosome separately