vpt HPC execution

[josemq]

Hello,

We've converted a Docker image to a Singularity image for execution in an HPC environment. After running for more than 3 hours and 120GB plus memory consumed, we are now being overloaded with the messages below.

2023-08-03 17:53:19,360 - ./task-20 - INFO - prepare_watershed_images
2023-08-03 17:54:43,495 - ./task-5 - INFO - watershed
2023-08-03 17:54:46,306 - distributed.worker_memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 11.09 GiB -- Worker memory limit: 15.72 GiB
2023-08-03 17:54:51,999 - distributed.worker_memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 11.00 GiB -- Worker memory limit: 15.72 GiB
2023-08-03 17:54:52,099 - distributed.worker_memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 11.00 GiB -- Worker memory limit: 15.72 GiB
2023-08-03 17:54:52,199 - distributed.worker_memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker-memory.html#memory-not-released-back-to-the-os for more information. -- Unmanaged memory: 11.00 GiB -- Worker memory limit: 15.72 GiB

I terminated the process but would like to know if this is expected and what are the typical hardware resource requirements for this type of execution. Please let me know if you need further details and how can we further troubleshoot. Below are the parameters we are executing with:

singularity exec /home/USER/VPT_Run/vpt/vpt_latest.sif vpt
--verbose
--processes 16
--log-file $OUTDIR'/run.log'
--log-level 1
run-segmentation
--segmentation-algorithm $OUTDIR'/watershed.json'
--input-images $OUTDIR'/'$RUN'/mosaic_(?P[A-Za-z]+)_z(?P[0-9]+).tif'
--input-micron-to-mosaic $OUTDIR'/'$RUN'/micron_to_mosaic_pixel_transform.mat'
--overwrite
--output-path $OUTDIR'/'$RUN

[lindsey-ching]

Hello,

I'm also having this same issue. Are there any updates on this problem?