Empty probability map found error

[sithin42]

Hi,

I am having a really hard time debugging the following error.

Why?

1. Probability map is not empty
2. It is already padded
3. I cannot reproduce the error

Can someone please help?

[romainVala]

your full probability map may not be empty, but a smaller patch can ...

[sithin42]

Thank you for your response.

I'm using torchio weighted sampler, so in theory, it should not sample patches from an empty probability map region, right? If I understood it correctly, weighted sampler will only sample patches from locations with non-zero values in the probability map as the center, right?

[romainVala]

yes
looking more carefully to your error, it is not related to the patch but as the message said
the subject 083 proba map is empty, ...
what doe contain the volume pointed in the error message,
is it 3D 4D ? (sure that it is not full of zero ... ?)

[sithin42]

The file points towards a uint8 sitk image file with mostly zeros and a few ones. It's really weird. I read the file mentioned in the path, it is 100% not full of zeros. It's quite random actually. It raised the same error with other samples as well. The training goes on for a few epochs and then it stops due to this error. It feels like the error might be misleading and is caused by something else. I have no clue on what could possibly be the source though!

The actual data is a 3D volume sitk image- HxWxD but when we use torchio to read the image, it becomes 4D so the format is CxHxWxD where C=1

[fepegar]

Can you identify the images causing this issue and share them here, and also a minimal reproducible example that we can run?

[sithin42]

I couldn't reproduce the error on a minimal example using the sample inside the path mentioned in the error message.

sample: https://wetransfer.com/downloads/96bea497b8a3a7837d2df5057079a27120211213123316/1ac131f10478d5f32f92d5ce416927a520211213123317/14eb52

import torchio as tio
import torch
import os
import numpy as np

DATA_DIR = r"./lung_083"
patch_size = (160,160,80)

subject = tio.Subject(img=tio.ScalarImage(os.path.join(DATA_DIR,"res_image.nii.gz")),mask=tio.ScalarImage(os.path.join(DATA_DIR,"res_mask.nii.gz")),prob_map=tio.Image(os.path.join(DATA_DIR,"res_prob_map.nii.gz"),type=tio.SAMPLING_MAP))
ds = tio.SubjectsDataset([subject],transform=tio.Pad(patch_size))

sampler = tio.LabelSampler(patch_size,'prob_map')
queue = tio.Queue(ds,100,100,sampler,0,verbose=True)

for i in range(1000):
    
    indeces = torch.arange(len(queue)).float().multinomial(num_samples=2)
    print(indeces)
    subset = list(torch.utils.data.Subset(queue, indeces))
    batch = torch.stack([patch["prob_map"][tio.DATA] for patch in subset])
    print(batch.sum())

[fepegar]

I've tried to reproduce with the following code (adapted from yours), with no luck:

from pathlib import Path
import torch
import torchio as tio

data_dir = Path('/tmp/lung_083')
patch_size = (160, 160, 80)

subject = tio.Subject(
    img=tio.ScalarImage(data_dir / 'res_image.nii.gz'),
    mask=tio.ScalarImage(data_dir / 'res_mask.nii.gz'),
    prob_map=tio.LabelMap(data_dir / 'res_prob_map.nii.gz'),
)
ds = tio.SubjectsDataset([subject], transform=tio.Pad(patch_size))

sampler = tio.LabelSampler(patch_size, 'prob_map')
queue = tio.Queue(ds, 10, 10, sampler, verbose=True)
loader = torch.utils.data.DataLoader(queue)

while True:
    for batch in loader:
        pass

Note that you can use tio.LabelMap instead of tio.Image.

[sithin42]

Really appreciate your help! Yeah, it is quite random, I couldn't reproduce it either. I have no clue what's causing this error. But I think the error message is misleading.

[romainVala]

If there is non zero element in your example, then it get zero after going to clear_probability_borders

https://github.com/fepegar/torchio/blob/3e07b78da16d6db4da7193325b3f9cb31fc0911a/torchio/data/sampler/weighted.py#L104

it means that you need to have non zeros within the given crop (related to your patch size I think)

[sithin42]

The use of tio.Pad(patch_size//2) augmentation should prevent that, No?

[romainVala]

yes, (may be try patch_size//2 + 1, to be 100% sure, although it should not change much.

that is indeed weird but as fernado said we need to reproduce to be able to help further

[sithin42]

When this error was raised, I was actually using tio.Pad(patch_size) instead of tio.Pad(patch_size//2) to be on the safe side. I am unable to reproduce the same error on a minimal example.