Possibility to reduce run time using Spacingd Transformation in this way?

[ghost]

Hello,
I use Spacingd and pixdim to standardise incoming image dimensions. Of course when I choose higher pixdim values I have smaller image volumes therefore having shorter run times. It turns out that I don't actually have sufficient computational power to use the pixdim resolution I ideally want to use, I run out of memory.

I have trained the model to a satisfactory level on a certain pixdim size i.e. 3 mm. Out of interest I then evaluated datasets with a smaller pixdim size i.e. 1 mm. As you would expect, the model unsurprisingly does a poor job: I can't really blame it! I wondered though, if there was anything I could do (besides accessing better computers!) which could enable this type of technique to work? I was considering if there was anything I could do like adding another layer, or upscaling the model in some way?

It is a long shot but wanted to ask!

Thanks.

[rijobro]

Are you already cropping your input images? If you aren't doing this already, it would enable you to train smaller images, meaning you might be able to do less downsampling.

The transform you might use for cropping would depend on your application:

• Segmentation: you could use CropForeground, which can be used to crop both the input image and segmentation to the area containing your segmentations.
• Classification (and segmentation): CenterSpatialCrop, or SpatialCrop. These also have random versions (RandCenterSpatialCrop, etc.) to enable data augmentation during training.

[ghost]

Thanks Richard!

Yes, I'm using both CropForegroundd and RandCropByPosNegLabeld to reduce the input image size.

Ideally I want to use a pixel/voxel size of 1 mm. If I choose both sufficiently small spatial_size and num_samples for the cropping I can get it the training to run at 1 mm, but it is a compromise.

From the training I've done so far, not only does it take longer to converge but the loss plateaus at a far higher level (~0.3 vs >0.1 at 3 mm voxel size), I think I'm having a fundamental issue where I'm not providing the model with large enough RandCrop subvolumes for it to actually learn what is required.

Doing a back of the envelope calculation of spatial_size*num_samples as a fraction of the total input image volume, I'm only utilising around 40% of the available training data. If I try to include more data eg by increasing spatial_size it won't run!

I have seen papers where they've simply resized to a greatly reduced resolution for the entire workflow but would of course I would ideally like to use the available resolution to enable outputting of high quality segmentations.

Thanks for your help!

[rijobro]

Hm, what size are your images? Have you thought about reducing your batch size?

And what GPU are you using? You could consider using Google Colab if its GPU specs are better than then what you have access to.

[ghost]

Great questions, advice for optimising the training should have really been my opening post!

For training I have 26 CT volumes, original dimensions are 512x512x~400 but voxel dimensions vary so using those two crop transformations reduces them to approx 350x425x300.
I'm using a batch size of 1.

GPU availability is certainly a major potential limitation. In my local system I just have a GT 730(!) and in some instances found it faster to revert to the CPU. I've started using HPC facilities which could do the trick (K20x and P100) but have struggled with support from the local team to interface utilising the GPUs so I'm embarased to admit that at the moment I've been running on the HPC not using the GPUs. Now I write that down I realise it should be a priority of mine to get it working! This is presumably potentially a pretty big bottleneck to enabling me to run these at the higher resolution?

Good idea trying Colab, I'll give that a go too.

Any other feedback or ideas would be appreciated. This is my first foray both in ML and even coding so if there are other obvious parameters to explore I'd really appreciate any advice!

[rijobro]

Yeah, 350x425x300 is quite a big. If you check our 3D spleen segmentation tutorial, we use 96x96x96, still with batch=1. What are you trying to segment? Because it seems like it occupies >60% of your image (if you crop from ~512 to ~300).

One thing to bear in mind is that autoencoders are pretty good at upsampling images, so you might look into that if you really can't get the pixel spacing that you want directly. We've got an implementation of an AE in MONAI.

If you have access to powerful GPUs then it definitely is worth the effort getting them set up, they'll likely save you time in the long run!

[ghost]

Thank you so much Richard!

I'm actually pleased to hear that the dimensions are comparatively large and so to face computational issues isn't crazy.
I infact based the whole workflow on the 3D spleen tutorial and simply adapted it to my use case. The aim is to segment the pulmonary artery which in fact only occupies ~5% of the volume. As in the spleen example for cropforeground I'm using source_key="image", so from my understanding it crops to the edges of image components rather than the segmentation location?

Any other tips to also reduce computational requirements apprecited!

Thank you, I'l look at autoencoders. And yes, you're right I work to access the GPUs I have available!!

Thanks again!