Fail to read data with different num_coil.

[NayeeC]

Hi,
I find a strange problem here, I set the batch=1 and suppose there are four data with different num_coil: A: num_coil=16, B: num_coil=16, C: num_coil=18, D: num_coil=20.
Then I use

data_module = FastMriDataModule(
        data_path=args.data_path,
        challenge=args.challenge,
        train_transform=train_transform,
        val_transform=val_transform,
        test_transform=test_transform,
        combine_train_val=False,  # True for org
        test_split=args.test_split,
        test_path=args.test_path,
        sample_rate=args.sample_rate,
        batch_size=args.batch_size,
        num_workers=args.num_workers,
        distributed_sampler=(args.accelerator == "ddp"),
        use_dataset_cache_file=False,  # when true training, delete this
    )

to read the data and train. And the following error occurs when the data D: num_coil=20 is in the folder, which contains data.

Traceback (most recent call last):
  File "F:/ch/fastMRI/fastmri_examples/varnet/varnet_brain_nopl.py", line 303, in <module>
    main()
  File "F:/ch/fastMRI/fastmri_examples/varnet/varnet_brain_nopl.py", line 299, in main
    main_loop(args)
  File "F:/ch/fastMRI/fastmri_examples/varnet/varnet_brain_nopl.py", line 110, in main_loop
    output = model(masked_kspace, mask)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\parallel\data_parallel.py", line 168, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\parallel\data_parallel.py", line 178, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\parallel\parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\_utils.py", line 425, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\parallel\parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\fastmri\models\varnet.py", line 249, in forward
    kspace_pred = cascade(kspace_pred, masked_kspace, mask, sens_maps)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\fastmri\models\varnet.py", line 293, in forward
    self.model(self.sens_reduce(current_kspace, sens_maps)), sens_maps
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\fastmri\models\varnet.py", line 129, in forward
    x = self.unet(x)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\fastmri\models\unet.py", line 102, in forward
    output = torch.cat([output, downsample_layer], dim=1)
RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 11.00 GiB total capacity; 9.32 GiB already allocated; 12.50 MiB free; 9.38 GiB reserved in total by PyTorch)

Does this mean that the num_coil should be larger than 20?
And if the data are arranged like A B C or A C B, as long as the data C: num_coil=18 is not on the top of all other data, C A B, the following error occurs:

Traceback (most recent call last):
  File "F:/ch/fastMRI/fastmri_examples/varnet/varnet_brain_nopl.py", line 303, in <module>
    main()
  File "F:/ch/fastMRI/fastmri_examples/varnet/varnet_brain_nopl.py", line 299, in main
    main_loop(args)
  File "F:/ch/fastMRI/fastmri_examples/varnet/varnet_brain_nopl.py", line 110, in main_loop
    output = model(masked_kspace, mask)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\parallel\data_parallel.py", line 168, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\parallel\data_parallel.py", line 178, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\parallel\parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\_utils.py", line 425, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\parallel\parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\fastmri\models\varnet.py", line 249, in forward
    kspace_pred = cascade(kspace_pred, masked_kspace, mask, sens_maps)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\fastmri\models\varnet.py", line 292, in forward
    model_term = self.sens_expand(
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\fastmri\models\varnet.py", line 275, in sens_expand
    return fastmri.fft2c(fastmri.complex_mul(x, sens_maps))
  File "D:\Anaconda3\envs\chtorch2\lib\site-packages\fastmri\fftc.py", line 90, in fft2c_new
    torch.fft.fftn(  # type: ignore
RuntimeError: CUDA out of memory. Tried to allocate 30.00 MiB (GPU 0; 11.00 GiB total capacity; 8.99 GiB already allocated; 28.50 MiB free; 9.37 GiB reserved in total by PyTorch)

And, this occurs at the moment that the network tries to load data C.

Why do these strange things happen? My GPU0 has 11264Mib memory, is this still too small to load the data? If so, how can I deal with it without changing my GPU?

[mmuckley]

Hello @NayeeC, yeah I think your GPU might be too small. You can decrease the size of the model by decreasing the number of layers or channels. I think for the leaderboard models we use 32 GB GPUs.