Questions about memory consumption.

[Wxsxulin]

hank you for your work.

I have a few questions about your paper related to the memory consumption of the training.

1.What kind/How many GPUS did you use in your experiments?
2.How many GB of GPU memory do you use at each step?
3.May I ask you how much time in hours/days lasts the full training

Thanks in advance.

[dengxl0520]

1. single 3090 GPU
2. about 18G
3. 100epochs took about 19 hours, CAMUS dataset.

[Wxsxulin]

1. 单个 3090 GPU
2. 约18G
3. 100epochs大约需要19个小时，CAMUS数据集。

thanks a lot

[jasonrichdarmawan]

My lab has 3090 GPU, but I cannot train the model

python train_video.py --modelname MemSAM --task CAMUS_Video_Semi
Namespace(modelname='MemSAM', encoder_input_size=256, low_image_size=256, task='CAMUS_Video_Semi', vit_name='vit_b', sam_ckpt='checkpoints/sam_vit_b_01ec64.pth', batch_size=1, n_gpu=1, epochs=100, base_lr=0.0001, warmup=False, warmup_period=250, keep_log=False, disable_memory=False, disable_reinforce=False, disable_point_prompt=False)
Single object mode: False
Total_params: 58047159
Segmentation fault

[xbbbbblv99]

Namespace(modelname='MemSAM', encoder_input_size=256, low_image_size=256, task='CAMUS_Video_Full', vit_name='vit_b', sam_ckpt='checkpoints/sam_vit_b_01ec64.pth', device='cuda', epoch=100, batch_size=2, n_gpu=1, base_lr=0.005, warmup=False, warmup_period=250, keep_log=False, frame_length=10, point_numbers=1, enable_memory=False, semi=False, reinforce=False, disable_point_prompt=False)

but

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 318.00 MiB. GPU 0 has a total capacity of 23.68 GiB of which 266.69 MiB is free. Process 787029 has 23.42 GiB memory in use. Of the allocated memory 22.33 GiB is allocated by PyTorch, and 802.83 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)