GPU memory comsumption increases every epoch during training

[jhkwag970]

Hello,

Thank you for sharing your work!

As I am training on ImageNet1K, I noticed that memory consumption increases by approximately 254MB with each epoch. If this trend continues, the total memory usage will reach 254MB * 300 = 76.2GB.

Is this the intended behavior?

Thank you!

[ahatamiz]

Hi @jhkwag970

Could you please provide more details ? specifically whether you are using this version of the codebase or another.

I have not faced this issue before in any of the training runs.

Best

[jhkwag970]

@ahatamiz
Hello, Thank you for your response. I am using the current MambaVision repo. After validation and reloading memory for training, memory consumption is higher than in the previous epoch. I tried using torch.cuda.empty_cache() as an alternative solution for now. I just wanted to make sure this will not cause any problems during training.