When fine-tuning llama-7b, approximately how much GPU memory is required for training?

[zty07]

When fine-tuning llama, approximately how much GPU memory is required for training?

[MJ10]

Hi @zty07, sorry for the extremely late response. Could you please clarify which experiment you are interested in running? The memory would depend on the task (specifically the sequence length). The quantization code is somewhat broken unfortunately but will be fixed soon which should help with lowering the memory requirements.

[isaacbmiller]

@MJ10 Did the quantization code ever get fixed?

[abdalgader-a]

@MJ10 -- running the next sentence code with 2B/3B parameter size model thrown OOM? any suggestion to resolve?
(PS: I used A100 80GB 8GPUs)

[isaacbmiller]

@abdalgader-a I have managed to get it running on a single A100, but my num_samples is way less than 20