training with lora？

[xiexing0916]

Is there a way (lora?) to train with less memory burden? Full fine-tuning is not trainable at 48G of RAM.

[ChrisLiu6]

The released codes do not support lora yet, but it is easy to implement and you can do it by yourself. You may also implement the get_trainable_params method for the model class to only train part of the model parameters.

Lumina-mGPT/xllmx/solvers/finetune/finetune.py

Line 273 in 104abe4

if hasattr(unwrapped_model, "get_trainable_params"):

The follows issues may be helpful:

#17
#18

[xiexing0916]

The released codes do not support lora yet, but it is easy to implement and you can do it by yourself. You may also implement the get_trainable_params method for the model class to only train part of the model parameters.

Lumina-mGPT/xllmx/solvers/finetune/finetune.py

Line 273 in 104abe4

if hasattr(unwrapped_model, "get_trainable_params"):

The follows issues may be helpful:

#17 #18

Thanks for the answer! I would also like to ask how much memory is needed to fine tune all the parameters? I currently can't work with 3 48G A6000.

[ChrisLiu6]

Thanks for the answer! I would also like to ask how much memory is needed to fine tune all the parameters? I currently can't work with 3 48G A6000.

We have not tested on the memory lower bound. 8*A100 * 80G should be enough, but the lower I have no idea😅.