llama3.2 fine tuning generates repeated pattern towards the end of one epoch #735
Comments
|
I get a similar observation on my fine-tuning on custom dataset. Did you plot your training loss with wandb? I wonder whether this is a learning rate adjustment. Also is there a possibility to schedule the learning rate decay? |
I printed to tensorboard, take a look at my loss below, I had to smooth it with 1.0 so you can see how it drops. I applied a cosine lr with 0.03 warm_up ratio, and 0.01 weight_decay. What I don't understand is why the model corrupted somewhere between 0.7 and 0.8 of the epoch.. 
|
|
Your loss seems to be fine, maybe train longer or increase learning rate? Repetitive answers usually mean that the model is still adapting to the new domain. |
Yup, those are probably the way out, just A100 rates are high and would like check if anyone has seen and solved the similar problem.
I added them manually. This is another thing wired about this repo, obviously almost all other fine-tuning repo has enbaled the cosine learning rate (idefics, intern, qwen, Aria etc.) but not this repo. It makes me worried fine-tuning script has not been well tested in this repo |
|
yeah I agree, my finetuned model performs worse than a smaller LLaVA-Onevision finetuned model on my custom dataset. The loss doesn't manage to go as much down. Let's see if there are any significant updates in the coming weeks |
System Info
PyTorch version: 2.4.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
GPU Type and number: A100 80GB x 1
Information
🐛 Describe the bug
GPUS=1 PER_DEVICE_BATCH_SIZE=2 nohup sh src/llama_lora_finetune.shError logs
I was fine tuning the
meta-llama/Llama-3.2-11B-Vision-Instructwith https://archive.physionet.org/mimic2/ with 170k image-text pairs. The checkpoints till 0.7 of one epoch generate output text as expected. But starting 0.8 epoch, the checkpoints and so forthgenerate a repeated pattern as belowExpected behavior
Expecting the model to generate normal output at 0.8 epoch training and after.
The text was updated successfully, but these errors were encountered: