-
Notifications
You must be signed in to change notification settings - Fork 58
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Missing key "lm_head.weight" in GemmaForCausalLM when loading lora finetuned TinyLLaVA-Gemma-SigLIP-2.4B #88
Comments
This just happened to me also. After the pre-training phase I am trying to perform inference however seems there is a miss-match between the saved model on |
A quick work around for now would be to follow the solkution made on the issue vllm-project/vllm#3323 linked by @Yuki-Kokomi, by copying the language_model_ckp_path = os.path.join(model_name_or_path, 'language_model/pytorch_model.bin')
language_model_ckp = load_base_ckp_for_lora(language_model_ckp_path)
# This line is what does the trick
language_model_ckp['lm_head.weight'] = language_model_ckp['model.embed_tokens.weight']
model.language_model.load_state_dict(language_model_ckp) However, I am not able to qualitatively validate that this fix works fine. |
Thank you @ggcr for the solution. It works perfectly. |
I can provide a PR by doing this checking if the LLM backbone used is Gemma if the authors want. |
Hi, we do encourage you to initiate a PR! |
When attempting to merge LoRA weights into the TinyLLaVA-Gemma-SigLIP-2.4B model, I encountered a RuntimeError due to a missing key lm_head.weight in the GemmaForCausalLM state_dict. The specific error traceback is as follows:
This issue seems similar to vllm-project/vllm#3323. Any insights or solutions to resolve the missing "lm_head.weight" key would be greatly appreciated.
The text was updated successfully, but these errors were encountered: