[Warning] Merge lora module to 4-bit linear may get different generations

[steveepreston]

System Info

peft 0.14.0
transformers 4.48.0
bitsandbytes 0.45.0

Who can help?

@BenjaminBossan @sayakpaul

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder
• My own task or dataset (give details below)

Reproduction

code:

base_model_id = "gemma-2-27b-it"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_storage=torch.bfloat16,
)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=quantization_config,
    attn_implementation="sdpa",
    torch_dtype=torch.bfloat16,
    use_cache=True,
)

peft_model = PeftModel.from_pretrained(base_model, adapter_path)

--> merged_model = peft_model.merge_and_unload()

Warning:

UserWarning: Merge lora module to 4-bit linear may get different generations due to rounding errors.

Expected behavior

merge_and_unload() correctly and without warning.