AttributeError: 'NoneType' object has no attribute 'to' when using wanda pruning

[NamburiSrinath]

Hi @Eric-mingjie,

I am also facing the same issue (as [#51]) when trying to prune llama-2-7b-chat-hf

Here's the command

python main.py --model meta-llama/Llama-2-7b-chat-hf --prune_method wanda --sparsity_ratio 0.5 --sparsity_type unstructured --save out/llama_7b/unstructured/wanda/0.5/

And the attention_mask seems to be None

These are the kwargs in the prepare_calibration_input function

{'attention_mask': None, 'position_ids': tensor([[ 0, 1, 2, ..., 4093, 4094, 4095]], device='cuda:0'), 'past_key_value': None, 'output_attentions': False, 'use_cache': False, 'cache_position': tensor([ 0, 1, 2, ..., 4093, 4094, 4095], device='cuda:0')}

Any idea how to go forward with this?

[NamburiSrinath]

Update

The same issue arises when compressing using sparsegpt as well.

python main.py --model 'meta-llama/Llama-2-7b-chat-hf' --prune_method sparsegpt --sparsity_ratio 0.1 --sparsity_type unstructured --save out/llama_7b/unstructured/sparsegpt/0.1/ --save_model out/llama_7b/unstructured/sparsegpt/0.1/

Note: Similar issue is observed even with llama-2-7b-hf i.e the base model!

The only change that was made was related to #36.

Tagging coauthor @liuzhuang13 to see if something is broken.

[Logan-007L]

hello, Have you solved the problem？

[NamburiSrinath]

@Logan-007L - Please refer #51

@Eric-mingjie, @liuzhuang13. Closing this as it resolved the issue. But I am not sure if it's technically correct though!! If so, would be helpful to make some code changes as Llama-2 is a popular model (even the paper reports results using this model)

[Logan-007L]

Thanks. I'll have a try