Skip to content

Commit

Permalink
Merge branch 'main' into config_update_tests
Browse files Browse the repository at this point in the history
  • Loading branch information
Sara Adkins authored Jun 3, 2024
2 parents 15c995c + 5caa557 commit 53ac12d
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions examples/llama7b_sparse_quantized/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ run the following:

```
import torch
from sparseml import SparseAutoModelForCausalLM
from sparseml.transformers import SparseAutoModelForCausalLM
model = SparseAutoModelForCausalLM.from_pretrained(output_dir, torch_dtype=torch.bfloat16)
model.save_pretrained(compressed_output_dir, save_compressed=True)
Expand All @@ -49,4 +49,4 @@ model.save_pretrained(compressed_output_dir, save_compressed=True)
### Custom Quantization
The current repo supports multiple quantization techniques configured using a recipe. Supported strategies are `tensor`, `group` and `channel`.
The above recipe (`2:4_w4a16_recipe.yaml`) uses channel-wise quantization specified by `strategy: "channel"` in its config group.
To use quantize per tensor, change strategy from `channel` to `tensor`. To use group size quantization, change from `channel` to `group` and specify its value, say 128, by including `group_size: 128`. Group size quantization example is shown in `2:4_w4a16_group-128_recipe.yaml`
To use quantize per tensor, change strategy from `channel` to `tensor`. To use group size quantization, change from `channel` to `group` and specify its value, say 128, by including `group_size: 128`. Group size quantization example is shown in `2:4_w4a16_group-128_recipe.yaml`

0 comments on commit 53ac12d

Please sign in to comment.