Extend DeepSpeed inference initialization API with a 'quantize_groups' argument

[sakogan]

This PR allows setting the number of quantization groups to be used during inference for the given model weights. Once merged, one should be able to control this number by passing the quantize_groups argument to a deepseed.init_inference call, e.g., like is done in this inference test script: microsoft/DeepSpeedExamples@master...sakogan:DeepSpeedExamples:inference-test-enhance
(Note that the latter example will be submitted to the DeepSpeedExamples repo as a separate PR if this PR is merged.)

Setting 'quantize_groups' is allowed only when the quantization is enabled, i.e., when dtype is set to int8.

[RezaYazdaniAminabadi]

Hi @sakogan,

The changes in this PR looks good to me. May I ask that you link the PR on the example side so that I know how to use this feature?
Thanks,
Reza

[sakogan]

thanks for the review @RezaYazdaniAminabadi. I will submit the corresponding PR into the examples repo shortly.

[sakogan]

@RezaYazdaniAminabadi I opened a PR #713 on the examples repo that adds the corresponding argument to the inference test script

[lekurile]

Were you able to test this change locally with a model?

I see your PR in DeepSpeedExamples as well, which looks good to me. Could you please provide a reproduction command?

Thanks,
Lev

[sakogan]

Yes. Here is an example command:
deepspeed /path/to/DeepSpeedExamples/inference/huggingface/text-generation/inference-test.py --batch_size 1 --use_kernel --use_meta_tensor --dtype int8 --model bigscience/bloom-7b1 --quantize_groups 32

[mrwyattii]

I believe you are adding quantize_groups as a shortcut for the admittedly convoluted current quantize config settings? For example, with this you could just pass quantize_groups=2 rather than quant={"weight":{"q_groups":2}}.

But perhaps we should look into how we can simplify the quantize settings or at the very least add this logic to the DeepSpeedInferenceConfig class as a pydantic validator (so that the config logic is consolidated there).

[sakogan]

Yes, I suppose one could use a config file with quant={"weight":{"q_groups":2}} instead. What I am suggesting here is a simple way to control this setting from a command line. I do agree that there might be better ways of achieving that than special-casing this argument in init_inference

[kiucho]

Hi, does deepspeed support int8 inference? With next command I get following error
Command: deepspeed --num_gpus 1 inference-test.py --use_kernel --dtype int8 --model EleutherAI/gpt-j-6b
Error: !!!! kernel execution error. (m: 4096, n: 3, k: 12288, error: 15)