is textencoder Quantized Diffusers models can be saved and loaded?

[lonngxiang]

No description provided.

[sayakpaul]

They can be. Follow this: https://github.com/huggingface/optimum-quanto?tab=readme-ov-file#llm-models

For the unimplemented classes, you can just refer to

optimum-quanto/optimum/quanto/models/transformers_models.py

Line 171 in 601dc19

class QuantizedModelForCausalLM(QuantizedTransformersModel):

And send us a PR.

[lonngxiang]

They can be. Follow this: https://github.com/huggingface/optimum-quanto?tab=readme-ov-file#llm-models

For the unimplemented classes, you can just refer to

optimum-quanto/optimum/quanto/models/transformers_models.py

Line 171 in 601dc19

class QuantizedModelForCausalLM(QuantizedTransformersModel):

And send us a PR.

how PixArt-alpha/PixArt-Sigma-XL-2-1024-MS model use save and load text Quantized model

[Ednaordinary]

They can be. Follow this: https://github.com/huggingface/optimum-quanto?tab=readme-ov-file#llm-models
For the unimplemented classes, you can just refer to

optimum-quanto/optimum/quanto/models/transformers_models.py

Line 171 in 601dc19

class QuantizedModelForCausalLM(QuantizedTransformersModel):

And send us a PR.

how PixArt-alpha/PixArt-Sigma-XL-2-1024-MS model use save and load text Quantized model

This is shown in the guide
https://huggingface.co/blog/quanto-diffusers

[Ednaordinary]

Also, @sayakpaul, is there a way to load models without entirely requantizing them? I'm trying to load flux from a quantized save (made a class), but it takes a while due to the requantization (I think), to the point where there's no real reason to not just load normally and quantize on the fly

[sayakpaul]

Will let @dacorvo comment on that. But he is on vacation so expect delay.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[dacorvo]

@Ednaordinary some recent pull-requests might have fixed the long requantization times (see #290).

[Ednaordinary]

@dacorvo Nice! Something along the way must have also changed how the first move between devices works because I'm seeing a lot faster transfer between CPU and GPU on the first move. Thanks for your hard work!

[dacorvo]

Something along the way must have also changed how the first move between devices works because I'm seeing a lot faster transfer between CPU and GPU on the first move.

This must be #291.

Thanks for your hard work!

Kudos to @latentCall145 !