New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fp8 support for inference #44

Open

JoeyYoung opened this issue Feb 25, 2025 · 1 comment

JoeyYoung commented Feb 25, 2025

Hi all,

Thanks for your work! I wanna achieve the fp8 inference on v3 model, is there any hints on how to do that or any plans to support it?

Collaborator

beginlner commented Feb 25, 2025

We are working on fp8 kv-cache.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment