-
Notifications
You must be signed in to change notification settings - Fork 291
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support bfloat16 execution #1121
Comments
To add another potential related data point of this, I converted SantaCoder ( a float16 trained model) to ctranslate2 for inference, when combined with
Converted model locates here: https://huggingface.co/TabbyML/SantaCoder-1B/tree/main/ctranslate2 The reported issue: TabbyML/tabby#236 |
The linked PR is adding support for bfloat16. Just like other types, you can select it during conversion with It would be helpful if anyone watching this issue can test the implementation and give feedback. To install the development build:
Note that a GPU with Compute Capability 8 or greater is required. |
Models that are trained with bfloat16 can have numerical issues when run with float16. See #1074 for example.
We should consider supporting bfloat16 execution which is supported on recent Intel CPUs and NVIDIA GPUs.
The text was updated successfully, but these errors were encountered: