New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run chat-model on secondary GPU? #721

Closed

PeterTucker opened this issue Nov 8, 2023 · 1 comment

Labels

enhancement fixed-in-next-release

PeterTucker commented Nov 8, 2023 •

edited

Loading

Please describe the feature you want
Any way to run chat-model on a secondary GPU?

Additional context
Add any other context or screenshots about the feature request here.

Please reply with a 👍 if you want this feature.

PeterTucker added the enhancement label

Member

wsxiaoys commented Nov 23, 2023

Hey, could you test this feature in 0.6.0? You could register a separate worker process for chat model with following command

tabby worker::chat --model TabbyML/Mistral-7B  --url <IP of webserver node>:8080 --token <registeration token, can be found in webserver node's dashboard>

wsxiaoys added the fixed-in-next-release label

wsxiaoys closed this as completed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment