LLM Docker server documentation. #35
Conversation
| LLM Endpoint: http://127.0.0.1:5000/v1 | ||
| LLM API Key: Can be blank, or anything at all. | ||
| ``` | ||
| So here is where things get a bit scuffed. Regardless of the model name you have loaded, you want to enter gpt-3.5-turbo because covas doesn't like anything else here. Also, when you click Start AI, Covas is going to complain about your model provider not servering gpt-4o-mini. Ignore it and continue anyway. It will still work fine, even with this nag. I don't understand it either. |
There was a problem hiding this comment.
What we do on covas:next side it a request to the models endpoint of the URL that was entered (https://platform.openai.com/docs/api-reference/models/list). The reason we do this, is because for openai the 4o model is sometimes not available when the account is very young. If detect that 4o-mini is not in that list, we downgrade the model to 3.5-turbo instead. But the catch is when using providers other than openai, those may not provide a models endpoint, in which case our request error and we just hope that what was entered as the model name is correct. What appears to happen in your case is that the text-gen webui does indeed provide a models endpoint, and this endpoint probably does not include gpt-4o-mini, so we show the error message, that gpt-4o-mini is not available, although it seems like the webui does not even care about what you entered, so this gets a bit misleading. One possibility to correct this behavior, is to manually check what the actual available model names are that the webui claims to support and then enter one of those, or to get the webui to state that it is indeed fine with providing a model called gpt-4o-mini...
There was a problem hiding this comment.
Could we make the check itself optional, depending on the entered LLM endpoint?
There was a problem hiding this comment.
The check is optional already, but we could disable the downgrade / upgrade if the provider is not openai
Adding documentation for a locally running LLM server.