OPENAI_BASE_URL instead of OPENAI_API_HOST

[johnd0e]

Right now /v1 is part of ENDPOINT.
But it is more common to have BASE_URL (or API_BASE) setting, which include /v1 part as well: https://api.openai.com/v1.
Btw, this is so for GOOGLE_API_HOST, but not for others.

The reason: there are some services, which do not contain /v1 part at all
E.g. https://models.inference.ai.azure.com/chat/completions

Ref: https://docs.github.com/en/github-models/prototyping-with-ai-models

[mountaineerbr]

That is great you brought up this subject. Indeed, I have struggled to understand how to try and make this feature the most compatible across multiple service providers. I will check if we can make this naming convention more correct and also check if we can have /v1 moved to API_HOST variable... However, I remember to have tried doing this once but soon gave up because it would need changing lots of code instances... Thank you so much for clarifying this issue and giving me some ref pointer as I was completely lost about how to implement this right!

[mountaineerbr]

It seems to be possible to correct both these issues without too many changes in the code of the script at this time...

Even though we have updated the script to solve these issues today, we need some testing before release. I also found some more bugs (minor bugs) which are fixed now, however I need to test at least all the seven service provider configurations and functions before public release.

I am not sure if I can create an account to use with Azure yet, but I will see it in the next few days so I can test your example server. In any case, I will do my best to make this script more compatible with other services out-of-the-box.
Cheers!

[johnd0e]

am not sure if I can create an account to use with Azure yet

It is github account. But you need to create special token.

[mountaineerbr]

Doing some research, according to ref1 and ref2, baseUrl in our case would be exactly /v1.

This means API_HOST points to the provider website root address, while BASE_URL would mean /v1.

As I tested chatgpt.sh script, it seems some service providers just ignore the /v1 part of the url path, so it even works with Mistral and GitHub Models APIs. For now!

As our script targets OpenAI API version 1, I reckon we shall not be fixing this seemingly issue.. If unlikely the project decides to move on and support a future /v2 past version 1 of OpenAI API, we may reconsider this issue ticket!

As for records, the script in the present version (v0.81.0) already works with GitHub Models. The syntax for it to work with GitHub Models would be:

OPENAI_API_HOST="https://models.inference.ai.azure.com" chatgpt.sh -cc -m Phi-3-small-8k-instruct --api-key $GITHUB_TOKEN

The workaround (which already exists) for the script to work with a service provider which fails when /v1 is set automatically is to set the following envar instead:

OPENAI_API_HOST_STATIC="https://models.inference.ai.azure.com/chat/completions"

The STATIC parameter disables the script auto-selection of endpoints!

Lastly, if the user creates a configuration file from the template (option -FF), there are examples of how to set these parameters in it! And also, it is possible to wrap things even further with bash aliases which may define different configuration files for each use case (using envars for the config filepath in this case), if hard-setting these variables in the configuration file stands in the way of the user using more than one service provider!

---

The syntaxes above are described in the man page and in the Repository README.md.

Therefore, it is unlikely that this issue ticket is going to be addresses because I don't see it as a real issue in our code context at the moment.

Even though, I note that there is around ~4 times more code mentions of OPENAI_BASE_URL than OPENAI_API_HOST when I search these terms in GitHub searching mechanism...

---

Most fortunately, I did find that the GitHub Models service is great and so I went ahead and implemented the command line option --github, which should set up the script. With this, available models are listed with option -l and the current GitHub Models maximum token capacity are set correctly!

Just remember to have $GITHUB_TOKEN set and exported as an environment variable!

This is going to be available in version v0.81.1 which shall be released when I finish testing the new API service provider...

[mountaineerbr]

Just in case you can help me, do you know any reliable way to dump GitHub Models as a list? I am using this for now:

curl "https://github.com/marketplace/models" |
sed -n 's/"original_name":"[^"]*",/\n&\n/gp' | sed -n 's/"original_name"://p' | sed 's/[",]//g'

And ALSO, when we try the /info endpoint and run:

curl https://models.inference.ai.azure.com/info\?model=Phi-3-small-8k-instruct -X POST -H 'Content-Type: application/json' -H 'Authorization: Bearer xoxoxoxo'

It returns:

{"error":{"code":"no_model_name","message":"No model specified in request. Please provide a model name in the request body or as a x-ms-model-mesh-model-name header.","details":null}}

[johnd0e]

curl -s https://models.inference.ai.azure.com/models \
    -H "Authorization: Bearer github_pat_xxx" \
   | jq .[].name

AI21-Jamba-Instruct
Cohere-command-r
Cohere-command-r-plus
Cohere-embed-v3-english
Cohere-embed-v3-multilingual
Meta-Llama-3-70B-Instruct
Meta-Llama-3-8B-Instruct
Meta-Llama-3.1-405B-Instruct
Meta-Llama-3.1-70B-Instruct
Meta-Llama-3.1-8B-Instruct
Mistral-large
Mistral-large-2407
Mistral-Nemo
Mistral-small
gpt-4o
gpt-4o-mini
text-embedding-3-large
text-embedding-3-small
Phi-3-medium-128k-instruct
Phi-3-medium-4k-instruct
Phi-3-mini-128k-instruct
Phi-3-mini-4k-instruct
Phi-3-small-128k-instruct
Phi-3-small-8k-instruct
Phi-3.5-mini-instruct

[johnd0e]

This means API_HOST points to the provider website root address, while BASE_URL would mean /v1.

It's not about proper terminology.

What I'm trying to convey is that most projects use https://api.openai.com/v1 as the configurable part that defines the API. There are real APIs that do not include the v1 part; some may use v2. This is not an issue, as all these cases can be addressed with a single option.

Of course, there's no practical reason to customize the host and base separately. Therefore, the main reason for continuing to use the current API_HOST setting is compatibility, to avoid disrupting existing user setups.

Therefore, it is unlikely that this issue ticket is going to be addresses because I don't see it as a real issue in our code context at the moment.

Well, you are the boss.

[mountaineerbr]

I really hope this project remains stable before /v2, and dies! It is too much code,and the AI models aren't that really worth the effort sometimes...

[johnd0e]

I really hope this project remains stable before /v2

It's not about api.openai.com compatibility.
It is about support of 3rd-party "openai-compatible" APIs, which may have different base URLs right now.
I have seen even '/v1/openai' for example.