Add vllm custom model support for OpenAI compatibility

[Navanit-git]

Hi,
Is there a way where we can add the vLLM OpenAI compatibility support.
vllm OpenAI Support

So that anyone can use any llm model calls.

[samuelcolvin]

Should work the same as ollama, see the code here - #112 (comment).

Happy to consider adding vllm as another custom model, but it would need more people to want it before we do the work.

[samuelcolvin]

See #239, that would mean we could add VLLMModel.

[daavoo]

See #239, that would mean we could add VLLMModel.

Ola! I was testing pydantic.ai alongside vLLM and llama.cpp server, which I think both fulfill the rules for adding a new model.

I have looked at the existing Ollama code and I am not sure I understand the value of adding a new VLLMModel.
Looks like there is no custom logic (beyond providing a default api_key value) and provides an ~arbitrary hardcoded list of model names (which don't cover all available models)

I got this same snippet working with both vLLM and llama.cpp server out of the box:

# For example with llama.cpp:
docker run -v ./models:/models -p 8080:8080 \
ghcr.io/ggerganov/llama.cpp:server -m /models/smollm2-360m-instruct-q8_0.gguf

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel

model = OpenAIModel(
    "mymodel", 
    base_url="http://localhost:8080/v1", 
    api_key="foo"
)

agent = Agent(  
    model=model,
    system_prompt='Be concise, reply with one sentence.',  
)

result = agent.run_sync('Where does "hello world" come from?')  
print(result.data)

So, is it better to just send a small documentation patch?

pd. I don't have a problem on contributing a new VLLM/LLAMACPP model myself, just wondering if it makes sense to keep adding those

[sadransh]

@daavoo This wouldnt work with tool calling and non str result type. you can re-use the example from here: to double check #398