Newbie question: Serving an OpenAI compatible API endpoint

[olafgeibig]

Hi, I own a M1 Mac but I didn't dive into mlx so far. I'm into developing agentic software and the agent frameworks usually allow to utilize an OpenAI compatible API endpoint as the LLM for agents. I did my research but I could not find a way how to do that for a mlx model. Did I miss something?

I find it quite interesting to fine-tune my own models for my agent use cases. Thank you!

[awni]

You didn't miss anything. Building a server around an MLX model would be super cool, but is definitely outside the scope of this repo.

It should be quite doable to wrap an MLX LLM in a python based web server like flask and make calls via the browser. Would love to see that!

[deepio-oc]

@olafgeibig you can check out this: https://github.com/ml-explore/mlx-examples/blob/main/llms/README.md and https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/SERVER.md