Skip to content

Commit

Permalink
LLM micro service extract input model name (opea-project#447)
Browse files Browse the repository at this point in the history
No input model from UI.
vLLM and OLLAMA get the model name from the envrionment
TGI mode model form ModelID

Signed-off-by: Wang, Xigui <[email protected]>
  • Loading branch information
xiguiw authored and BaoHuiling committed Aug 15, 2024
1 parent c2ce13a commit f543c20
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion comps/llms/text-generation/ollama/llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
def llm_generate(input: LLMParamsDoc):
ollama = Ollama(
base_url=ollama_endpoint,
model=input.model,
model=input.model if input.model else model_name,
num_predict=input.max_new_tokens,
top_k=input.top_k,
top_p=input.top_p,
Expand Down Expand Up @@ -49,4 +49,5 @@ async def stream_generator():

if __name__ == "__main__":
ollama_endpoint = os.getenv("OLLAMA_ENDPOINT", "http://localhost:11434")
model_name = os.getenv("OLLAMA_MODEL", "meta-llama/Meta-Llama-3-8B-Instruct")
opea_microservices["opea_service@llm_ollama"].start()

0 comments on commit f543c20

Please sign in to comment.