Update chatqna.py to support vLLM embeddings

[raravena80]

Description

Changes needed to support ARM without rerank with specific embeddings endpoint input/output of vllm

Need to optionally set

export EMBEDDINGS_USE_VLLM=true

Re-opening #1232 due to DCO failing

Issues

n/a

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)
• Others (enhancement, documentation, validation, etc.)

Dependencies

List the newly introduced 3rd party dependency if exists.

Tests

Test locally with API successfully

curl http://localhost:8888/v1/chatqna     -H "Content-Type: application/json"     -d '{
        "messages": "What is the revenue of Nike in 2023?"
    }'
data: b''

data: b' As'

data: b' of'

data: b' now'

data: b','

data: b' the'

data: b' exact'

data: b' revenue'

data: b' of'

data: b' N'

data: b'ike'

data: b' for'

data: b' '

data: b'2'

data: b'0'

data: b'2'

data: b'3'

data: b' is'

data: b' not'

data: b' available'

data: b'.'

data: b' However'

data: b','

data: b' you'

data: b' can'

data: b' find'

data: b' the'

data: b' latest'

data: b' revenue'

data: b' information'

data: b' by'

data: b' searching'

data: b' for'

data: b' their'

data: b' financial'

data: b' reports'

data: b' or'

data: b' visiting'

data: b' their'

data: b' official'

data: b' website'

data: b'.'

data: b''

data: b''

data: [DONE]

[raravena80]

@lvliang-intel @mkbhanda need a reviewer for this PR

[eero-t]

I think that in addition to vLLM embedding support:
https://docs.vllm.ai/en/latest/models/supported_models.html#text-embedding-task-embed

This should include also reranking support:
https://docs.vllm.ai/en/latest/models/supported_models.html#sentence-pair-scoring-task-score

Otherwise the required changed cannot be properly reviewed.

(E.g. why potential graph nodes are not just declared once, and routing between them done based on the args, instead of everything being copy-pasted 4 times, and assumable one more copy-paste with reranker...)

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files

[lvliang-intel]

We are refactoring the GenAIComps embedding code to ensure full compatibility with the OpenAI format. With this update, there is no need to differentiate vLLM in this context. All embedding requests should use the endpoint /v1/embeddings and include the model parameter.

[raravena80]

thanks

[eero-t]

We are refactoring the GenAIComps embedding code to ensure full compatibility with the OpenAI format. With this update, there is no need to differentiate vLLM in this context. All embedding requests should use the endpoint /v1/embeddings and include the model parameter.

I found following 3 PRs:

• Define embedding/ranking/llm request/response format GenAIComps#289
• compatible with openai/tgi/vllm request format GenAIComps#275
• update tei embedding format. GenAIComps#1035

Are there (going to be) more?