Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ChatQnA - add files for deploy ChatQnA application on AMD ROCm with vLLM service #1181

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

chyundunovDatamonsters
Copy link
Contributor

Description

Add files for deploy ChatQnA application on AMD ROCm with vLLM service:

ChatQnA/docker_compose/amd/gpu/rocm-vllm/compose_vllm.yaml - Docker Compose file
ChatQnA/docker_compose/amd/gpu/rocm-vllm/set_env_vllm.sh - set envs script
ChatQnA/docker_compose/amd/gpu/rocm-vllm/Dockerfile-vllm - Dockerfile for build vLLM Docker image for service on ROCm
ChatQnA/docker_compose/amd/gpu/rocm-vllm/README.md - README file

Issues

It was required to be able to deploy the ChatQnA application on AMD hardware using the vLLM Service

Type of change

  • [* ] New feature (non-breaking change which adds new functionality)

Dependencies

Tests

Testing is performed by a script GenAIExamples/ChatQnA/tests/test_compose_on_rocm_vllm.sh

Chingis Yundunov added 2 commits November 22, 2024 00:11
…LLM service:

1. ChatQnA/docker_compose/amd/gpu/rocm-vllm/compose_vllm.yaml
2. ChatQnA/docker_compose/amd/gpu/rocm-vllm/set_env_vllm.sh
3. ChatQnA/docker_compose/amd/gpu/rocm-vllm/Dockerfile-vllm
4. ChatQnA/docker_compose/amd/gpu/rocm-vllm/REAMDE.md.
5. ChatQnA/tests/test_compose_on_rocm_vllm.sh
Fix build.yaml and playwright.config.ts

Signed-off-by: Chingis Yundunov <[email protected]>
@joshuayao joshuayao linked an issue Nov 22, 2024 that may be closed by this pull request
5 tasks
@@ -0,0 +1,18 @@
FROM rocm/vllm:rocm6.2_mi300_ubuntu20.04_py3.9_vllm_0.6.4
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's put the dockerfile under GenAIExamples/ChatQnA/, it can be named as Dockerfile.vllm_rocm.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good. I will place the file according to the proposed path and adapt the scripts

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chyundunovDatamonsters will you continue working on this PR?

@chensuyue
Copy link
Collaborator

chensuyue commented Nov 27, 2024

Let use one folder GenAIExamples/ChatQnA/docker_compose/amd/gpu/rocm/ to contain all the implementation on rocm, like this one https://github.com/opea-project/GenAIExamples/tree/60871f2622001a42de050f5606de22072d905fa6/ChatQnA/docker_compose/intel/hpu/gaudi.

@@ -0,0 +1,265 @@
#!/bin/bash
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ChatQnA/tests/test_compose_on_rocm_vllm.sh --> ChatQnA/tests/test_compose_vllm_on_rocm.sh
The naming format will impact CI trigger.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this file name, rocm_vllm will be recognized as a hardware label, but we just need rocm. So it need to be update like what I commented.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accepted. Let's put everything in one place for the Arm architecture.

Chingis Yundunov added 7 commits December 6, 2024 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Examples on AMD ROCm for OPEA v1.2
3 participants