Migration suggestion

ChatQnA/kubernetes/helm-chart/Chart.yaml

@@ -0,0 +1,41 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+apiVersion: v2
+name: chatqna
+description: The Helm chart to deploy ChatQnA
+type: application
+dependencies:
+  - name: tgi
+    version: 0-latest
+    alias: tgi-guardrails
+    repository: "oci://ghcr.io/opea-project/charts/tgi"
+    condition: guardrails-usvc.enabled
+  - name: guardrails-usvc
+    version: 0-latest
+    repository: "oci://ghcr.io/opea-project/charts/guardrails-usvc"
+    condition: guardrails-usvc.enabled
+  - name: tgi
+    version: 0-latest
+    repository: "oci://ghcr.io/opea-project/charts/tgi"
+  - name: tei
+    version: 0-latest
+    repository: "oci://ghcr.io/opea-project/charts/tei"
+  - name: teirerank
+    version: 0-latest
+    repository: "oci://ghcr.io/opea-project/charts/teirerank"
+  - name: redis-vector-db
+    version: 0-latest
+    repository: "oci://ghcr.io/opea-project/charts/redis-vector-db"
+  - name: retriever-usvc
+    version: 0-latest
+    repository: "oci://ghcr.io/opea-project/charts/retriever-usvc"
+  - name: data-prep
+    version: 0-latest
+    repository: "oci://ghcr.io/opea-project/charts/data-prep"
+  - name: ui
+    alias: chatqna-ui
+    version: 0-latest
+    repository: "oci://ghcr.io/opea-project/charts/ui"
+version: 0-latest
+appVersion: "v1.0"

ChatQnA/kubernetes/helm-chart/README.md

@@ -0,0 +1,83 @@
+# ChatQnA
+
+Helm chart for deploying ChatQnA service. ChatQnA depends on the following services:
+
+- [data-prep](../common/data-prep/README.md)
+- [embedding-usvc](../common/embedding-usvc/README.md)
+- [tei](../common/tei/README.md)
+- [retriever-usvc](../common/retriever-usvc/README.md)
+- [redis-vector-db](../common/redis-vector-db/README.md)
+- [reranking-usvc](../common/reranking-usvc/README.md)
+- [teirerank](../common/teirerank/README.md)
+- [llm-uservice](../common/llm-uservice/README.md)
+- [tgi](../common/tgi/README.md)
+
+## Installing the Chart
+
+To install the chart, run the following:
+
+```console
+cd GenAIInfra/helm-charts/
+./update_dependency.sh
+helm dependency update chatqna
+export HFTOKEN="insert-your-huggingface-token-here"
+export MODELDIR="/mnt/opea-models"
+export MODELNAME="Intel/neural-chat-7b-v3-3"
+# If you would like to use the traditional UI, please change the image as well as the containerport within the values
+# append these at the end of the command "--set chatqna-ui.image.repository=opea/chatqna-ui,chatqna-ui.image.tag=latest,chatqna-ui.containerPort=5173"
+helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set global.modelUseHostPath=${MODELDIR} --set tgi.LLM_MODEL_ID=${MODELNAME}
+# To use Gaudi device
+#helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set global.modelUseHostPath=${MODELDIR} --set tgi.LLM_MODEL_ID=${MODELNAME} -f chatqna/gaudi-values.yaml
+# To use Nvidia GPU
+#helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set global.modelUseHostPath=${MODELDIR} --set tgi.LLM_MODEL_ID=${MODELNAME} -f chatqna/nv-values.yaml
+# To include guardrail component in chatqna on Xeon
+#helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set global.modelUseHostPath=${MODELDIR} -f chatqna/guardrails-values.yaml
+# To include guardrail component in chatqna on Gaudi
+#helm install chatqna chatqna --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set global.modelUseHostPath=${MODELDIR} -f chatqna/guardrails-gaudi-values.yaml
+```
+
+### IMPORTANT NOTE
+
+1. Make sure your `MODELDIR` exists on the node where your workload is scheduled so you can cache the downloaded model for next time use. Otherwise, set `global.modelUseHostPath` to 'null' if you don't want to cache the model.
+
+## Verify
+
+To verify the installation, run the command `kubectl get pod` to make sure all pods are running.
+
+Curl command and UI are the two options that can be leveraged to verify the result.
+
+### Verify the workload through curl command
+
+Run the command `kubectl port-forward svc/chatqna 8888:8888` to expose the service for access.
+
+Open another terminal and run the following command to verify the service if working:
+
+```console
+curl http://localhost:8888/v1/chatqna \
+    -H "Content-Type: application/json" \
+    -d '{"messages": "What is the revenue of Nike in 2023?"}'
+```
+
+### Verify the workload through UI
+
+The UI has already been installed via the Helm chart. To access it, use the external IP of one your Kubernetes node along with the NGINX port. You can find the NGINX port using the following command:
+
+```bash
+export port=$(kubectl get service chatqna-nginx --output='jsonpath={.spec.ports[0].nodePort}')
+echo $port
+```
+
+Open a browser to access `http://<k8s-node-ip-address>:${port}` to play with the ChatQnA workload.
+
+## Values
+
+| Key               | Type   | Default                       | Description                                                                            |
+| ----------------- | ------ | ----------------------------- | -------------------------------------------------------------------------------------- |
+| image.repository  | string | `"opea/chatqna"`              |                                                                                        |
+| service.port      | string | `"8888"`                      |                                                                                        |
+| tgi.LLM_MODEL_ID  | string | `"Intel/neural-chat-7b-v3-3"` | Models id from https://huggingface.co/, or predownloaded model directory               |
+| global.monitoring | bool   | `false`                       | Enable usage metrics for the service components. See ../monitoring.md before enabling! |
+
+## Troubleshooting
+
+If you encounter any issues, please refer to [ChatQnA Troubleshooting](troubleshooting.md)

ChatQnA/kubernetes/helm-chart/ci-gaudi-values.yaml

@@ -0,0 +1 @@
+gaudi-values.yaml

ChatQnA/kubernetes/helm-chart/ci-guardrails-gaudi-values.yaml

@@ -0,0 +1 @@
+guardrails-gaudi-values.yaml

ChatQnA/kubernetes/helm-chart/ci-values.yaml

@@ -0,0 +1 @@
+values.yaml

ChatQnA/kubernetes/helm-chart/cpu-values.yaml

@@ -0,0 +1,109 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+# Override CPU resource request and probe timing values in specific subcharts
+#
+# RESOURCES
+#
+# Resource request matching actual resource usage (with enough slack)
+# is important when service is scaled up, so that right amount of pods
+# get scheduled to right nodes.
+#
+# Because resource usage depends on the used devices, model, data type
+# and SW versions, and this top-level chart has overrides for them,
+# resource requests need to be specified here too.
+#
+# To test service without resource request, use "resources: {}".
+#
+# PROBES
+#
+# Inferencing pods startup / warmup takes *much* longer on CPUs than
+# with acceleration devices, and their responses are also slower,
+# especially when node is running several instances of these services.
+#
+# Kubernetes restarting pod before its startup finishes, or not
+# sending it queries because it's not in ready state due to slow
+# readiness responses, does really NOT help in getting faster responses.
+#
+# => probe timings need to be increased when running on CPU.
+
+tgi:
+  # TODO: add Helm value also for TGI data type option:
+  # https://github.com/opea-project/GenAIExamples/issues/330
+  LLM_MODEL_ID: Intel/neural-chat-7b-v3-3
+
+  # Potentially suitable values for scaling CPU TGI 2.2 with Intel/neural-chat-7b-v3-3 @ 32-bit:
+  resources:
+    limits:
+      cpu: 8
+      memory: 70Gi
+    requests:
+      cpu: 6
+      memory: 65Gi
+
+  livenessProbe:
+    initialDelaySeconds: 8
+    periodSeconds: 8
+    failureThreshold: 24
+    timeoutSeconds: 4
+  readinessProbe:
+    initialDelaySeconds: 16
+    periodSeconds: 8
+    timeoutSeconds: 4
+  startupProbe:
+    initialDelaySeconds: 10
+    periodSeconds: 5
+    failureThreshold: 180
+    timeoutSeconds: 2
+
+teirerank:
+  RERANK_MODEL_ID: "BAAI/bge-reranker-base"
+
+  # Potentially suitable values for scaling CPU TEI v1.5 with BAAI/bge-reranker-base model:
+  resources:
+    limits:
+      cpu: 4
+      memory: 30Gi
+    requests:
+      cpu: 2
+      memory: 25Gi
+
+  livenessProbe:
+    initialDelaySeconds: 8
+    periodSeconds: 8
+    failureThreshold: 24
+    timeoutSeconds: 4
+  readinessProbe:
+    initialDelaySeconds: 8
+    periodSeconds: 8
+    timeoutSeconds: 4
+  startupProbe:
+    initialDelaySeconds: 5
+    periodSeconds: 5
+    failureThreshold: 120
+
+tei:
+  EMBEDDING_MODEL_ID: "BAAI/bge-base-en-v1.5"
+
+  # Potentially suitable values for scaling CPU TEI 1.5 with BAAI/bge-base-en-v1.5 model:
+  resources:
+    limits:
+      cpu: 4
+      memory: 4Gi
+    requests:
+      cpu: 2
+      memory: 3Gi
+
+  livenessProbe:
+    initialDelaySeconds: 5
+    periodSeconds: 5
+    failureThreshold: 24
+    timeoutSeconds: 2
+  readinessProbe:
+    initialDelaySeconds: 5
+    periodSeconds: 5
+    timeoutSeconds: 2
+  startupProbe:
+    initialDelaySeconds: 5
+    periodSeconds: 5
+    failureThreshold: 120

ChatQnA/kubernetes/helm-chart/gaudi-values.yaml

@@ -0,0 +1,76 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+# Accelerate inferencing in heaviest components to improve performance
+# by overriding their subchart values
+
+# TGI: largest bottleneck for ChatQnA
+tgi:
+  accelDevice: "gaudi"
+  image:
+    repository: ghcr.io/huggingface/tgi-gaudi
+    tag: "2.0.6"
+  resources:
+    limits:
+      habana.ai/gaudi: 1
+  # higher limits are needed with extra input tokens added by rerank
+  MAX_INPUT_LENGTH: "2048"
+  MAX_TOTAL_TOKENS: "4096"
+  CUDA_GRAPHS: ""
+  OMPI_MCA_btl_vader_single_copy_mechanism: "none"
+  ENABLE_HPU_GRAPH: "true"
+  LIMIT_HPU_GRAPH: "true"
+  USE_FLASH_ATTENTION: "true"
+  FLASH_ATTENTION_RECOMPUTE: "true"
+
+  livenessProbe:
+    initialDelaySeconds: 5
+    periodSeconds: 5
+    timeoutSeconds: 1
+  readinessProbe:
+    initialDelaySeconds: 5
+    periodSeconds: 5
+    timeoutSeconds: 1
+  startupProbe:
+    initialDelaySeconds: 5
+    periodSeconds: 5
+    timeoutSeconds: 1
+    failureThreshold: 120
+
+# Reranking: second largest bottleneck when reranking is in use
+# (i.e. query context docs have been uploaded with data-prep)
+teirerank:
+  accelDevice: "gaudi"
+  OMPI_MCA_btl_vader_single_copy_mechanism: "none"
+  MAX_WARMUP_SEQUENCE_LENGTH: "512"
+  image:
+    repository: ghcr.io/huggingface/tei-gaudi
+    tag: 1.5.0
+  resources:
+    limits:
+      habana.ai/gaudi: 1
+  securityContext:
+    readOnlyRootFilesystem: false
+  livenessProbe:
+    timeoutSeconds: 1
+  readinessProbe:
+    timeoutSeconds: 1
+
+# Embedding: Second largest bottleneck without rerank
+# By default tei on gaudi is disabled.
+# tei:
+#   accelDevice: "gaudi"
+#   OMPI_MCA_btl_vader_single_copy_mechanism: "none"
+#   MAX_WARMUP_SEQUENCE_LENGTH: "512"
+#   image:
+#     repository: ghcr.io/huggingface/tei-gaudi
+#     tag: 1.5.0
+#   resources:
+#     limits:
+#       habana.ai/gaudi: 1
+#   securityContext:
+#     readOnlyRootFilesystem: false
+#   livenessProbe:
+#     timeoutSeconds: 1
+#   readinessProbe:
+#     timeoutSeconds: 1