Add video-llama LVM microservice under lvms #495

BaoHuiling · 2024-08-15T11:02:23Z

Description

Add support for video-llama LVM, which allows us to chat based on video file with LVM. The first usecase is VideoRAGQnA use case from Multimodal RAG proposal.

Issues

List the issue or RFC link this PR is working on. If there is no such link, please mark it as n/a.
The RFC is still under review. opea-project/docs#49

Type of change

List the type of change like below. Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds new functionality)
Breaking change (fix or feature that would break existing design and interface)
Others (enhancement, documentation, validation, etc.)

Dependencies

python:3.9-slim git git-lfs faiss-cpu sentencepiece pandas Pillow prometheus_fastapi_instrumentator ftfy decord einops iopath omegaconf opencv-python-headless timm torch torchaudio langchain-core transformers webdataset pytorchvideo DAMO-NLP-SG/Video-LLaMA(github repo) DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned (huggingface repo)

Tests

Describe the tests that you ran to verify your changes.
https://github.com/siddhivelankar23/GenAIComps/blob/huiling-lvm/tests/test_lvms_video-llama.sh: build, start, validate, clean the microservice

Signed-off-by: BaoHuiling <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: BaoHuiling <[email protected]>

codecov · 2024-08-16T02:45:27Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Files	Coverage Δ
comps/cores/proto/docarray.py	`99.00% <100.00%> (+0.06%)`	⬆️

... and 1 file with indirect coverage changes

Signed-off-by: BaoHuiling <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: BaoHuiling <[email protected]>

BaoHuiling · 2024-08-16T10:14:19Z

@chensuyue Hello Suyue, one question here, does the server that runs the CI test have access to the huggingface repo? I need to download files under https://huggingface.co/DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned/tree/main, but it seems that this step is bypassed or failed.

chensuyue · 2024-08-16T12:44:16Z

@chensuyue Hello Suyue, one question here, does the server that runs the CI test have access to the huggingface repo? I need to download files under https://huggingface.co/DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned/tree/main, but it seems that this step is bypassed or failed.

Yes, generally it should have. Use $HF_TOKEN in your test scripts.

Signed-off-by: BaoHuiling <[email protected]>

BaoHuiling · 2024-08-17T04:06:19Z

@chensuyue Thank you for your comment, the image is download successfully.

However another problem occured, the port that i'm using 9000 is already allocated. The old containers are already cleaned before running the test, could you help to check environment and remove the conflict?

chensuyue · 2024-08-17T16:02:34Z

@chensuyue Thank you for your comment, the image is download successfully.

However another problem occured, the port that i'm using 9000 is already allocated. The old containers are already cleaned before running the test, could you help to check environment and remove the conflict?

port 9000 is widely used, also in GenAIExamples CI test, and they share the test machine. So please change the port into 5xxx which is suggested to be used in GenAIComps CI.

BaoHuiling · 2024-08-19T02:04:21Z

@Spycsh @lvliang-intel Please help to review the code, thanks!

Signed-off-by: BaoHuiling <[email protected]>

BaoHuiling · 2024-08-20T01:49:03Z

Hello, @chensuyue @Spycsh @lvliang-intel could anyone help to review the code? The DDL is tomorrow and we may need some time to fix required changes.🙂

Spycsh · 2024-08-20T03:15:24Z

Hi @BaoHuiling , thanks for this exciting feature! It looks good to me. I just have a few questions: This only support Xeon CPU/CUDA, right? Have you tried video-llama LVM on Gaudi? Do you have a plan on optimizing it on Gaudi? Don't worry, we will still merge it for now if the HPU support is not clear for this component.

BaoHuiling · 2024-08-20T03:17:56Z

Hi @BaoHuiling , thanks for this exciting feature! It looks good to me. I just have a few questions: This only support Xeon CPU/CUDA, right? Have you tried video-llama LVM on Gaudi? Do you have a plan on optimizing it on Gaudi? Don't worry, we will still merge it for now if the HPU support is not clear for this component.

Thanks for the comment. Yes only Xeon is supported now. We haven't planned optimization on Gaudi yet, but I think it's good to have in the future.

Signed-off-by: BaoHuiling <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit db8c893)

Signed-off-by: BaoHuiling <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Signed-off-by: BaoHuiling <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <[email protected]>

Signed-off-by: BaoHuiling <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…e for v0.9 (#538) * clip embedding support Signed-off-by: srinarayan-srikanthan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: srinarayan-srikanthan <[email protected]> * test script for embedding Signed-off-by: srinarayan-srikanthan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: srinarayan-srikanthan <[email protected]> * fix freeze workflow (#522) Signed-off-by: Sun, Xuehao <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * Fix Dataprep Potential Error in get_file (#540) * fix get file error & refine logs Signed-off-by: letonghan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: letonghan <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <[email protected]> * Support SearchedDoc input type in LLM for No Rerank Pipeline (#541) Signed-off-by: letonghan <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * Add dependency for pdf2image and OCR processing (#421) Signed-off-by: srinarayan-srikanthan <[email protected]> * Add local_embedding return 768 length to align with chatqna example (#313) Signed-off-by: Chendi.Xue <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * add telemetry doc (#536) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <[email protected]> * Add video-llama LVM microservice under lvms (#495) Signed-off-by: BaoHuiling <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: srinarayan-srikanthan <[email protected]> * Fix the data load issue for structured files (#505) Signed-off-by: XuhuiRen <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * Add finetuning component (#502) Signed-off-by: Xinyu Ye <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: lkk <[email protected]> Co-authored-by: test <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: Letong Han <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * add torchvision into requirements (#546) Signed-off-by: chensuyue <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * Use Gaudi base images from Dockerhub (#526) * Use Gaudi base images from Dockerhub Signed-off-by: Abolfazl Shahbazi <[email protected]> * Fixing the malformed tag Signed-off-by: Abolfazl Shahbazi <[email protected]> * fix another malformed tag Signed-off-by: Abolfazl Shahbazi <[email protected]> --------- Signed-off-by: Abolfazl Shahbazi <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * Add toxicity detection microservice (#338) * Add toxicity detection microservice Signed-off-by: Qun Gao <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Modification to toxicity plugin PR (#432) * changed microservice to use Service.GUARDRAILS and input/output to TextDoc Signed-off-by: Tyler Wilbers <[email protected]> * simplify dockerfile to use langchain Signed-off-by: Tyler Wilbers <[email protected]> * sort requirements Signed-off-by: Tyler Wilbers <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Tyler Wilbers <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Minor SPDX header update (#434) Signed-off-by: Abolfazl Shahbazi <[email protected]> * Remove 'langsmith' per code review (#534) Signed-off-by: Abolfazl Shahbazi <[email protected]> * Add toxicity detection microservices with E2E testing Signed-off-by: Qun Gao <[email protected]> --------- Signed-off-by: Qun Gao <[email protected]> Signed-off-by: Tyler Wilbers <[email protected]> Signed-off-by: Abolfazl Shahbazi <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abolfazl Shahbazi <[email protected]> Co-authored-by: Tyler W <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * rename script and use 5xxx Signed-off-by: BaoHuiling <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * add proxy for build Signed-off-by: BaoHuiling <[email protected]> Signed-off-by: srinarayan-srikanthan <[email protected]> * fixed commit issues Signed-off-by: srinarayan-srikanthan <[email protected]> * Fix docarray constraint Signed-off-by: srinarayan-srikanthan <[email protected]> * updated docarray Signed-off-by: srinarayan-srikanthan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: srinarayan-srikanthan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * rm telemetry which cause error in mega Signed-off-by: BaoHuiling <[email protected]> * renamed dirs Signed-off-by: srinarayan-srikanthan <[email protected]> * renamed test Signed-off-by: srinarayan-srikanthan <[email protected]> --------- Signed-off-by: srinarayan-srikanthan <[email protected]> Signed-off-by: Sun, Xuehao <[email protected]> Signed-off-by: letonghan <[email protected]> Signed-off-by: Chendi.Xue <[email protected]> Signed-off-by: BaoHuiling <[email protected]> Signed-off-by: XuhuiRen <[email protected]> Signed-off-by: Xinyu Ye <[email protected]> Signed-off-by: chensuyue <[email protected]> Signed-off-by: Abolfazl Shahbazi <[email protected]> Signed-off-by: Qun Gao <[email protected]> Signed-off-by: Tyler Wilbers <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sun, Xuehao <[email protected]> Co-authored-by: Letong Han <[email protected]> Co-authored-by: Zaili Wang <[email protected]> Co-authored-by: Chendi.Xue <[email protected]> Co-authored-by: Sihan Chen <[email protected]> Co-authored-by: Huiling Bao <[email protected]> Co-authored-by: XuhuiRen <[email protected]> Co-authored-by: XinyuYe-Intel <[email protected]> Co-authored-by: lkk <[email protected]> Co-authored-by: test <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: chen, suyue <[email protected]> Co-authored-by: Abolfazl Shahbazi <[email protected]> Co-authored-by: qgao007 <[email protected]> Co-authored-by: Tyler W <[email protected]>

BaoHuiling requested a review from lvliang-intel as a code owner August 15, 2024 11:02

BaoHuiling and others added 16 commits August 15, 2024 19:04

initial commit

8ebf205

Signed-off-by: BaoHuiling <[email protected]>

update video llama, model download

bcffdff

Signed-off-by: BaoHuiling <[email protected]>

add test script

2cd5e55

Signed-off-by: BaoHuiling <[email protected]>

Add LVM container, update video_path to URL

354166c

Signed-off-by: BaoHuiling <[email protected]>

update for prod

c8aed32

Signed-off-by: BaoHuiling <[email protected]>

save

275bb3d

Signed-off-by: BaoHuiling <[email protected]>

test script passed

d49969a

Signed-off-by: BaoHuiling <[email protected]>

add readme

25d643e

Signed-off-by: BaoHuiling <[email protected]>

add clean in readme

140f041

Signed-off-by: BaoHuiling <[email protected]>

change input param name to "input"

d4144e0

Signed-off-by: BaoHuiling <[email protected]>

fix typo

57be2e6

Signed-off-by: BaoHuiling <[email protected]>

Add support for UI

80a8366

Signed-off-by: BaoHuiling <[email protected]>

fix typo in test

149b6c9

Signed-off-by: BaoHuiling <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

584ddec

for more information, see https://pre-commit.ci

fix check

90cfea0

Signed-off-by: BaoHuiling <[email protected]>

fix typo in patch

c2a3e89

Signed-off-by: BaoHuiling <[email protected]>

BaoHuiling force-pushed the lvm branch 2 times, most recently from 7476487 to ff6110c Compare August 16, 2024 02:39

fix bug caused by bot

ff6110c

Signed-off-by: BaoHuiling <[email protected]>

BaoHuiling force-pushed the lvm branch from 1d82d50 to 6f367c8 Compare August 16, 2024 03:06

BaoHuiling and others added 5 commits August 16, 2024 11:17

do not modify patch

88545b7

Signed-off-by: BaoHuiling <[email protected]>

extract when build

6f367c8

Signed-off-by: BaoHuiling <[email protected]>

support local file incase no internet

439174a

Signed-off-by: BaoHuiling <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

40a0605

for more information, see https://pre-commit.ci

update script to use local file

000e2a5

Signed-off-by: BaoHuiling <[email protected]>

kevinintel requested a review from Spycsh August 16, 2024 06:38

use localhost in test

a59766b

Signed-off-by: BaoHuiling <[email protected]>

Merge branch 'main' into lvm

45bf663

BaoHuiling mentioned this pull request Aug 16, 2024

Add local Rerank microservice for VideoRAGQnA #496

Merged

4 tasks

BaoHuiling added 4 commits August 16, 2024 21:44

add debug steps

5869d9f

Signed-off-by: BaoHuiling <[email protected]>

Merge branch 'main' into lvm

db805e8

asdd Pillow for lvm

aabd962

Signed-off-by: BaoHuiling <[email protected]>

Merge branch 'main' into lvm

eb0437e

chensuyue added this to the v0.9 milestone Aug 17, 2024

Merge branch 'main' into lvm

c72482c

BaoHuiling added 3 commits August 19, 2024 10:16

use 5xxx for CI port

3bda667

Signed-off-by: BaoHuiling <[email protected]>

Merge branch 'main' into lvm

2b7cb4d

update volume name

1ec2e48

Signed-off-by: BaoHuiling <[email protected]>

Merge branch 'main' into lvm

db1d78f

Spycsh approved these changes Aug 20, 2024

View reviewed changes

lvliang-intel approved these changes Aug 21, 2024

View reviewed changes

chensuyue merged commit db8c893 into opea-project:main Aug 21, 2024
14 checks passed

BaoHuiling mentioned this pull request Aug 29, 2024

Multimodal dataprep #575

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add video-llama LVM microservice under lvms #495

Add video-llama LVM microservice under lvms #495

BaoHuiling commented Aug 15, 2024

codecov bot commented Aug 16, 2024 •

edited

Loading

BaoHuiling commented Aug 16, 2024

chensuyue commented Aug 16, 2024

BaoHuiling commented Aug 17, 2024

chensuyue commented Aug 17, 2024

BaoHuiling commented Aug 19, 2024

BaoHuiling commented Aug 20, 2024

Spycsh commented Aug 20, 2024

BaoHuiling commented Aug 20, 2024

Add video-llama LVM microservice under lvms #495

Add video-llama LVM microservice under lvms #495

Conversation

BaoHuiling commented Aug 15, 2024

Description

Issues

Type of change

Dependencies

Tests

codecov bot commented Aug 16, 2024 • edited Loading

Codecov Report

BaoHuiling commented Aug 16, 2024

chensuyue commented Aug 16, 2024

BaoHuiling commented Aug 17, 2024

chensuyue commented Aug 17, 2024

BaoHuiling commented Aug 19, 2024

BaoHuiling commented Aug 20, 2024

Spycsh commented Aug 20, 2024

BaoHuiling commented Aug 20, 2024

codecov bot commented Aug 16, 2024 •

edited

Loading