Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM #12069

Open
wants to merge 75 commits into
base: main
Choose a base branch
from
Open

[Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM #12069

wants to merge 75 commits into from
+1,472 −175

Conversation

HwwwwwwwH
Copy link
Contributor

@HwwwwwwwH HwwwwwwwH commented Jan 15, 2025 •
edited
Loading

This PR aims to adapt and support all the features of MiniCPM-V and MiniCPM-o. It is designed to be compatible with various modalities (image, video, audio), different model versions (2.0, 2.5, 2.6, o), and diverse input types (raw, embeddings), while maintaining support for LORA, which might require significant effort.

Below is the roadmap for this PR:

  • Refactor the input processor code of MiniCPM-V for MultiModalInputsV2 of vLLM.
    • Support for image and video inputs.
    • Support for image embeddings inputs.
    • Support for video embeddings inputs.
    • Previous supports for MiniCPM-o.
    • Verify LORA support.
  • Adapt new features of MiniCPM-o.
    • Support for audio and audio embeddings inputs.
    • Support for image and audio interleave inputs.
    • Support for audio outputs (using hidden states).
    • Streaming multimodal inputs (may be complex; consider starting a new PR for this feature in the future).

This PR is still in development. Once I complete the support for audio, I will request to merge. I'll get this work done ASAP.

FIX #12162

@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

This was referenced Jan 15, 2025
Open
Open
@DarkLight1337 DarkLight1337 self-assigned this Jan 15, 2025
@ywang96
Copy link
Member

ywang96 commented Jan 15, 2025 •
edited
Loading

Really appreciate your effort planned on this PR!

Support for audio outputs (using hidden states).
Streaming multimodal inputs (may be complex; consider starting a new PR for this feature in the future).

It would be great if you can share some design decisions for the these two items as an RFC (or two separate RFCs) first before we proceed with implementation. We (vLLM team) are also thinking about how we want to support multimodal output and streaming/realtime API on vLLM so it's probably the best time for us to discuss these items!

@HwwwwwwwH
Copy link
Contributor Author

Really appreciate your effort planned on this PR!

Support for audio outputs (using hidden states).
Streaming multimodal inputs (may be complex; consider starting a new PR for this feature in the future).

It would be great if you can share some design decisions for the these two items as an RFC (or two separate RFCs) first before we proceed with implementation. We (vLLM team) are also thinking about how we want to support multimodal output and streaming/realtime API on vLLM so it's probably the best time for us to discuss these items!

Thank you for suggestion! I'll start these two RFCs tomorrow.

@HwwwwwwwH
Copy link
Contributor Author

@DarkLight1337 I think I might need some help for verifying LoRA support. Should I do any changes for it?

@DarkLight1337
Copy link
Member

@jeejeelee can help with this. Please keep in mind though that currently LoRA is only supported for the language part of multi-modal models.

@DarkLight1337 DarkLight1337 mentioned this pull request Jan 17, 2025
Open
1 task
@HwwwwwwwH HwwwwwwwH mentioned this pull request Jan 18, 2025
Open
2 tasks
@linyinli linyinli mentioned this pull request Jan 19, 2025
Closed
HwwwwwwwH and others added 19 commits January 22, 2025 14:28
@HwwwwwwwH
refactor for images
f78ad12 
Signed-off-by: hzh <[email protected]>
@HwwwwwwwH
supprot image embedding for minicpmv
95230b9 
Signed-off-by: hzh <[email protected]>
@llsj14 @HwwwwwwwH
[Bugfix][SpecDecode] Adjust Eagle model architecture to align with in…
42ffb1b 
…tended design (vllm-project#11672)

Signed-off-by: Sungjae Lee <[email protected]>
Signed-off-by: hzh <[email protected]>
@shaochangxu @HwwwwwwwH
[Bugfix] fused_experts_impl wrong compute type for float32 (vllm-proj…
43ff2e9 
…ect#11921)

Signed-off-by: shaochangxu.scx <[email protected]>
Co-authored-by: shaochangxu.scx <[email protected]>
Signed-off-by: hzh <[email protected]>
@DarkLight1337 @HwwwwwwwH
[CI/Build] Move model-specific multi-modal processing tests (vllm-pro…
0ec9974 
…ject#11934)

Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: hzh <[email protected]>
@DarkLight1337 @HwwwwwwwH
[Doc] Basic guide for writing unit tests for new models (vllm-project…
b4a9094 
…#11951)

Signed-off-by: DarkLight1337 <[email protected]>
Signed-off-by: hzh <[email protected]>
@NickLucche @HwwwwwwwH
[Bugfix] Fix RobertaModel loading (vllm-project#11940)
ac29198 
Signed-off-by: NickLucche <[email protected]>
Signed-off-by: hzh <[email protected]>
@sixsixcoder @Isotr0py @HwwwwwwwH
[Model] Add cogagent model support vLLM (vllm-project#11742)
286107f 
Signed-off-by: Isotr0py <[email protected]>
Co-authored-by: Isotr0py <[email protected]>
Signed-off-by: hzh <[email protected]>
@ywang96 @HwwwwwwwH
[V1] Avoid sending text prompt to core engine (vllm-project#11963)
535e120 
Signed-off-by: Roger Wang <[email protected]>
Signed-off-by: hzh <[email protected]>
@rafvasq @HwwwwwwwH
[CI/Build] Add markdown linter (vllm-project#11857)
925562b 
Signed-off-by: Rafael Vasquez <[email protected]>
Signed-off-by: hzh <[email protected]>
@Isotr0py @DarkLight1337 @HwwwwwwwH
[Model] Initialize support for Deepseek-VL2 models (vllm-project#11578)
936b306 
Signed-off-by: Isotr0py <[email protected]>
Co-authored-by: Cyrus Leung <[email protected]>
Signed-off-by: hzh <[email protected]>
@Akshat-Tripathi @mosalov
@jeejeelee @Isotr0py @HwwwwwwwH
[Hardware][CPU] Multi-LoRA implementation for the CPU backend (vllm-p…
141151f 
…roject#11100)

Signed-off-by: Akshat Tripathi <[email protected]>
Signed-off-by: Oleg Mosalov <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Co-authored-by: Oleg Mosalov <[email protected]>
Co-authored-by: Jee Jee Li <[email protected]>
Co-authored-by: Isotr0py <[email protected]>
Signed-off-by: hzh <[email protected]>
@avshalomman @HwwwwwwwH
[Hardware][TPU] workaround fix for MoE on TPU (vllm-project#11764)
eac7811 
Signed-off-by: hzh <[email protected]>
@robertgshaw2-redhat @HwwwwwwwH
[V1][Core][1/n] Logging and Metrics (vllm-project#11962)
e251866 
Signed-off-by: [email protected] <[email protected]>
Signed-off-by: hzh <[email protected]>
@Isotr0py @DarkLight1337 @HwwwwwwwH
[Model] Support GGUF models newly added in transformers 4.46.0 (vll…
e46c06b 
…m-project#9685)

Signed-off-by: Isotr0py <[email protected]>
Co-authored-by: Cyrus Leung <[email protected]>
Signed-off-by: hzh <[email protected]>
@robertgshaw2-redhat @HwwwwwwwH
[V1] [2/n] Logging and Metrics - OutputProcessor Abstraction (vllm-…
d12c0de 
…project#11973)

Signed-off-by: [email protected] <[email protected]>
Signed-off-by: hzh <[email protected]>
@yyccli @HwwwwwwwH
[MISC] fix typo in kv transfer send recv test (vllm-project#11983)
e459c90 
Signed-off-by: hzh <[email protected]>
@liaoyanqing666 @HwwwwwwwH
[Bug] Fix usage of .transpose() and .view() consecutively. (vllm-…
93a78ba 
…project#11979)

Signed-off-by: hzh <[email protected]>
@llsj14 @HwwwwwwwH
[CI][Spec Decode] fix: broken test for EAGLE model (vllm-project#11972)
dd2f627 
Signed-off-by: Sungjae Lee <[email protected]>
Signed-off-by: hzh <[email protected]>
DarkLight1337
DarkLight1337 reviewed Jan 23, 2025
View reviewed changes
vllm/model_executor/models/minicpmv.py Outdated
Comment on lines 934 to 940
if "image" in result["mm_placeholders"] and \
self.info.get_model_version() in [(2, 6), (2, "6O")]:
result["mm_placeholders"]["image"] = [
PlaceholderRange(offset=p["offset"] + 3 + idx // 10,
length=p["length"] - 3 - idx // 10)
for idx, p in enumerate(result["mm_placeholders"]["image"])
]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use PromptReplacementDetails (introduced by #12269) to simplify this code

@DarkLight1337
Copy link
Member

Please take a look at the CI failures from before: https://buildkite.com/vllm/fastcheck/builds/12337#0194961e-b0c4-46a6-8033-93d6ca567c9f

Can you fix the tests locally? Afterwards I'll unblock the tests on CI.

@PancakeAwesome
Copy link

When will this feature be available if only non-streaming image input is used？

@HwwwwwwwH
Copy link
Contributor Author

When will this feature be available if only non-streaming image input is used？

For none-streaming image input support of MiniCPM-o-2_6, you can use our own fork and build vllm from source.
And I think this PR will get merged in these two days. Sry for waiting.

@zhy844694805
Copy link

I am very eager to test the Minicpm-o2.6 model using the vllm engine, but you haven't merged the PR yet. Now, I would like to know, for a 4090/5090 graphics card using vllm to infer the Minicpm-o2.6 model with audio input and output, what do you estimate the concurrency and latency to be approximately?

@zhy844694805
Copy link

如果仅使用非流式图像输入，什么时候可以使用此功能？

对于 的非流式图像输入支持MiniCPM-o-2_6，您可以使用我们自己的 fork 并vllm从源代码构建。 我认为这个 PR 将在这两天内合并。抱歉让您久等了。

TTS功能会在这两天合并吗？（因为您的路线图上没有勾选TTS功能）如果是，那就太棒了

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DarkLight1337 DarkLight1337 DarkLight1337 left review comments

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@DarkLight1337 DarkLight1337

Labels
ci/build documentation Improvements or additions to documentation frontend
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[New Model]: openbmb/MiniCPM-o-2_6