[RFC]: Multi-modality Support on vLLM

## Scheduled meetings We host a community sync every two weeks on 21:00-22:00 Pacific Time. Meeting link and notes can be found [here](https://docs.google.com/document/d/1rQN5GeZZfBINksWCeDwh4fgJJNUc2p0dTKsHaK-mC8U/edit?usp=sharing). ## Active projects We have migrated to GitHub Projects to track the latest issues: - [Core tasks](https://github.com/orgs/vllm-project/projects/8) - [Model requests](https://github.com/orgs/vllm-project/projects/10) ## Legacy items **Update [11/18] - In the upcoming months, we will focus on performance optimization for multimodal models as part of vLLM V1 engine re-arch effort** **P0** (We will definitely work on them): - [x] V1 re-arch for multimodal models - See high-level design ([Slides](https://docs.google.com/presentation/d/1e3CxQBV3JsfGp30SwyvS3eM_tW-ghOhJ9PAJGK6KR54/edit#slide=id.g31455c8bc1e_2_122), [Doc](https://docs.google.com/document/d/11_DFQTku6C2aV6ghK21P76ST6uAUVjMlEjs54prtb_g/edit?usp=sharing)) - [x] Core - [x] [1/N] #9871 - [x] [2/N] #10374 - [x] [3/N] #10570 - [x] [4/N] #10699 - [x] [5/N] #11210 - [x] [6/N] #12128 - [x] [7/N] Enable rest of single-modality LMMs on V1 - [x] #11632 (Aria, BLIP-2, Chameleon, Fuyu) - [x] #14275 - [x] #11685 - [x] #11733 - [x] #12069 - [x] #12504 - [x] #12660 - [x] #15487 - [x] [8/N] Enable mixed-modality inference on V1 - [x] #11685 - [x] #12259 - [x] [9/N] Enable interleaved-modality infer

Posted by ywang96 on Apr 19, 2024, 7:51 AM

Tags

• support (Software)

Related issues

• Add 65536 size support for maps
• TypeScript 7.1 Iteration Plan
• nvim-treeconfig: upstream nvim-treesitter
• Add the "action" and "method" attributes to <button>
• [FEATURE] Support multiple Connector accounts (same connector, different accounts) in Claude and Claude Code on the web (claude.ai/code)
• > We were imposed GitHub for so many exercises!