[RFC]: Rust front-end

### Motivation vLLM has always had a strong python bias to make it accessible to a wide range of contributors, and to exploit ML libraries including pytorch, triton, etc. Historically, we have managed the inherent performance downsides of python, which include garbage collection and restricted parallelism due to the GIL. The measures taken include resorting to multiprocessing for CPU parallelism, which includes the capability to scale out the API server process. As GPU latency continues to fall, request concurrency grows, and large-scale deployments become the norm, the CPU parts of the system have again become a bottleneck in many situations. This is often seen in the front-end process where the asyncio event loop can’t keep up. Further, our front-end python code has become increasingly complex and fragile as it has evolved organically. With the advent of coding agents, the python vs rust contributor accessibility issue is minimized. It seems the time is right to explore a rust-based alternative to the python front-end process: - It will allow us to eliminate the local api server process scale-out pattern which imposes some awkward limitations and is the source of significant complexity - It is an excuse to restructure the front-end code cleanly “from scratch” - It can be implemented as a “drop in” alternative to the existing front-end api server process transparently to users - It’s much better suited for maintainable production software: - It’s much harder to wr

Posted by njhill on Apr 24, 2026, 11:46 PM

Tags

• github (SaaS)

Related issues

• 2026 - Microsoft Office 365
• [Ubuntu] `ubuntu-latest` label will use Ubuntu 26.04 in November 2026
• Gamut mapping
• [Codex Desktop] Editing and resending a queued message triggers "App-server queued follow-up no longer exists"
• [Windows] ChatGPT 26.820.60940 fails with "invalid transport in mcp_servers.codex_app"; Beta 26.727.40816 works
• [BUG]: infinite loading of videos