Inside vLLM: The Engine Powering Open-Source AI

vLLM is an inference engine that turns available GPUs into endpoints for AI models. The host says it is running on about half a million GPUs at any moment, while Simon Moe says it supports more than 1,000 model architectures. Serving language models is more demanding than conventional machine-learning workloads because request and response lengths vary, so batching and scheduling strongly affect speed and efficiency. Moe says the task is becoming harder as models diversify and agent workloads run longer.

Moe says vLLM works with NVIDIA, AMD, Google, Amazon, Intel, and other hardware vendors to run new chips, and is sometimes used to benchmark them. Its “Day Zero Modalities” effort aims to make a model usable from the time it moves from research prototype to accessible open-weight release. That can require coordination among model labs, hardware makers, Hugging Face, and inference or cloud providers. Some labs use vLLM in production or research and contribute code; others rely on internal engines or lack systems expertise. Moe says vLLM helped get inference working behind the scenes over a weekend for Mistral’s first release, enabling others to use and build on the model.

Moe says his organization signed the open-weights letter because open weights and open development matter, and the ecosystem should not be confined to proprietary APIs. Matt Bornstein distinguishes open weights from true open source, while noting that closed models remain more widely used and generally more critical. Open models, however, have enabled some innovative applications; he cites Cursor, Decagon, and Harvey as companies building differentiated products on open source. Customers value both cost and control, with cost more salient recently but control still important. Running an open-weight model on a company’s infrastructure can allow workload-specific tuning and greater control over performance, data handling, security, and compliance. A voice-agent company, for example, may need to manage its model and infrastructure to meet response-time SLAs. Moe says Kimi K3 costs less than Claude or GPT, but more than GLM 5.2.

The speakers stress that training open-weight models requires substantial funding, unlike open-source software that can be built largely through volunteer effort. One uses a hypothetical $100 million training run to illustrate the expense, adding that five large-scale runs might fail before one succeeds. Meta’s Llama, MiniMax M2.7, and Kimi are cited as releases with usage or commercial terms intended to help fund development; one speaker expects labs to adopt more such economic mechanisms. Once released, models can be adapted by the community for different hardware and deployment settings, from edge devices to large clusters and voice or coding agents. Broad use of open inference engines can improve performance and reliability, and give inference clouds and API services access to battle-tested recipes. Community testing can also expose rare bugs; the guest gives a remaining-bug rate of roughly 0.0001% as an example.

Open models also give users more control over safeguards, which the speakers say can matter when proprietary-model guardrails block legitimate uses through false positives. They note that some AI companies impose ethical restrictions beyond legal requirements, sometimes frustrating users—for example, when safeguards block translation. One speaker says Anthropic safeguards blocked developers’ research, including a GPU-kernel test that triggered a red line over an invalid-memory-access error; some developers then shifted to Kimi models. The speakers argue that moderating AI is more difficult than moderating social media because AI is increasingly used for work and to take actions, while AI providers do not have the liability carveout they describe for social platforms. Ian Stoica’s open-source-first approach at Infrakt is to build what can be built openly, then close remaining gaps for customers and partners.

The speakers see model progress as depending not only on data, but also on the environments where models learn and improve. Moonshot’s front-end coding environment, for example, lets models inspect rendered output and iterate. One speaker expects open-model work over the next year to focus on better learning environments and optimization methods, and predicts no open-versus-closed model difference in a year, while saying it is difficult to know whether open models will fully close the frontier gap over five years. They also emphasize how incomplete the theory remains: simpler approaches can sometimes work better, and Kimi K3 removed rotary positional embeddings, with the technique’s inventor explaining in the model’s technical report why they were unnecessary. One speaker says distillation is not currently a major driver of progress, which instead comes from researchers, algorithms, data, environments, and compute; learning in an environment cannot readily be distilled by others without access to it.