Inside vLLM: The Engine Powering Open-Source AI

vLLM is an inference engine running on roughly half a million GPUs at any given moment, according to Simon Moe. Its origins trace to around 2022, before ChatGPT, when a team rooted in UC Berkeley's open source systems research culture set out to accelerate a slow open source demo. vLLM, OpenRouter, and Ollama all emerged in the same 2022 to 2023 window built around open models. Simon Moe argues that serving large language models is fundamentally different from prior machine learning workloads because it requires accelerators, handles non-deterministic output distribution, and demands batching and scheduling as core functions, complexities that earlier models like ResNet or BERT never imposed.

vLLM supports more than a thousand model architectures and runs a process called day zero model release, meaning new open weight architectures go live immediately upon transitioning from research prototype. When Mistral dropped their first model via a torrent link, the vLLM team worked over a weekend and announced support by Monday or Tuesday. A typical model release now involves the model lab, hardware vendors, vLLM, Hugging Face, and up to ten to twenty release partners including inference clouds and hyperscalers. NVIDIA, AMD, Google, Amazon, and Intel all work closely with vLLM to verify performance on new silicon, using it as a benchmark for new chips.

Matt Borenstein noted that roughly a year ago open source models crossed a threshold where serious application companies began depending on them deeply rather than treating them as a curiosity. Companies like Cursor, Decagon, and Harvey concluded they could not build solely on closed source and needed open weight models for mid-training, post-training, and inference customization that proprietary API vendors do not permit. Simon Moe says infrastructure control has been important to customers for several years, while cost sensitivity became a major driver more recently, accelerating as token spend on coding tools skyrocketed. A voice agent company needing to guarantee response times under SLA requirements can only achieve that with self-hosted infrastructure, since proprietary APIs introduce downtime risk and potential contract violations.

For open weight models, operators can offer up to ten different levels of inference speed, compared to only a regular mode and a fast mode for proprietary models. The fastest open weight mode can reach 400 to 500 tokens per second and is typically two to three times faster than the fast mode available from proprietary providers. Simon Moe described Kimi K3 as bridging approximately a ten times cost gap, landing between closed source frontier pricing and GLM 5.2 pricing, and bringing an Opus 4.8 level model to self-hosted infrastructure that users can run, fine-tune, and fully profile. He cautioned that open weight models are not always cheaper than proprietary models and that cost varies by model.

Licensing terms attached to open weight releases are becoming more common as model labs seek to fund ongoing development. Meta's Llama license included a commercial threshold based on daily active users or annual recurring revenue that effectively applied to only about two companies in the world at the time. Minimax's M2.7 and Kimi's releases included usage-focused or derivative-work restrictions that became notable when Fireworks and Cursor built on top of the Kimi model. Matt Borenstein argued that open weight models cannot be maintained through volunteer contributions because training frontier models requires millions or billions of dollars in compute, making attached economics more necessary than in traditional open source software. He noted that Chinese open weight models face a particular funding risk because without commercial revenue the alternative funding source is government, which he described as worse for the ecosystem.

Developers at InfraX and vLLM are retreating from Anthropic models and using Kimi K3 instead because Anthropic's guardrails produce high false positive rates that block legitimate use cases including GPU kernel research, where a two-hour job can be lost entirely when a false positive trigger occurs mid-session. Simon Moe and others described content moderation as an evergreen unsolved problem and predicted that for trusted use cases users will default to open weight models where guardrails can be controlled or are less restrictive. Consolidating AI work on one or two platforms creates a moderation and liability problem larger than what communication platforms faced, because it involves actual work and action rather than speech.

Simon argues the capability gap between open and closed weight models is not large even today and predicts that within one year there will be no meaningful capability difference, with the next phase of progress centered on recursive self-improvement and the environments model labs build to let models improve on themselves. Moonshot is cited as an example, having built one of the best environments for front-end coding through iterative model improvement via rendering and looping. Simon contends that distillation is not a cornerstone of progress seen in Chinese open weight labs, arguing that environments and learning processes cannot be distilled from other models, and that distillation policy restrictions would not solve AI competitiveness problems because the underlying creative work is distributed globally.