Inferact: Building the Infrastructure That Runs Modern AI

The conversation centers on the development of VLM, an open-source inference engine aimed at becoming the world's standard for AI infrastructure. Co-founders Simon Moe and Woosa Kwan emphasize the importance of open-source capabilities in creating a universal inference layer that enhances AI advancements across various hardware and applications. They note that the complexity of running AI systems now rivals that of building them, with unique requests leading to unpredictability in outputs and increased demands on hardware.

Woosa shares the origins of VLLM, which began as a prototype during his PhD at UC Berkeley and evolved into an open-source project after optimizing a demo service for Meta's OPT model. The project was driven by curiosity and the desire for hands-on experience with large models. The discussion touches on the technical differences between autoregressive transformers and traditional machine learning, highlighting the shift from CPU reliance in traditional workloads to GPU utilization in deep learning, particularly for large language models.

Simon discusses the growth of the VLM community, which has expanded significantly, reflecting a shared interest in advancing AI technology. They address challenges in scheduling and memory management in LLMs, particularly the need for systems that can handle varying input and output lengths effectively. The conversation concludes with a focus on collaborative efforts to create a unified system that benefits both model and hardware providers, streamlining compatibility and performance.

The role of VL in building infrastructure for modern AI applications is emphasized, highlighting the benefits of collaboration and innovation. A community-driven approach is central to managing contributors, drawing lessons from successful open-source projects. Clear requirements, roadmaps, and milestones are essential for maintaining high code quality, alongside regular code reviews and refactoring.

VL operates similarly to traditional engineering organizations, setting quarterly visions and inviting community contributions. In-person meetups foster effective communication and collaboration, while community support and corporate sponsorship play vital roles in funding and sustaining the project. The financial aspects are significant, with operational costs underscoring the importance of thorough testing for reliability across millions of GPUs.

The discussion transitions to technical aspects, defining an inference engine as a system that utilizes pre-trained models to generate outputs efficiently. Key components like tokenizers, schedulers, and memory managers are crucial for handling the increasing complexity of inference due to larger models and diverse architectures. Challenges arise from the need to shard large models across multiple GPUs, balancing communication costs and load distribution.

The speakers note the growing diversity in models and architectures, with NVIDIA leading the way in releasing open-source models across various domains. This trend is accompanied by a divergence in standardization, as different labs develop unique methods for input/output processing. The implementation of new features, such as sparse attention, is facilitated through collaboration with model vendors and the open-source community.

As the paradigm shifts towards multi-agent interactions and complex inference scenarios, new infrastructure is required to manage state effectively. The complexity of multi-turn conversations and interactions with external environments introduces challenges in maintaining cache access and eviction patterns, necessitating innovative solutions in inference architecture. The speakers express a strong belief in open-source AI, emphasizing its importance for fostering diversity in models and chip architecture, arguing that it promotes innovation and common standards.

The discussion includes notable deployments of VLM, such as Amazon's use of it for their Rufus assistant bots, showcasing VLM's capabilities in real-time user interactions. Other companies, like LinkedIn, are also adopting VLM features, demonstrating its growing significance in the industry. InfraRact, initiated by the maintainers of the VLM project, aims to establish VLM as the world's inference engine and create a universal inference layer that supports various hardware and future AI workloads.

The speakers stress that open source is critical to AI infrastructure and the mission of InfraRact, viewing it as a "secret weapon" for community collaboration. Jan Stoica, a co-founder of Databricks, plays a key role in advising InfraRact and guiding talent acquisition. The primary challenge faced by the company is inference at scale, prompting a search for experienced ML infrastructure engineers to optimize advanced hardware for large open-source models.

The focus of InfraRact is on creating a horizontal abstraction layer for machine learning systems, akin to operating systems and databases. This involves developing a fundamental runtime layer for accelerated computing devices and refining the best abstractions for building systems, highlighting the transition from serving systems to the necessary runtime for intelligence.