Why the agentic era is already hitting resource walls

Nathan Wittemore discusses the practical insights into AI developments and the growing viability of agents for enterprises. He notes a shift towards agent usage in businesses, supported by KPMG's pulse report, which highlights recognition of agents by both power and non-power users. Steve Chase from KPMG addresses the challenges companies face in the agent era, particularly resource constraints and token usage, questioning the sustainability of agents' cost-effectiveness compared to human labor. He emphasizes the need to prioritize outcomes based on investments in agents and associated costs, pointing out that even large companies like OpenAI are experiencing significant resource constraints.

The conversation includes OpenAI's decision to shut down Sora, reflecting a shift in resource management despite user engagement. There are limits to available compute resources, and competition exists between training and inference. Increased productivity from AI has led to users working longer hours, resulting in higher resource consumption. While inference costs are decreasing, a surge in usage could worsen resource constraints.

Robert Leonard highlights disparities in AI deployment among companies, with market leaders utilizing more advanced, token-hungry agents. Concerns about peak demand during work hours are raised, as slowdowns in AI systems could increase reliance on human resources. The need for robust systems design around agentic deployments and human capability training is emphasized.

The limitations of current AI models are discussed, with some being effective while others lag. Companies are encouraged to adopt a multi-model approach to avoid lock-in and ensure flexibility. Internal evaluations of models are crucial, as external assessments cannot replace knowledge of one's own systems. Effective resource management is critical, as excessive token use can lead to costly mistakes, necessitating close monitoring of consumption.

Meta's approach to token consumption is mentioned, promoting a leaderboard for maximizing usage. The challenge of implementing sophisticated resource management strategies in average organizations is acknowledged, highlighting the need for understanding the context of high token usage. Historical trends indicate that higher AI usage correlates with better performance, but effective metrics for usage are necessary.

The discussion concludes with thoughts on potential future measures, such as token taxes, to address the societal impacts of AI and the widening gap between leaders and laggards in agentic usage. Leaders in organizations addressing new challenges demonstrate a strong grasp of the costs associated with their solutions, engaging in meaningful discussions about trade-offs and prioritization. Currently, there is a lack of effective systems for tracking token consumption at a granular level, leading to improvised solutions. Implementing automated alerts for overdrawn accounts could enhance management efficiency. As demand for AI and agent-integrated applications surges, there will be a heightened focus on token efficiency and lower-cost models, with anticipation building for innovations that could significantly alter the landscape.