How Claude 3.7 Plays Pokémon

Alessio introduces David Hershey from Anthropic and co-host Veeboo, who share insights on Pokémon and David's project, Clod Place Pokémon. Initially focused on Magic the Gathering, the project shifted to Pokémon due to its engaging nature. David discusses the evolution of Clod Place Pokémon, which began in June of the previous year, aiming to create a framework for testing long-running tasks. He notes significant advancements in the recent Sonic 3.7 version, despite some limitations.

The project serves as both entertainment and a means to analyze the model's capabilities. David highlights improvements over eight months, attributing the choice of Pokémon to nostalgia and straightforward mechanics. The model's core loop involves building prompts, calling the model, resolving tool use, and summarizing results, with a knowledge base maintained through conversation history.

A speaker discusses the Navigator tool, which enhances spatial awareness on a Game Boy screen, noting challenges with transitioning between zones. The model's knowledge of Pokémon types and weaknesses raises questions about its gameplay effectiveness, as it sometimes hallucinates information and struggles with spatial awareness, particularly in understanding its position on the screen.

The conversation delves into token usage, with the system prompt around 1,000 tokens and the knowledge base reaching up to 8,000 tokens. Memory management reveals that 30 turns before summarization yield better results. The model's navigation performance is hindered with complex tasks, and while it can generate plans, it sometimes leads to incorrect assumptions.

The speakers express uncertainty about the model's knowledge being beneficial or detrimental, aiming to evaluate its capabilities beyond memorization. Transitioning from version 3.5 to 3.7 did not significantly degrade performance, and simplifying prompts has proven advantageous. A tense Pokémon battle illustrates the model's attachment to its Pokémon, even nicknaming them, fostering a protective instinct.

The potential for the model to learn from various Pokémon games and retain knowledge is discussed, with limited insights noted. The conversation emphasizes the challenges of training models to understand gameplay, particularly the confusion caused by excessive button pressing. Concerns about memory management arise, as the model sometimes forgets or underutilizes information.

Gaps in the model's ability to visually navigate and remember game environments are significant challenges. Despite skepticism about achieving major milestones quickly, there is optimism for future improvements. A personal highlight is the excitement of beating Brock after eight months of effort. Future projects may include work related to Magic: The Gathering, with discussions on the model's real-world applications and the need for more quantitative measures to assess progress effectively.