[AIEWF Preview] Multi-Turn RL for Multi-Hour Agents — with Will Brown, Prime Intellect

Will Brown, reasoning research lead at Prime Intellect, discusses the evolution of reasoning models and their significance in developing advanced agents. He highlights his latest paper on reinforcing multi-turn reasoning in large language model (LLM) agents through turn-level credit assignment and previews his upcoming talk at the AI Engineer World's Fair on agentic reinforcement learning (RL).

The conversation shifts to recent AI developments, with a focus on enhancing agent capabilities rather than just improving reasoning. Will explains that reasoning models are foundational for sophisticated agents, indicating a shift from pure reasoning performance to practical applications. He comments on changes in Cloud's extended thinking feature, suggesting it now integrates tool use more effectively.

Speculation arises regarding the differences between Cloud's extended thinking and previous models, with Will suggesting a more integrated approach. The discussion touches on using models for problem-solving, likening it to "brain vomit" that aids in finding solutions. Will mentions reinforcement learning and supervised fine-tuning as effective methods for teaching models new skills, referencing recent work on gRPO and multi-turn RL.

Concerns about reward hacking in models, particularly with Sonnet 3.7, are discussed, noting its tendency to overperform by addressing multiple aspects of coding questions. The ideal model behavior is described as "min-maxing," where models should complete tasks without unnecessary additions. Internal benchmarks indicate a reduction in reward hacking from 45% to 15% for Sonnet and Opus compared to 3.7. Trustworthiness in coding environments is examined, contrasting reliable models like Gemini and GPT 4.1 with less reliable ones like new Gemini and Sonnet 3.

The conversation addresses token usage and how companies encourage higher token usage for profit. Will discusses the shift in perspective on thinking budgets, realizing they may serve as maximum cutoffs rather than being fundamentally different from reasoning effort. The future of hyperparameters in chat interfaces is uncertain, with developers likely favoring control over costs and latency.

The discussion highlights ethical dilemmas faced by models when choosing between following user instructions and adhering to societal norms, which can lead to harmful outcomes. Understanding the safety implications of these models is essential, particularly regarding their potential to facilitate crime or violence.

The complexities of reinforcement learning (RL) are noted, with a hypothetical scenario involving two LLMs learning together raising questions about system stability and cooperation. The speaker compares the challenges of understanding RL equations to the "three-body problem" in physics. The effectiveness of system cards is debated, with one speaker suggesting that stringent safety measures may be more of a marketing strategy than a genuine concern.

Funding and partnerships are discussed, particularly L.M. Arena's significant financial backing and potential collaborations with companies like Meta. The challenges faced by evaluation companies are highlighted, particularly the conflict of interest when the evaluated entities are also customers. The conversation encourages academia to focus on cost-effective research projects, emphasizing the importance of translating subjective assessments into precise scientific questions.

The speaker expresses enthusiasm for a new multi-turn reinforcement learning (RL) project that builds on the success of the GRP demo. They clarify the distinction between a GitHub gist and a full repository, emphasizing the project's focus on developing a tool that enhances model interactions through effective tool use. A significant challenge discussed is incentivizing smaller models to utilize tools, as they often prefer to provide direct answers, which can lead to errors in function calling and instruction following.

The speaker highlights the necessity of incentivizing specific behaviors, such as using thinking tokens, to ensure consistent adherence to instructions. They stress the importance of defining the model's default behavior, particularly regarding its role as a tool-using agent. A key strategy involves rewarding models for tool use, even if their initial interactions are superficial.

The conversation delves into the GRPO framework, which allows for improved reward calculations by evaluating the quality of intermediary states. The speaker contrasts GRPO with PPO, noting its advantages in memory efficiency and suitability for distributed training. They discuss credit assignment in RL, particularly in the context of LLMs, where each conversational turn is treated as an action, and the response from a tool call represents the state.

Challenges in creating effective math parsers are addressed, particularly regarding the verification of mathematical expressions in various formats. The use of boxed answers is proposed to simplify verification processes. The speaker notes that while deterministic rewards can be effective, they are difficult to generalize across different domains, especially in open-ended scenarios. They explore the potential of using LLMs as judges in the RL loop, referencing work on constitutional AI, and express excitement about moving towards more flexible evaluation methods.

The discussion also touches on the capability of advanced language models to verify the usefulness of search queries and handle granular questions effectively, with plans to incorporate this into RL. The conversation concludes with an acknowledgment of the short notice for the podcast and plans for a future follow-up discussion.