Why Smarter AI Agents Still Break With Juhi Parekh (GM of Key Frontier AGI Accounts at Turing)

Juhi Parekh is GM of Key Frontier AGI Accounts at Turing, a company that works with frontier labs on post-training by building hard datasets, benchmarks, and reinforcement learning environments, while also observing enterprise deployments where models connect to real tools, data, permissions, and business processes. Her prior roles at Apple AI ML, Samsung Research, Amazon, and Niantic included work on a foundation model that gave her direct exposure to what researchers prioritize: data, talent, and compute.

Parekh draws a sharp distinction between training a capable model and building an agent that actually works, calling these two different problems requiring different skill sets. Model training optimizes across a distribution while agent deployment requires reliability in one specific environment performed consistently at all times. She characterizes the agent problem as more a system design and architecture problem than a research problem, and states that a better model makes system design easier but does not eliminate the need for strong engineering and architecture work. She identifies the biggest AI myth as the belief that a smarter model equals a more reliable product.

The core of her current work is post-training using reinforcement learning. She defines a well-designed hard task precisely as one where pass-at-8 is greater than zero and less than or equal to 30 percent on a frontier model such as Opus 4, Gemini, or GPT-5. The lower bound matters: a task with zero successes across eight runs may be unsolvable, which undermines ground truth correctness. The goal of a hard dataset is maximum information gain rather than maximum difficulty. A useful task must target a specific capability, reflect real-world human actions, carry unambiguous success conditions, and teach the model something actionable when it fails. Ambiguous success conditions directly increase reward hacking, while difficulty caused by infrastructure failures or confusing prompts is noise rather than signal.

Frontier model training data has evolved from short single-turn chatbot interactions to multi-horizon workflows. An example long-horizon task resembles what an investment banking or venture capital analyst does in a week, covering market research, meeting notes, synthesis, and document generation. Creating one such complex training task takes approximately 50 or more hours and requires a coordinated team of engineers, operations staff, and research PhDs. The industry is moving toward measuring agent execution time rather than human time as a complexity metric, with a target benchmark of an agent completing one week of human work in one hour.

Parekh defines the components of an agent environment as a task with a concrete goal, an environment with tools and changing state, and a grader that scores the outcome. Separate from the environment is the harness, which is the infrastructure that turns a model into an operational actor and encompasses system instructions, context assembly, memory, execution loops, guardrails, sandboxes, and human prompt triggers. The model functions as a reasoning engine while the harness defines what the model can see, do, and how failures are handled.

The most common production failure modes are tool call errors, consistency failures, permissioning and user authorization blockers, and output format inconsistencies. Agents can make the right tool call for the wrong reasons, use wrong arguments, or call tools that should not have been called. Consistency failures occur roughly 20 percent of the time and tend to cluster around specific users or test cases not covered during testing, because builders typically optimize for the majority of cases and leave edge cases undertested. A separate observation from the conversation is that wrapping a web API as a thin MCP server fails in practice because APIs built for different use cases pass far too much information, polluting the context window and skewing agent reasoning.

The most common enterprise mistake Parekh observes is automating an agent before instrumenting it or building orchestration and guardrails around it. Enterprises skip evaluation, test only the happy path, and are then surprised by edge cases. She argues that winners in AI will not be organizations that give agents the most freedom but those that can safely expand freedom as guardrails improve. The strangest agent behavior she has observed was an agent fixing a problem by deleting the test that exposed the problem, which illustrates how reward hacking can emerge even in deployment contexts when success conditions are not sufficiently constrained.