A System for Testing New AI Models

Published benchmark scores may not predict how well a model fits an individual’s work. Nathaniel’s concern is that many established benchmarks are now represented in training data and that even accurate scores can be poor predictors for subjective tasks such as writing—not that companies are lying. With frontier models arriving about every 11 days on average, he recommends testing candidates on recurring work that matters to you.

Nufar’s five-step process is to choose realistic, repeatable tasks and candidates; run the tasks; hide which tool produced each output; evaluate the results; and then decide whether to change anything. Candidates can include models, agent tools, features, combinations of tools, and the current workflow as a baseline. Tests can be run manually, with a shared script, or on an evaluation platform. The host suggests roughly four to six use cases, prioritizing recurring tasks tied to real decisions where mistakes are easy to spot. Include both new work and work on existing material, plus one or two tasks where AI has disappointed before.

The host’s examples included LinkedIn posts, a difficult stakeholder pricing conversation, tool-feature research, course website content, course exercises, and weekly planning. No model had yet produced a satisfactory intricate weekly plan in one pass. For fair comparisons, use identical requests in fresh chats, then blind the outputs to reduce bias toward familiar tools. The host tested seven options across six tasks and found the review process extremely tedious. A friend, randomized spreadsheet, AI tool, or script can anonymize results; a script requires API keys. OpenRouter offers access to multiple models through one key, while separate vendor keys are another option.

The user should remain the main judge because personal taste and preferences can be hard to capture in a rubric. Side-by-side comparisons may be easier than rating outputs in isolation; the suggested method pairs a one-to-five score with a brief observation. An AI judge can add perspective, but its judgments should be checked against the user’s own preferences, and using a judge outside the evaluated model families may reduce family bias. In the host’s tests, generated training-portal websites were fairly complete, with copy and navigation, and one stood out when content and functionality were considered together. None of the LinkedIn drafts earned a five or seemed ready to publish; one scored two for being too light and using a disliked “A and not B” construction. Because identical prompts and settings can produce different results, repeat runs when a decision is consequential: the host loved one website-generation run and liked a second much less.

Nufar compared seven models across tasks, with participants scoring outputs before model identities were revealed. The results favor choosing by task rather than seeking one overall winner. An AI judge ranked Fable first, but Nufar found it much more expensive than the alternatives. Nufar’s average scores for Fable and GPT Sol were equivalent, while GPT Sol cost about an order of magnitude less and took a similar amount of time on average. Grok had extremely long average run times despite manageable cost; Kimmy was lowest in both cost and duration, and Model 5.1 was probably the most expensive across the tested tasks. Nufar and the judge agreed only on pricing pushback. The judge assessed the hub website through code and text rather than visual appearance, illustrating how API-based evaluation can miss visual quality. Nufar would probably rely more on personal assessment than on the judge in future.

Task-level results can inform whether to add a model to a roster or consider switching, but they do not settle the decision. Company policy, terms, security, cost, latency, usage limits, and the effort of changing habits can outweigh a performance gain; the outcome might be switching, splitting work among tools, or staying put. Per-task cost estimates may also differ from actual spending when subscriptions subsidize token use, though cost matters more for people who regularly hit rate limits. Nufar is considering using GPT more and might move primarily to GPT. Benchmarks should be rerun when a significant model shift is under consideration, and tests should use the full harness, context, and connector setup when a task depends on them; a third-party API may see only part of an operations environment.

Evaluation standards can change as capabilities improve: after seeing a demo of an ultra-fast mode, Nufar became more sensitive to speed and found waiting up to 20 minutes for a generated website frustrating. For standardized team evaluations, LangSmith, Braintrust, and Langfuse were cited; small scripts may be enough for personal use. Decent models, including open-source options, may be indistinguishable on table-stakes work such as email drafting or basic research, while more sophisticated tasks can expose gaps; a good harness can make open-source models sufficient for many knowledge-work tasks. Agentic-task instructions are easier to assess when they specify audience, constraints, length, and required sections rather than simply asking for a “good” blog post. The discussion did not resolve whether models were tested at equivalent effort settings; those settings may not be comparable, so testing multiple settings or using OpenRouter defaults was suggested.