Why Real-World AI Is Harder Than the Benchmark | EliseAI’s Mario Martone

Mario Martone spent 15 years in theoretical physics, working on subjects including black holes and string theory, but says production AI can be more intellectually complex. Its decisions affect people’s lives, with consequences that may be positive or negative. That changes the standard for success: a system must do more than perform well in a small demonstration or benchmark. It must behave reliably across large numbers of real interactions, where individual failures can carry practical costs.

EliseAI handles about five million calls a month, or roughly 60 million a year. At that volume, Martone says, 90% reliability would still produce an unbearable number of failures; even 99% accuracy is inadequate for high-risk applications. Each additional nine toward 99.999% presents another reliability challenge. The scale makes evaluation central: a method that looks successful across five or six examples may behave differently across millions of conversations, so apparent progress in a small test set is not enough to establish that a system is ready for production.

EliseAI has built evaluation loops in which about 10 people review automated conversations daily. Martone says the company has largely cracked the challenge of systematically evaluating those interactions, though he notes that its testing loop is not yet fully automated. The approach reflects a broader point about real-world AI: performance must be examined repeatedly against actual use, not inferred from a handful of favorable examples. At the same time, reaching very high reliability remains a continuing engineering problem, rather than a threshold that can be assumed once a system works in typical cases.

Voice illustrates the technical demands. EliseAI has worked on it for three years, beginning before the technology was ready. A voice system may need to perform tens of operations in about a second and a half, making latency and coordination between components important constraints. Building the voice stack in-house gives the team visibility into those components and potential bottlenecks, and internal benchmarks let it compare models directly. The company has also fine-tuned models for production. But even with those tools and experience, Martone says the testing loop is not yet completely automated.

The company does not treat large language models as the answer to every task. It also uses traditional machine learning: renewal prediction, for example, relies on an in-house deep neural network rather than an LLM. Martone says LLMs can struggle with determinism and probability estimation, so the choice of approach depends on the task. His team weighs the potential impact of a problem and how quickly a given method could get close to an optimal result before committing to it. The implication is that deploying AI in consequential workflows requires matching methods to requirements, rather than defaulting to the newest or most general-purpose model.

In housing, EliseAI can handle a prospective renter’s process from initial interest through tour booking, home access, and move-in. For repair requests, it can assess urgency, create a work order, find a vendor, and call someone. But that does not eliminate the need for physical labor: the limits of robotics mean the system cannot fully automate the repair itself. Martone suggests that this could change in a year or two, presenting that timeline as a possibility rather than a current capability. He argues that moving early on high-friction problems affecting millions of apartments could leave EliseAI in a better position six months later. He also sees room to expand in healthcare, which he describes as about 40% of U.S. GDP, and in lowering affordable-housing costs in a complex sector with little technology.