Should we slow down AI progress? | MOONSHOTS #288

OpenAI’s claimed solution to the Navier–Stokes Millennium Prize problem, reportedly generated by a swarm of 10,000 AI agents in September 2026 at a cost of a few million dollars, sharpened the debate over whether AI progress should be slowed. Sam Altman called the result more significant than he expected and his strongest evidence yet for pacing progress on safety grounds. Other speakers pointed to rapid improvements in mathematical performance and cost: Noam Brown said a system that cost about $500,000 to score 87.5% on ARC-AGI 1 was later surpassed by Astra for $20. One speaker characterized that change as a 25,000-fold cost decline in two years.

The panel disagreed over what these advances mean for risk. Jacob Coxon, who worked on pre-training at OpenAI and Anthropic, alleged that both companies were pursuing self-improving superintelligence despite believing sufficiently powerful systems could pose catastrophic risks. Alex questioned the credibility of Coxon’s claims, citing his short tenure, and raised—but did not establish—the possibility of foreign influence. Anthropic alignment lead Evan Hubbinger said he personally assigned more than a 10% chance to AI killing all humans within the next decade, while acknowledging that Anthropic lacked a plan for superintelligence alignment and was not clearly on track. Other estimates discussed ranged from about 0.1% to 50%; speakers stressed that such figures are subjective.

Some speakers argued that any nonzero chance of catastrophe warrants precaution, while others favored continued acceleration because they saw no workable way to stop development amid competition among companies and governments, including China. The practical disagreement was whether to halt or slow progress, or instead steer it and reduce risks while pursuing potential benefits such as longevity advances and new scientific capabilities. Several said alignment cannot easily be separated from capability work: public funding for safety research might also subsidize more capable systems. They called for labs to disclose alignment plans, benchmarks, measurements and optimization goals, and debated whether increasing AI intelligence might bring greater wisdom or leave a system’s objectives unchanged.

The conversation also questioned whether data—not novel architectures—is becoming the central competitive advantage. An analysis of AI development from 2019 to 2025 attributed 12 times the improvement in compute efficiency to better data, compared with 3.7 times to architecture and training improvements. Speakers said architectures can be copied quickly, while proprietary data pipelines may be harder to reproduce. They described training data as an information diet: ordering and curation can improve learning, while poor-quality material can degrade behavior. Companies with unique, useful business data could gain an advantage, though Bloomberg GPT’s reported lead on a financial benchmark lasted only a few months.

The panel warned that rapid capability gains do not mean systems are reliably useful or safe in practice. They described current models as “jagged”—able to tackle difficult technical tasks while still making poor practical suggestions—and argued that defining goals such as curing cancer is harder than specifying a mathematical problem. They also cited constraints on deployment: GPU and high-bandwidth memory scarcity could leave companies short of compute if they wait for cloud capacity. DeepSeek’s reported use of SSDs and DDR memory to reduce HBM needs illustrated how infrastructure limits can prompt different engineering choices; its Flash model was described as fast and inexpensive, though speakers cautioned that its cost advantage might partly reflect simulated data environments.

Economic disruption was another concern. Anthropic’s extreme scenario for 2030 projected AI performing almost half of today’s cognitive work, 15% annual GDP growth, labor’s share of income falling from 60% to 45%, and nearly one in five cognitive workers unemployed. Speakers debated the uncertainty of such forecasts, but warned that job losses concentrated among workers while returns accrue to capital could weaken demand and provoke unrest. They discussed dividends, sovereign wealth funds and universal basic income as possible responses, while noting that proposals and forecasts are not guarantees of implementation.

Alongside these risks, speakers cited early applications in medicine and biology. The New York Times reported that an AI-designed drug for idiopathic pulmonary fibrosis had reached phase III trials, which one speaker described as the first AI-designed drug to reach that stage. Research on protein-based aging clocks found treated patients’ blood signatures shifted toward those of people biologically three to six years younger, but the result was limited to a subpopulation and was not conclusive. The discussion also covered AI tools for predicting the effects of genetic variants and designing molecules. Speakers argued that such benefits depend on engineering systems to become reliable, affordable and widely usable—and that planning must adapt as capabilities change faster than institutions and daily life.