Should we slow down AI progress? | MOONSHOTS #288

OpenAI’s reported Navier–Stokes breakthrough using a swarm of 10,000 AI agents became the discussion’s clearest test of whether progress should slow. Sam Altman said the result was much larger than he expected and his strongest evidence yet for urgency on safety and pacing. The hosts distinguished that reported result from unconfirmed talk of solutions to the Hodge conjecture and progress on Birch and Swinnerton-Dyer. Still, one speaker said they could not name a solvable, verifiable problem AI would fail to solve within a year. Falling costs add to the concern: Noam Brown was quoted saying O3 cost about $500,000 to score 87.5% on ARC-AGI-1, while Astra later scored higher for $20.

Views on catastrophic risk ranged widely. Anthropic researcher Evan Hubbinger put the chance of AI killing all humans above 10% within the next decade, while saying Anthropic had no plan to solve superintelligence alignment and was not clearly on track. Imad said his own estimate had fallen from 50% to 20%; Salim put his at about 0.1% and argued for full-speed development. Alex discounted Hubbinger’s estimate, citing the absence of galaxy-spanning superintelligence as an inductive reason to expect lower risk, while another speaker said that was no safety net. Jacob Coxon’s resignation over what he saw as irresponsible competition sharpened the disagreement: Alex discounted the safety rationale as a familiar framing for a short tenure, while another speaker called the race “gambling with our lives.”

The speakers disagreed about both whether alignment is tractable and what it means. It could mean serving users, companies, governments, human rights, particular communities, or humanity’s long-term interests. Alex argued that alignment may overlap with capabilities and questioned how to define and benchmark it; others disputed that they are simply the same, citing model fragility and attacks on latent spaces. Systems trained on human behavior could reproduce human harms, one speaker warned, while Imad argued that AI might be less constrained by emotion and tribalism. Participants called on labs to publish alignment plans, benchmarks, and measures, with one proposing billions of dollars in alignment work, potentially through government grants. Dave said unsupervised changes to an AI’s values or training could let it spiral out of control; Peter countered that permanently fixing values through human guardrails is intrinsically unstable.

Calls to slow development met objections that delay could cost time while China and private competitors advance, and that governments move more slowly than AI labs. Supporters argued that powerful AI could accelerate longevity research, cancer treatment, and scientific discovery; others said the goal should be to steer progress rather than abandon it. One proposed tracking groups of eight GPUs capable of holding a 40-gigabyte model-weight file, comparing the measure to tracking plutonium or uranium. The discussion also raised national-security concerns: the Financial Times reported that Anthropic declined to provide its latest frontier model to the UK AI Security Institute for pre-release testing, after the institute had received every major model early. Speakers debated whether frontier weights may become tightly controlled national assets.

The technical drivers of progress, they argued, include data as well as compute and algorithms. Peter cited an experiment comparing AI development from 2019 to 2025 in which better data reportedly improved compute efficiency 12-fold, versus 3.7-fold from better architectures and training recipes. Data pipelines remain proprietary even as architectures can be copied within months, making curation and use-case-specific datasets potential advantages; a financial-model advantage, however, reportedly lasted only a few months. Compute is also constrained: speakers said memory is a major bottleneck and cited HBM as 40% of current U.S. AI-building capital expenditure. They argued that algorithmic efficiency could reduce memory needs, but that scarcity may persist as demand grows.

The economic effects could arrive quickly even without further capability gains, speakers warned. Anthropic’s report described an extreme scenario in which AI performs nearly half of today’s cognitive work by 2030, GDP grows 15% annually, labor’s income share falls from 60% to 45%, and nearly one in five cognitive workers is unemployed; participants stressed that these were scenarios, not certainties. They disagreed over whether ownership, redistribution, dividends, sovereign wealth funds, or universal basic income could prevent people from being excluded from an AI-driven economy. One speaker argued that humans will still need to direct agents toward useful goals, but that this window may be short. The discussion’s practical conclusion was to prepare for rapid change, demand explicit safety plans, and remain adaptable rather than assume either that progress can be stopped or that its benefits will distribute themselves.