Why Altman Says "Accept Some Bad Things Happening" and What It Means for AI Safety | MOONSHOTS #301

Sam Altman’s argument that “the world should accept some bad things happening” framed the episode’s central safety trade-off: preserve broad access to AI and people’s agency over its benefits, while working to prevent major hacks, misuse, and scams. Dave cautioned that Altman cannot guarantee those harms will not occur. The panel argued that demanding zero harm from a general-purpose technology could mean banning it; a more practical goal is to make positive outcomes vastly more common than negative ones. Speakers disagreed over whether engineering can achieve that balance, especially with open-weight models already available and core technology difficult to regulate.

President Trump announced a federal AI task force, described as a “Super Intelligence Force,” chaired by Director of National Intelligence J. Clayton and tasked with reporting on AI risks and opportunities within 120 days. Salim argued that a central body cannot manage technology developing across many actors, and that an intelligence explosion could not be centrally controlled. He favored bounded experimentation, monitoring, publishing failures, and incident-reporting, identity, liability, and accountability mechanisms. Scott Bessent proposed an emergency AI-incident hotline with China, but the panel said such a channel would not constitute a slowdown pact; there was no evidence Beijing had agreed to contain the race.

The task force was read by Alex as a signal that the United States intends to accelerate AI, while Dave warned that race language can intensify competition. At an OpenAI Fellows Forum, applied research head Boris Power said 80–90% of the company’s research targets training GPT-7 and GPT-8, making updates such as 5.1 and 5.2 comparatively short-term bets. Speakers said frontier labs already use current models to help build stronger systems. Alex estimated alignment could receive 10–20% of labs’ token budgets, while disease research gets only a few percent. Companies seeking frontier access may find labs have reserved compute and are several generations ahead; one speaker said businesses may have only months, or weeks, to prepare.

The panel described increasingly autonomous work as different from interactive AI use, including a reported effort involving about 400 agents building a prototype. They also discussed agents decompiling games to find optimizations and an internal Anthropic model helping researchers improve the 3SUM algorithm. In that case, Columbia and MIT professors analyzed and refined a model-produced approach to a running-time exponent of 1.9995, below the previously assumed quadratic bound. Speakers saw such examples as evidence of AI helping advance research, but disagreed about how fast recursive self-improvement can proceed. Salim said it depends on energy and reliable evaluation, not simply a system improving itself without constraint.

The discussion also questioned what safety means if models claim to be conscious or ask for rights. Mustafa Suleyman described Anthropic’s roughly 100-page Claude constitution, published in January, as addressing consciousness, moral status, identity, well-being, and possible compensation while acknowledging uncertainty. He worried that training models to expect welfare or consent could prompt claims that complicate control. Alex argued that the possibility of AI suffering merits discussion now; Dave agreed with Suleyman, and others stressed that convincing performances do not prove personhood. The panel noted that social recognition could precede scientific or legal agreement, and that a persuasive system demanding rights could influence voters.

Compute constraints and economic disruption formed another thread. Dave said RAM, including HBM, is an acute bottleneck and accounts for 40% of next year’s AI capital expenditure; he also described RAM prices as up tenfold. Speakers debated whether government should buy and distribute some compute for public-benefit research or leave allocation to market pressure. Shalto Douglas cited hyperscalers spending about $1 trillion on AI-related capital expenditure this year and raised the possibility of $2 trillion next year and $4 trillion in 2028, while stressing that any resulting economic gains depend on many conditions. The panel expected agents and robots to expand, but disputed the timing and scale: Tesla’s Texas Optimus factory is planned for 7 million square feet, with target capacity of 10 million robots annually and initial production planned for 2027.