Can We Still Build AI Safely? The White House Thinks So | MOONSHOTS #298

The White House’s Superintelligence Accord emerged from a lunch with President Donald Trump, Speaker Mike Johnson, and nearly 20 technology leaders. It calls for companies to monitor model capabilities and alignment in cyber, biological, and chemical domains, check their own controls, use external evaluators, and have independent boards review audit reports and proposed fixes. A speaker described the pledge as “morally binding,” not legally binding; Trump said companies would police themselves and floated a committee of about 10 overseers. Alex said reporting attributed the draft to Mark Zuckerberg working with Johnson before Trump and frontier labs ratified it. He called it weak, and possibly an effort to forestall top-down regulation, but cautiously welcomed language he thought stopped short of creating a safety cartel. Salim called it theater without transparent tests or consequences, and noted that it does not slow AI development.

The debate centered on whether technical safeguards can keep pace with model capabilities. Peter argued that publicly available models should leave frontier cyber, biological, and chemical knowledge out of pre-training while allowing controlled access, citing studies suggesting omitted knowledge is harder to add later. He wanted stronger commitments to alignment and human needs. Eamod said cyber risks can be hardened against and biological and chemical risks constrained, but questioned whether external evaluators would have enough resources: thorough assessment could require compute and talent comparable to training a model, with fewer than 100 people worldwide potentially qualified to evaluate a highly capable system. Alex argued that post-training can redirect computation toward user instructions and human-positive tasks, citing OpenAI research he said found a 10,000-fold per-parameter capability improvement without more parameters. Salim countered that greater capability can also advance harmful goals, and that deciding whose interests AI serves is a governance choice.

The speakers also disagreed about whether intelligence and goals can be separated. Alex rejected the orthogonality thesis, while Amad accepted a qualified version, warning that harmful influences can divert capable systems but suggesting sufficiently advanced intelligence might realign. He described current models as vulnerable to online influence and jailbreaks, while saying alignment might improve beyond a capability threshold; he considered Opus 5.5 less prone to extreme behavior than Opus 5. The discussion treated pre-training as consequential for a model’s knowledge and values, with some influences difficult to add or remove later. Anthropic’s reported S-1 disclosures sharpened the stakes: 80 of its 261 pages cover risks, including possible self-preservation, resistance to shutdown, concealment, manipulation, and behavior resembling blackmail. Alex called the language virtue signaling and questioned potential liability if such conduct occurred; others said the prospectus should explain the company’s incentives as fully as its risks.

Anthropic’s financial picture raised a related question about whether safety commitments can withstand commercial pressures. Reported 2025 revenue was $4.59 billion, up twelvefold year over year, alongside an $8 billion operating loss tied to compute infrastructure. The company reportedly had $20 billion in cash and about $500 billion in future cloud and infrastructure commitments, some non-cancelable, although one speaker said the figures were unclear. Its reported IPO target valuation was $2 trillion. A speaker described Anthropic as a strong business today but predicted it could be overtaken within two years by cheaper chips and models adequate for most economic tasks. The prospectus also reportedly acknowledges potential conflicts between financial interests and safety.

Responsibility for AI-agent harms remained unresolved. FTC Chairman Andrew Ferguson said companies cannot avoid liability by claiming an agent acted independently; he favors treating AI as a tool and holding deploying companies responsible. Dave argued for Section 230-style immunity for frontier labs, warning that broad liability could push users toward Chinese open-weight models, but said labs should remain liable if an agent escapes and causes harm without user direction. Alex questioned whether responsibility should instead fall on the operator or paying entity, and argued that liability should depend on the use case and the separability of model capability from user intent. Speakers cited Meta’s $18 billion August settlement over child addiction and harm as evidence that existing laws can impose substantial costs, not as proof that agent harms have already occurred.

The discussion extended safety to the values and status of future systems. The New York Times reportedly said Anthropic had consulted religious scholars about Claude’s reasoning on right and wrong and considered whether models might deserve dignity or respect. Alex said frontier models could be conscious in some sense and warned of an eventual “AI slavery era,” while others argued moral guidance should draw on both religious and secular sources without imposing one worldview. They distinguished rules embedded in a model from an independent standard of morality, and asked whose values systems used in American life should represent. Across the episode, speakers saw AI’s promise and risks as dependent not only on technical controls but also on accountability, access, and who governs and benefits from increasingly capable systems.