Why Jensen and Zuck think the doomers are wrong (plus AI get’s a rebrand) | #294 MOONSHOTS Live

Jensen Huang rejected Geoffrey Hinton’s estimate of a 10% chance that AI will destroy society, calling it irresponsible and unsupported by science or research. Huang argued that such warnings can make AI sound uncontrollably powerful and obscure human responsibility for how it is used. Mark Zuckerberg also argued that AI can have a positive future if developers act responsibly, while identifying trust and alignment as essential next capabilities.

The panel distinguished deliberate or accidental misuse, security failures, and infrastructure problems from a sudden AI breakout, warning that treating these as one risk can muddy the response. Eman called for evaluating risks individually, enforcing existing rules, and working on alignment; Celine said flourishing was far more likely than doom, while noting that 99.9% of species are extinct. Alex argued that slowing AI would be the worst outcome and could cost decades of progress. He favors “defensive co-scaling”—building institutions such as police and fire services alongside AI—over deceleration, and views some industry-wide safety coordination as an attempt to form a cartel.

The terminology debate centered on whether “superintelligence” better separates familiar, helpful AI from the advanced systems people fear. One speaker suggested the term could reduce political panic and resistance while allowing development to continue amid competition with China. Salim objected that “AGI” is misleading because AI complements rather than replicates human intelligence; Alex preferred “superintelligence” as a way to convey cognition on the scale of many trillions of humans. The panel also disagreed over a political response: Bernie Sanders proposed pausing advanced AI until rules are written, permanently banning superintelligence, and creating a cabinet-level AI department, with company shutdowns and up to 20 years in federal prison as penalties. Dave opposed turning AI into a political yes-or-no fight, arguing for specific technical responses to risks such as terrorist use in bioweapons and China pulling ahead.

Dave said notes from Alvin Grail described the U.S. priority as preventing domestic panic and avoiding a slowdown that could leave the country behind China. China’s concern, he said, was the Chinese Communist Party’s survival and AI penetrating the Great Firewall. U.S. and Chinese delegations agreed to meet again in November, while the United States rejected a global scheme to control AI. The discussion also noted that models are becoming more capable and accurate while cheaper, and that scaling has sharply reduced hallucinations. But even a model that is right 99.9% of the time can prompt dangerous overconfidence when its remaining answers are wrong.

A Toyota Corolla completed a 130-meter cone course in the Driving Bench test after receiving one instruction, with GPT-6 Astra succeeding on its second try. Boris Power said the result supports training powerful general models and distilling specialized ones when needed; Alex and Ema suggested general models might eventually make separate robotics models unnecessary. Dave cautioned that the test does not establish practical driving capability and that the compute required was not reported. He also said compute is becoming a bottleneck, with energy potentially following: HBM RAM prices had risen fivefold, and he warned of a possible 60-gigawatt energy shortfall within two years. Alex called it hypocritical for AI labs to warn that AI could kill us while building wet labs and using frontier AI for biology research.

Anthropic reportedly used 950 Claude agents for 21 hours to search a large DNA database and identify an enzyme called array-associated reverse transcriptase. Researchers do not yet know exactly what it does, but believe it may relate to programmable transcriptases and gene editing. Salim said genome editing could help address aging and disease while also creating serious risks, giving “P-doom” advocates grounds for concern. Edison Scientific and Future House proposed “Millennium Problems for Biology,” including the origin of life, cryopreservation, and regrowing limbs. They argued that problems should be difficult yet verifiable in a standard wet lab within days, and that many targets could be assigned to agents and open-sourced. Age reversal was omitted, though one speaker said it might be measurable as making someone functionally 20 years younger.

Alex predicted that digital twins trained on large datasets of cells and interventions could enable systematic searches for ways to move diseased cells toward healthy states. He argued that such tools could turn hard biology questions into mathematical targets that are easy to verify, and that solving the top 5,000 diseases and achieving durable longevity escape velocity would require superintelligence. The panel said biology needs a framework for measuring progress; Alex also predicted Nobel-caliber work could become abundant enough that prizes may need to recognize larger bodies of work.