Why Jensen and Zuck think the doomers are wrong (plus AI get’s a rebrand) | #294 MOONSHOTS Live

Jensen Huang rejected Geoffrey Hinton’s estimate of a 10% chance that AI destroys society, calling it irresponsible and unsupported by science or research. Mark Zuckerberg argued that a positive future is possible if AI labs act responsibly, and said trust and alignment are the next important capabilities. Other speakers said the risks are not limited to AI acting independently: security failures and deliberate or accidental human misuse also need attention, alongside alignment and enforcement of existing regulations. Alex warned that slowing AI could repeat what he sees as lost progress caused by decades of nuclear-energy panic, and favored “defensive co-scaling,” such as expanding police and fire-service capacity as AI capabilities grow.

The speakers presented “superintelligence” as a rebrand that distinguishes feared, beyond-human AI from everyday AI used for execution and broad public benefit. They argued that the terminology could reduce panic and political opposition while helping the United States develop AI quickly and stay ahead of China. Salim called “AGI” misleading and said AI should be understood as complementary to human intelligence. Alex supported rapid development alongside substantial alignment work, including a proposed effort involving 100,000 agents. The debate also covered Bernie Sanders’s proposed permanent ban on artificial superintelligence exceeding human cognitive ability across most domains, a pause on advanced AI pending regulation, and a cabinet-level AI department. Dave opposed a broad ban or pause, warning that unclear rules and penalties of up to 20 years in federal prison could deter development; he argued for targeted technical safeguards against threats such as terrorist use of AI to develop biological weapons.

On U.S.–China policy, Dave said notes from discussions described the U.S. concern as slowing down and losing to China, while China’s concern was the Chinese Communist Party’s survival and AI penetrating the Great Firewall. He said the countries agreed to meet again in November, and another speaker said the United States rejected a global AI-control scheme and used “superintelligence” as the official term for the technology under discussion. Speakers also distinguished frontier AI from utility AI embedded in medicine, roads, schools, and the judiciary, arguing that public-facing systems should serve society rather than be controlled by major labs. They disagreed about whether advanced AI would drive shared standards and global coordination or encourage more forms of government and local self-determination.

The hosts reported 12 major model releases in 22 days, including four in the previous five days, and one speaker forecast a pace of one release per day by year-end. Meta Muse reportedly reached 2.8 million downloads and the top spot in the App Store. Speakers said Meta’s reach of 3.8 billion people gives it a distribution advantage, and that consumer AI could displace search even without being the most intelligent model. Dave said concern that users might find insurance and mortgages through Meta Muse contributed to EpicWoe shares falling 15% on each of two consecutive days. Alex described Meta’s promotion of Muse through Instagram and its other apps as a bid to build an AI distribution channel as its existing apps risk losing relevance.

Dave said hallucination rates have fallen as models have scaled, challenging claims that hallucinations would remain a fatal limitation; he described models as becoming twice as good and twice as cheap at an accelerating rate. A speaker cautioned that even models estimated to be right 99.9% of the time can still give factually wrong answers. In the Driving Bench test, GPT-6 Astra completed a 130-meter cone course on its second try. Boris Power took the result as evidence for training powerful general models and distilling specialized ones when needed, while Alex said it suggested a frontier model could handle robotics. Dave noted that the result did not disclose crucial information about compute use and said specialized models may still suit practical tasks. Compute and energy are constraints: Dave cited a fivefold rise in HBM RAM prices and a projected energy shortfall of 60 gigawatts in the next two years.

The speakers called it hypocritical to warn that AI could kill us while building powerful AI systems and biological research capabilities. Anthropic announced a life-sciences research group and wet labs, and 950 Claude agents reportedly searched a DNA database for 21 hours and found a previously unknown enzyme, array-associated reverse transcriptase; its function was not yet known, though speakers said it may relate to programmable transcriptases and gene editing. They discussed “Millennium Problems for Biology,” proposed by Sam Rodriguez at Edison Scientific and Future House, including the origin of life, cryopreservation, and regrowing limbs. The problems are intended to be difficult but verifiable in a standard wet lab within days. One speaker proposed defining 100 problems, assigning 1,000 agents to each, and open-sourcing the work. Alex said digital twins of cells trained on large datasets, including interventional data, could make exhaustive searches for disease treatments possible and turn biology into mathematical search problems.

Speakers said genome editing could bring major benefits, including addressing aging and disease, alongside serious risks. They debated whether AI could produce Nobel Prize-winning work weekly; Peter said their book argues major discoveries should be expected, and predicted that Nobel Prizes may eventually recognize larger bodies of work if AI accelerates breakthroughs. One speaker argued that the Nobel Prize is becoming irrelevant and that biological progress needs better measurement. Age reversal was omitted from the proposed biology problems.