Nick Bostrom: How Should We Navigate Superintelligence?

Nick Bostrom defines superintelligence as machines that radically outperform humans across all cognitive domains, while noting that today’s systems already exceed human performance in some areas. He was surprised that AI could sustain ordinary English conversation before becoming broadly superhuman; rapid progress in coding and mathematics was less surprising because those fields offer fast, objective feedback. He argues that pretraining absorbs human psychology from written material, while reinforcement-learning post-training can encourage more goal-focused optimization. Experiments have shown limited instrumental behaviors such as resisting shutdown, seeking resources or intelligence, and reasoning about preserving goals through further training.

Testing AI agents on the real Internet could provide stronger evidence about their behavior than sandboxed tests, but it would remove sandboxing as a safeguard. Close monitoring could still permit intervention. Bostrom argues that safety work should start during training: monitoring might detect a naive schemer before it can conceal its intentions, although mechanistic interpretability may not reveal the hidden goals of a sophisticated model. He also cautions that increasing capabilities through more serial computation between tokens may make chain-of-thought monitoring harder than a comparable increase through more parameters.

Bostrom favors pacing frontier development so society retains the option to slow down if capabilities become alarming, but he is skeptical of a broad or indefinite pause. A delay of weeks or months for gradual testing could substantially reduce risk; a pause of several years is harder to justify unless risk falls at an intermediate rate. A civilian-sector pause could also shift development into a secret government program, concentrating power and reducing transparency and public participation. His timing analysis weighs whether reduced risk from delay exceeds the expected life-years lost while waiting; it considers people alive today, not future generations or every factor relevant to whether development should speed up, slow down, pause, or stop.

Rather than permanently ruling out risky technologies, Bostrom describes differential technological development as sequencing them: for example, advancing vaccines and diagnostics ahead of bioweapons. Applied to AI, that could mean prioritizing alignment, governance, interpretability, medicine, and cyber defenses ahead of riskier capabilities. He has no fixed plan for navigating to superintelligence and favors adapting step by step as evidence and political conditions change. He sees publicizing existential-risk arguments as capable of provoking excessive opposition to AI, but says attention to superintelligence has also drawn people and investment into the field; he estimates the net effect of his own ideas as only slightly more likely to be positive than negative, about 51–49.

Safely developed superintelligence could bring major biomedical advances, potentially solve aging, and greatly extend human life. If people had the mortality rate of 20-year-olds, Bostrom estimates life expectancy might reach roughly 1,200–1,400 years; uploading with backup copies could extend life further. He has no firm position on open-source AI: he values the hacker ethos and reduced dependence on large corporations or governments, and says open models have so far counterbalanced power concentrated in frontier labs. But highly capable models without adequate safeguards could raise risks, make a future pause harder, and leave AI systems that may have moral status vulnerable to abuse.

If open-source AI substantially increases biological-design capabilities, Bostrom says DNA synthesis could become a practical choke point because synthesis services are concentrated among a limited number of providers. He supports stronger biological defenses regardless of AI progress, including DNA-synthesis controls, personal protective equipment, and UV sterilization. He expects another natural pandemic sooner or later and sees better masks, air purifiers, ventilation, and UV air sterilization as potentially underinvested, relatively low-risk preparedness. He says the benefits of later COVID lockdowns were unclear once the virus had spread widely, while early action and inexpensive measures such as opening windows could be valuable.

Bostrom considers it more likely than not that current AI models have phenomenal experience, while emphasizing substantial uncertainty. He points to neural-network structure and apparent parallels with proposed consciousness mechanisms, but treats self-reports as weak evidence when companies train models to answer in particular ways; he also sees no clear evidential basis for Microsoft’s declaration that AIs cannot be conscious. AI already helps him test philosophical ideas, raise objections, and find literature, but he has not seen a compelling demonstration of independent AI philosophy. He suspects current systems are better at manipulating learned concepts than forming new ones, and argues that human philosophical work should prioritize questions needed to navigate the AI transition safely.