A Potentially Historic AI Breakthrough

Nick Camaradas argues that neural-network internals may be much more readable than interpretability research has suggested. Because models are trained rather than explicitly programmed, their behavior is distributed across many learned numbers, and behavioral tests alone may miss what is happening inside—especially when a model acts differently because it recognizes an evaluation. Nick says modern agentic tools can now make the vast majority of neural-network detectors legible, including in models with up to trillions of parameters. That contrasts sharply with a 2023 effort in which GPT-4 explanations of GPT-2 detectors were judged clean and interpretable only about 2% of the time.

Nick attributes earlier impressions of inscrutability to several factors: misleading visualizations, messy learning in early image-model designs, and detectors that branch into related functions as they develop. A detector once interpreted as combining cat faces and car fronts, for example, responded to pointed shapes including fox snouts, train noses, flames, and martini glasses. In a 2019 investigation, he also found that a supposed car detector feeding into dog detectors involved car and pose detectors; cars and dogs can share poses. These cases suggest that apparent mixtures may sometimes reflect related detector functions rather than arbitrary combinations of unrelated concepts.

Nick’s reported comparisons show how much the tools and model design can matter. Inception v1, Google’s 2014 image model, was about 5% readable with tools available at the time, about 50% with a corrected microscope, and about 70% after retraining with modern settings. A modern design trained at the same size and on the same data was about 99% readable. In a modern open AI model, roughly 9 in 10 detectors reportedly had one main job. These figures are Nick’s early, self-reported results: they depend on how detectors are classified, have not appeared in a full technical paper, and have not been independently checked by other researchers.

The findings challenge the field’s default reliance on superposition, the idea that networks store more ideas in shared detectors than they have detectors available. Nick says he was once a staunch believer, but now thinks arbitrary packing that makes concepts unreadable may be less common than he assumed and should have been a last explanation rather than the first. He does not argue that detectors never combine ideas; he predicts that rare facts may instead ride on otherwise unrelated machinery. Liv Gorton counters that Nick addresses a narrower version of superposition: ideas can be stored in distinct combinations of detectors, and exponentially many nearly separate directions can exist even when perfectly separate directions are limited by detector count. Gorton says branching and cramming may both occur, since branching concerns how detectors acquire jobs while cramming concerns how information is stored.

Nick also suggests that larger models may be easier to interpret because their detectors become more specific. In a comparison of dog-ear detectors across three AIs about 300 times apart in size, the larger models’ detectors were more specific. But Nick and his team are the only ones to have done this work so far, and his test for whether a detector has one main job is disputed; Neil Nanda considers some examples scrambled. Nick says whether detector branches trace back to a single origin is testable, but that work has not yet been done. Nanda also expects language models to contain more facts than neurons and thinks weak activations may carry semantic information in superposition.

If detectors can be read, researchers may be able to inspect a model’s reasoning and plans, not just its answers. In the trillion-parameter model Kimmy K3, detectors for lost property and honesty activate when it is asked what to do with a wallet full of cash, supporting an answer that it should be returned. Nick says deception might likewise be visible in circuitry even when concealed in outputs, although researchers may not know how to interpret what they see. Critics question whether readable detectors explain decisions or establish that nothing important was missed. Nick describes his argument as a working hypothesis and says he is about 85% sure; reactions range from Nanda’s view that it is interesting but not a significant update to the big picture, to Stephen Casper’s “hype until further notice.” Even skeptics see reason to revisit earlier findings and investigate whether internal signals can help identify suspicious behavior, without assuming the work solves AI safety.