Technology & AI
Neural Networks Are Already Clean
The field believed neural networks were tangled knots. That idea is called superposition: when a network runs out of neurons, it packs unrelated concepts into the same neuron. So researchers studied second models like sparse autoencoders instead of the network itself. Cammarata, who helped propose superposition, now argues the famous evidence was misread. Messy neurons came from fixable problems: weak tools (feature visualization getting reward hacked), weak architecture (InceptionV1 lacked residual connections and LayerNorm), too-small models, and not knowing what patterns to look for.
Key ideas
- The famous 'cat and car' neuron is really a V-shape detector. Pointed things fire it. Round things push it negative.
- The canonical superposition circuit (car into dog neurons) was actually shared pose between cars and dogs.
- Recursive generalization: neurons grow like tree branches. An 'iy' neuron grows to 'diy', then 'do it yourself'. Related offshoots, not random packing.
- Squeeze a network smaller and neurons get coarser (one car neuron instead of many), not scrambled.
- A modern ConvNeXt the same size as InceptionV1 has about 99% interpretable last-layer neurons, versus about 0-5% before.
- In Qwen3-32B, about 90% of neurons have one main job, about 7% have two related jobs, and only 2-4% are not yet understood. Similar numbers show up in trillion-parameter Kimi K3.
- Think of neurons as verbs or gates, not nouns. The negative side matters as much as the positive. 'Santa' lives on negative poles.
- Understanding compounds like a crossword. The more you fill in, the easier the rest gets.
- A possible bitter lesson for interpretability: scaling ordinary investigation of the real network beats clever decompositions.
Why it matters
If networks are already legible, understanding and auditing AI (safety, debugging, trust) could be far more tractable and nearer-term than feared. The piece frames this as a working hypothesis, not a settled claim.
My takeaway
Still sitting with this. The claim I want to keep testing is that the network itself may already be the thing to study.