circuit - Crowded Space

Almost Orthogonal

2026-07-23T00:00:00+00:00

The neural storage problem</a></h2>
Here's a strange fact about large language models. A single model knows about millions of things: names, places, languages, programming syntax, old chess openings. But everything it holds at any one point of the text is a single list of a few thousand numbers [1</a>, 15</a>]</span>. Millions of things, a few thousand slots. This series is about how that's even possible, and what it costs.</p>
The clearest sign that something odd is happening comes from watching individual neurons. First, two quick definitions, so that nothing here is magic. A language model is a mathematical function: you give it a piece of text, and it gives back a probability for every possible next token, where a token is a word or a piece of a word. Inside, the function has a huge number of adjustable values, and training means tuning those values on enormous amounts of real text until the predictions become good. A neuron is one of the function's smallest parts, and it's a tiny function itself: it takes the numbers flowing through the model and produces a single number of its own. The exact formula is short, and it appears later in this article. For now, the only thing that matters is that watching a neuron means watching one number.</p>

</div>
Figure 1.</span> the picture you already know, and what one unit of it really is: numbers arrive, one number leaves</figcaption> </figure>
In 2023, researchers at Anthropic trained a small model on purpose: one layer, 512 neurons, small enough to study closely [2</a>]</span>. One neuron's number became large whenever the text was an academic citation. The same number became large on English dialogue. And on HTTP requests. And on Korean text [2</a>]</span>. Four completely unrelated kinds of text, and one number that rises for all of them.</p>

</div>
Figure 2.</span> four completely unrelated kinds of text, and one number that rises for all of them</figcaption> </figure>
The natural reaction is that the neuron is broken. The model was too small, the training too short.</p>
There's a name for neurons like this, and it comes from Greek: poly</em>, many, sema</em>, meaning. A *polysemantic</em> neuron is one unit carrying many meanings at once [**2</a>]</span>. And they aren't rare. In large models, polysemantic neurons are the norm, and a neuron with one clean meaning is the exception [2</a>, 3</a>]</span>.</p>*
Which is why one odd neuron deserves six articles. A model that predicts text well has to track far more things than it has neurons, so sharing isn't optional [1</a>]</span>. And linear algebra, at least the way most of us remember it, says sharing shouldn't work: a space holds only as many independent quantities as it has dimensions. The models clearly didn't read that theorem. Something in the usual understanding of dimension is incomplete.</p>
That gap is what this article is about. Closing it takes four ideas, each rebuilt from the ground: space, dimension, the dot product, and orthogonality. You probably use all four every week. You've probably never had to examine any of them closely. I hadn't either.</p>
One promise before we start. By the end of these six parts, you'll read the research papers this series is built on, Anthropic's superposition papers above all, the way you'd read a colleague's notes. Each part ends by naming exactly which sections of those papers just became readable, so you can check.</p>
What a space is</a></h2>
"Space" might be the most casually used word in mathematics, and it doesn't mean the room around you. A space is a collection of things that we agree to treat as locations. The points of a line are a space. So are the points of a plane, and so is the set of all lists of three numbers. In a few pages, a set of functions will be one too.</p>

</div>
Figure 3.</span> four collections, one move: treat the members as locations, choose a distance, and geometry begins</figcaption> </figure>
The word is this broad on purpose. Whenever you can point at the members of a collection and say here's one, here's another, you can treat the collection as a space, and once you add a rule for measuring distance, you can do geometry with it. That pattern never changes. Euclidean spaces, metric spaces, every exotic space you'll ever meet: some collection of locations, plus some chosen structure on it.</p>
One distinction is worth fixing early, because it prevents half the confusion people have about dimension. An object is not the space that holds it. A cube gets called "the third dimension" all the time, and it's neither a dimension nor a space. It's a three-dimensional object, one shape out of the infinitely many that three-dimensional space can hold [4</a>]</span>. Objects sit in spaces. The space is everywhere they could be.</p>
What dimension is</a></h2>
The cleanest way I know to build dimension is with motion.</p>
Take a point. It has no length, no width, no depth, just a location. Zero dimensions.</p>
Drag it in one direction, and its path is a line segment, which has length and nothing else. That's one dimension. Drag the segment perpendicular to itself, and it traces a square: length and width, two dimensions. Drag the square perpendicular to its own plane, and you get a cube. Three.</p>

</div>
Figure 4.</span> one motion, repeated: each object drags perpendicular to everything it already has</figcaption> </figure>
Notice that every step is the same step: move the object perpendicular to every direction it already has. And nothing stops the next one. Drag the cube perpendicular to all three of its directions, and it traces a four-dimensional object called the tesseract. Our world has no room for that motion, and nobody has ever seen a tesseract. But you can still count its parts.</p>

Interactive</span>