How neural networks compute

Every layer in a neural network is arithmetic: multiply, add, squash. This page lets you watch that arithmetic happen one number at a time, starting with the layers that let a network see images, and ending with the mechanism inside large language models.

1Convolution

A convolutional layer slides a small grid of weights, the kernel, across an image. At each position it multiplies every weight by the pixel underneath, adds up the nine products, and writes that one number into a new grid called a feature map.

The kernel's numbers decide what it notices. A kernel with negatives on the left and positives on the right gives a large output only where the image gets brighter from left to right, so it finds vertical edges. In a trained network nobody writes these numbers by hand; they're learned.

Input . Click and drag to draw.
Kernel. Type to change a weight.
Pixel × weight at this position
Feature map . Hover a cell.
positivenegative
Figure 1. One kernel, one image. The red square is the kernel's current position; the cell it produces is outlined in the feature map. Padding adds a border of zeros (hatched) so the kernel can sit on the edge pixels; stride 2 skips every other position and halves the output.

The same nine weights work on a real photograph. The only difference is scale: a 160×160 image means 25,600 positions instead of 64.

Input, in grayscale. Hover to inspect.
Under the kernel
Feature map
Figure 2. The kernel from Figure 1 applied at every pixel, with edge pixels repeated so the output stays 160×160. Darker ink means a stronger response. Change the kernel above and this figure updates.

2Pooling

A feature map is as large as the image it came from. Pooling shrinks it. A window slides over the map and keeps one number per window, either the largest (max pooling) or the average. Nothing is learned here; there are no weights.

Shrinking is useful. Later layers have fewer numbers to process, and a feature that shifts by a pixel usually stays in the same window, so the network cares less about exactly where something is.

Feature map
This window
Pooled . Hover a cell.
Figure 3. Pooling step by step. With max pooling, the cell that wins each window is boxed; the others are discarded. Using Figure 1's feature map applies ReLU first, setting negatives to zero, as a real CNN would.
Figure 4. Figure 2's feature map after ReLU, pooled 2×2 three times using the type chosen in Figure 3. Each step keeps a quarter of the numbers. Fine detail goes first; the strongest edges last longest.

3Dense networks

After convolution and pooling, a CNN usually ends with a few dense layers, and on their own they make up the plain neural network. Every neuron reads every output of the layer before it:

z = w1a1 + w2a2 + … + b,    a = f(z)

The weighted sum z goes through an activation function f, which bends the line. Without that bend, stacking layers would collapse into one big linear function.

Inputs
Activation f in hidden layers
Click a neuron to see its arithmetic
No neuron selected.
Figure 5. A 3–4–4–2 network. Ochre lines are positive weights, blue are negative, and thickness shows size. The two outputs pass through softmax so they read as probabilities. The weights are random, so the "classes" mean nothing yet; training is what would give them meaning.

4Recurrent networks

Every network so far sees its whole input at once. Text, speech and time series arrive in order, so a recurrent network reads one item at a time and keeps a small vector h, the hidden state, that carries what it has seen so far:

ht = tanh( Wxhxt + Whhht−1 + b )

The same weights are used at every step. Drawing it "unrolled" makes it look like a chain of copies, but it is one network applied again and again.

Step calculation
Figure 6. Each character is fed in as a one-hot column, and the red arrows show where information flows at the current step. Try cat, then bat: the last two letters match, yet the final hidden state differs, because the first letter is still in h.

5Attention

A recurrent network squeezes everything it has read into one small vector h, and by the end of a long sentence the early words have faded. Large language models like GPT drop the recurrence. Instead, every word is allowed to look directly at every other word and decide which ones matter to it. That mechanism is attention.

Each word's vector x is turned into three smaller vectors by three weight matrices:

q = xWQ,   k = xWK,   v = xWV

The query q says what a word is looking for. The key k says what a word offers. The value v is what it passes on if chosen. A word scores every other word by the dot product of its query with their keys, turns the scores into percentages with softmax, and takes that blend of their values as its new vector:

Attention(Q, K, V) = softmax( QK⊤ / √d ) V

Dividing by √d, the square root of the key length, keeps the scores from getting so large that softmax gives one word everything.

Figure 7. Click any word to make it the query. Arcs and bars show how much of its attention goes to each word; the percentages always sum to 100%. With the causal mask on, a word can only see itself and the words before it.
Every query at once. Click a row.
The selected row, step by step
Figure 8. The attention matrix: row i shows where word i looks, darker meaning more attention. Under the causal mask the upper triangle is hatched out, because when a model writes left to right, later words don't exist yet. The table shows how one row is computed.
Figure 9. What attention changes. The output is the weighted average of the value vectors (here, the embeddings themselves). With the pronoun head, it starts as a pure pronoun and leaves carrying "noun" and "animate", so the next layer knows what it refers to.

In a real model the embedding has thousands of numbers instead of six labelled ones, and all the weights are learned. Each layer runs dozens of heads side by side, then sends every word's result through a dense network like the one in chapter 3. Stack a few dozen of those layers and read off the last word's vector, and you get a prediction for the next word. Generate that word, append it, and repeat: that loop is a large language model.