What is a neural network?

Recognizing a 3 scribbled in 28×28 pixels is instant for you. Writing an if/else that does the same — a curve here, a stem there, and what if it is crooked? — is a nightmare. The neural network in the video But what is a neural network?, by 3Blue1Brown (Grant Sanderson), breaks it down: it is not magic, it is a function with thousands of knobs.

This text follows chapter 1 of the deep learning series. The calculation the GPU performs in GPU vs TPU post is exactly this: multiplying matrices, a million times.

A neuron is a number

Forget the biological diagram. In silicon, a neuron is a container holding a number — its activation. In MNIST (handwritten digits):

• Input: 28×28 = 784 neurons. Each stores a pixel's brightness, from 0 (black) to 1 (white).
• Output: 10 neurons, one for each digit 0–9. The highest value is the prediction.
• Hidden: in the video's example, two layers of 16. The network becomes 784 → 16 → 16 → 10.

entrada 784 px   →   oculta 16   →   oculta 16   →   saída 10
  (pixels)            (bordas)         (formas)        (0..9)

Why layers

A pixel alone means nothing. The network stacks abstractions:

• Layer 2: edge fragments (short stroke, diagonal).
• Layer 3: joins edges — circle on top + stem = 9; two circles = 8; three straight lines = 4.
• Layer 4: combines the pieces and chooses the digit.

The same scheme in audio: sample → phoneme → syllable → word. That is why “deep”: depth is hierarchy, not marketing.

Weights, bias, and activation

Each edge between neurons has a weight w. Positive values stimulate; negative values inhibit (light in that region? Then this is not the pattern). Zero ignores the pixel.

The stimulus reaching neuron j is the weighted sum of previous activations, plus a bias b (the threshold: with b = -10, the sum must exceed +10 for the neuron to “light up”):

estímulo = (w1·a1 + w2·a2 + … + wn·an) + b

This number can be any real number. The activation function squeezes it into a useful range. The video uses the sigmoid (logistic curve), the classic analogy with biological firing:

σ(x) = 1 / (1 + e^(-x))
  x → -∞  ⇒  0
  x = 0     ⇒  0,5
  x → +∞  ⇒  1

In practice, deep networks abandoned the sigmoid along the way: it saturates and the gradient disappears (vanishing gradient). The standard became ReLU — cheap and sufficiently nonlinear:

ReLU(x) = max(0, x)

13.002 knobs

How many free numbers are in this teaching network?

Training means finding those 13 thousand numbers. Gradient descent + backpropagation — the next video in the series. A 2026 LLM has billions of the same knobs; the calculation does not change, only the scale. That is why a model in Ollama consumes RAM: each parameter is a float that needs to fit in memory.

One line of linear algebra

Working neuron by neuron does not scale. Stack the activations in vector a, the weights in matrix W, and the biases in vector b. The entire layer becomes:

a¹ = σ( W · a⁰ + b )

Matrix-vector product, add the bias, apply σ at each position. NumPy, PyTorch, and TensorFlow exist to do this quickly — BLAS/GEMM on the GPU. The data center's “AI” is fundamentally this line repeated.

So what is a neural network?

A function f. In MNIST: the input is ℝ⁷⁸⁴, the output is ℝ¹⁰, with 13.002 parameters inside in matrix products with nonlinearity in between. With enough data, this function learns patterns nobody would write by hand. The rest — Transformer, attention, GPU — is a variation on the same trick.

References

• Video: 3Blue1Brown — But what is a neural network? (Grant Sanderson)
• Series: Neural networks · 3blue1brown.com
• GPU vs TPU: how AI chips work
• Running local AI on Linux with Ollama

Thirteen thousand numbers to read a digit. Trillions to converse. The machine is the same.