What a neural network actually computes

"Neural network" evokes brain imagery, but the actual computation is much more mechanical: a neural network takes a list of input numbers, multiplies each by a weight, adds them up along with a bias term, and passes the result through a simple nonlinear function. That's one neuron. A network is many of these, arranged in layers, each layer's output feeding the next layer's input.

Without that final nonlinear step (the activation function — common ones are ReLU and sigmoid), stacking layers would be mathematically pointless: multiplying and adding numbers through many layers with no nonlinearity in between collapses back down to the exact same thing as one single layer, no matter how many layers you stack. The nonlinearity is what actually lets a deep network represent complex, curved relationships in data rather than only straight-line ones — it's not a minor implementation detail, it's the specific thing that makes "deep" (many-layered) networks meaningfully more capable than shallow ones.

A network stacks many of these neurons into layers, and stacks many layers on top of each other:

Each layer receives the previous layer's output and produces a new representation — early layers in an image-recognition network tend to detect very simple patterns (edges, colors), and later layers combine those into increasingly complex, abstract concepts (shapes, then object parts, then whole objects). Nobody explicitly programs what each layer should detect — it emerges from training, which is exactly why interpreting why a specific neural network made a specific decision is often genuinely difficult, even for the people who built it.

Every one of those multiplication weights (and biases) starts as a random number. Training is the repeated process of: run an example through the network, compare its output to the correct answer, and nudge every weight slightly in whatever direction would have made that output a little closer to correct — repeated across millions of examples, adjusting billions of weights incrementally each time. This nudging procedure has a name — backpropagation, paired with gradient descent — but the mechanism-level takeaway that matters here is simpler: training is just repeated, automatic, small corrections to a huge number of adjustable numbers, driven by how wrong the current output was.

A neural network with a handful of neurons can only represent very simple patterns. Modern large language models have on the order of hundreds of billions of these weights. The core computation described in this lesson — multiply, add, squash, stack in layers — is genuinely the same basic idea at every scale; what changed to produce today's dramatically more capable models is overwhelmingly the scale of data, weights, and compute, not a fundamentally different algorithm. Understanding this one mechanism at small scale is what makes "how does GPT actually predict text" (three lessons ahead) demystifiable rather than magical.