Neurons, Chaining, and Specialisation

So far every demo has had a single neuron: one weight, one bias, one activation function. Once you start connecting neurons together, something interesting happens, the combined output can model shapes that no single neuron ever could.

Chaining neurons (depth)

Take two ReLU neurons in series. The first neuron transforms the input with its weight and bias, then applies ReLU. The second neuron takes that output as its input, applies its own weight and bias, then applies ReLU again. Drawn with the notation from the last part, the chain looks like this:

The second neuron doesn't see the original input. It sees whatever the first neuron decided to pass on. As you tune the weight and bias of the second neuron, you are reshaping a signal that has already been non-linearised. The combined result can look like a clipped ramp, a shifted step, or other shapes that neither neuron could produce alone.

Follow some actual numbers through the chain to see this happen. Set w1 = 1, b1 = 0 on the first neuron and w2 = 1, b2 = -1 on the second (the same values the demo below starts with), and trace each input through all four steps:

Read the columns left to right and you can watch each neuron take its turn. The first neuron silences everything below x = 0 (the h column). The second neuron then subtracts 1 and silences anything that remains below that bar, so the output stays at zero all the way up to x = 1 and only then starts to rise. Neither neuron alone can produce a ramp that switches on at 1 with these ingredients, but the chain does it easily: the first neuron's cutoff and the second neuron's cutoff compose.

Try it: two ReLU neurons in series

The dashed line below is the first neuron's output, h = ReLU(w1·x + b1). The solid dark-red line is the final output after the second neuron, y = ReLU(w2·h + b2). Tune all four knobs and watch the second neuron reshape what the first one hands it.

Some shapes to hunt for: with the default settings the second neuron delays the first one's ramp, the output stays silent until the first neuron's signal is strong enough to clear b2. Now set w2 = -1, b2 = 1: the output becomes a plateau that falls to zero, a shape no single ReLU neuron can make (a lone ReLU can only be flat-then-rising or falling-then-flat, never high-then-low-then-flat with two kinks). Two bends from two neurons: each layer of depth adds another chance to fold the signal.

Parallel neurons (specialisation)

Instead of chaining, put neurons in parallel. Each gets the same input but has its own weight and bias, its own private view of the input. Their outputs are summed to produce the final result:

Now each neuron is free to focus on a different region of the input space. One neuron might only activate for inputs near -1.5; another fires for inputs between 0 and 1. When you sum them all together, you are stitching together several local linear approximations into a single curve.

Try it: three ReLU neurons in parallel

Each thin coloured line below is one neuron's output; the thick dark-red line is their sum. Because every ReLU neuron contributes a hockey-stick that switches on at its own spot (set by its bias) and rises at its own rate (set by its weight), each knob you turn reshapes one bend of the total curve while the others stay put. That is the coordination trick in miniature: no neuron knows about the others, no one is in charge, yet the sum is a shape none of them could make alone, each neuron simply handles its own part of the problem.

Slide neuron 2's bias around and watch only the left side of the sum move; neuron 1's knobs own the right side, and neuron 3 tilts the middle. Every neuron minds its own patch of the input, and the network's answer is simply everyone's contributions added up. Scale this idea from three neurons to millions and you have the working principle of every neural network: lots of simple parts, each specialised on a fragment of the problem, coordinated by nothing more than addition.

This is specialisation, neurons carving up a problem into smaller sub-problems, each handling one piece. VGGNet, a famous image classification system, uses exactly this idea: early neurons look at tiny patches of pixels, and later layers combine those local features into global understanding.

Demo 1: five neurons specialising on regions

The demo below has five neurons with fixed activation regions. Each neuron handles a different slice of the x-axis. Their summed output tries to approximate sin(x). Adjust the sliders to see how each neuron's weight and bias contributes to (or breaks) the overall fit.

Notice how changing one neuron's weight or bias only shifts that neuron's region of the output curve. Each neuron owns its slice.

Demo 2: gradient descent training a mini neural network

Now let gradient descent find the weights and biases automatically. The network below has 5 parallel neurons (each with its own weight and bias) feeding into a single output neuron that combines their signals. A group of neurons sitting side by side like this is called a layer; because this layer sits between the input and the output, hidden from both, it is called a hidden layer. The goal: approximate sin(x).

Press Train to run gradient descent. You can adjust the number of iterations and learning rate before training. Press Reset to start over with new random weights.

What to explore

• Press Train multiple times (after Reset), because weights are initialised randomly, you will get different results each time. Some runs converge faster, some get stuck.
• Try 3 neurons vs 12 neurons: more neurons give the model more capacity to approximate the sine curve. Fewer neurons means a coarser approximation.
• Adjust the learning rate: try 0.005 (slow), 0.05 (good), and 0.2 (may diverge).
• Try 50,000 iterations: the model keeps improving as long as training continues.
• Look closely at the fitted curve: every hidden neuron here uses ReLU, so the "smooth" fit is really many straight-line pieces stitched together. With 3 neurons the kinks are easy to spot; with 12 they blur into an apparently smooth wave.

The broader picture

What you just saw, a tiny network with a hidden layer learning to approximate a function, is the same fundamental mechanism behind all modern neural networks. ChatGPT, image classifiers, recommendation engines: they are all doing the same thing at a larger scale.

• Width (more neurons in a layer) gives the model more capacity to specialise.
• Depth (more layers) lets the model learn hierarchical features.
• Gradient descent finds the weights and biases.
• The loss function tells it how well it is doing.

This is also the moment to pay off a term that has been hovering since the training-loop diagram in Part 5: backpropagation. With one neuron, working out each parameter's slope was a single formula. In a network, a hidden neuron affects the loss only through the neurons after it, so to know how much w1 mattered, you first have to know how much the output it fed into mattered. Backpropagation is simply the bookkeeping that works this out efficiently: start from the error at the output and pass responsibility backwards through the network, layer by layer, so every weight and bias learns its own share of the blame. That is exactly what the training demo above is doing on every step. The result is the same gradient descent you already know, just with the slopes computed in reverse order.

Everything you have learned in this series, weights, biases, loss functions, gradient descent, learning rate, generalisation, local minima, activation functions, neurons, is operating inside every model you use.

Next: From Straight Lines to ChatGPT: the series opened with "LLMs are just predicting the next token." Time to close the loop and see how everything you have learned scales up to a language model.