Activation Functions

The linear model y = wx + b is powerful, but it is fundamentally limited: it can only represent straight lines. Real-world data might curve, twist, or look like someone had too much coffee before plotting it.

Three ways to go non-linear

When a straight line won't fit, there are a few options.

Use a non-linear function as the model. If the data looks like a wave, fit a sine function; if it is S-shaped, fit a tanh. The equations are more complex, but gradient descent still works the same way, calculate gradients, apply the learning rate, adjust the weights.

Add higher-order terms. Instead of just x, include x², x³, and so on, each with its own weight (all sharing a single bias). This is the trick from Part 3: still regression, but it can fit far more interesting shapes than a straight line.

Both of these work, but each one commits you to a specific shape picked in advance. The third option is the trick that powers neural networks:

Activation functions keep the simple, trainable linear structure and layer the non-linearity on top. The idea:

1. Compute the linear output: z = wx + b
2. Pass it through an activation function: output = activation(z)

The activation function has no weights or biases of its own. It is just a fixed mathematical transformation. But it makes the output non-linear, and once you chain non-linear units together, you can model surprisingly complex patterns.

This little unit, a weight, a bias, and an activation function, has a famous name: it is a neuron, the basic building block of every neural network. The name is borrowed from biology, and the analogy is loose but helpful: like a brain cell, it receives a signal (x), decides how strongly to respond (w and b), and either fires or stays quiet (the activation function). Keep that picture in mind for the rest of this article; the next part is about what happens when neurons team up.

Neurons are so central to machine learning that they have a standard way of being drawn, and it is worth learning to read it now, because every neural network diagram you will ever see is built from copies of this picture:

ReLU (Rectified Linear Unit)

The simplest and most widely used activation function:

ReLU(z) = max(0, z)

It strips away the negative half of the line, leaving zero for any negative input and the value itself for positive inputs. The result looks like a hockey stick.

To see what this does inside a neuron, run some numbers through one. Take a neuron with w = 2 and b = -4, so its raw output is z = 2x - 4, and follow each input through both steps:

The left two columns are plain Part-2 material: a straight line. The third column is where the activation earns its keep. For every input up to 2 the neuron says exactly 0, total silence, and from 3 onwards it passes the line through untouched. The weight and bias chose where the neuron wakes up (at x = 2, where z crosses zero) and how steeply it responds after that; ReLU is what lets it sleep through everything else.

Where it's useful: Sensor values that can't go negative, image pixel intensities, anything where "nothing happened" is meaningfully different from "something happened."

Leaky ReLU

LeakyReLU(z) = z if z > 0, else α × z   (typically α ≈ 0.01)

Like ReLU but allows a small negative slope for negative inputs. Think of a diode: it blocks negative voltage, but in reality a tiny leakage current still flows.

It is useful when negative values still carry weak but useful information (a financial loss still tells you something about risk). It also fixes a real training hazard called the dying ReLU problem: if a plain ReLU neuron's input becomes negative for every data point, its output is always zero, so nudging its weight or bias no longer changes anything, the gradient is zero, and gradient descent can never wake it up again. The neuron is dead for the rest of training. The leak keeps a faint signal flowing, so the neuron can always recover.

Clipped ReLU

ClippedReLU(z) = max(0, min(cap, z))

Clips the output at a maximum value. Useful when the output has a physical ceiling: a sensor's maximum reading, a network link's maximum throughput, screen brightness between zero and full.

Sigmoid

sigmoid(z) = 1 / (1 + e^{-z})

(The e here is Euler's number, ≈ 2.718, a constant that appears throughout maths; all you need to know is that e^{-z} shrinks rapidly towards zero as z grows.)

Sigmoid squashes any input to a value between 0 and 1, with an S-shaped curve. It models phenomena that start slow, accelerate, then saturate: population growth, epidemic spread, probability outputs.

Tanh (Hyperbolic Tangent)

tanh(z) = (e^z - e^{-z}) / (e^z + e^{-z})

Like sigmoid but ranges from -1 to +1. Useful when you need a symmetric output centred at zero.

Interactive demo

Use the controls below to explore each activation function. The thin dark line is the raw linear output z = wx + b, exactly the straight line you know from Part 2. The thick dark-red line is what comes out after the activation function is applied. Adjust weight and bias to see how the activation shapes the output.

What to try

• Pick ReLU, set the weight to 1 and slide the bias around. The kink (where the neuron "switches on") slides left and right: the bias decides where the neuron starts responding.
• Now slide the weight. The active side gets steeper or shallower: the weight decides how strongly it responds. With a negative weight, the neuron responds to small inputs and ignores large ones, the hockey stick flips.
• Switch to sigmoid and push the weight up to 4: the gentle S sharpens towards a step. Weight controls how abrupt the transition is; bias slides the step along the x-axis.
• Compare ReLU and leaky ReLU with a negative weight and bias: watch the leaky version keep a faint slope where plain ReLU flatlines at zero. That faint slope is exactly what keeps a dying neuron trainable.

One neuron with an activation function can bend a line once, place the bend anywhere, and control the slope on each side. That may not sound like much, but it is the only ingredient we were missing.

Next: Neurons, Chaining, and Specialisation: what happens when neurons connect, in series and in parallel, and how a handful of them can learn to draw a sine wave.