Loss Functions

In the previous part we kept using the word error: how far a particular weight and bias land from a good fit. To do anything useful with that idea, we first need to turn it into a number we can actually compute. That number is called the loss (or cost), and the formula that produces it is the loss function.

From predictions to a single number

Let us make this concrete with the same study-hours data from Part 2. We are handed a set of data points, each one a feature (hours studied) paired with its target (the exam score that student actually got):

The trick a loss function uses is to pretend we do not know the targets. Picture covering up the score column, so all we are left with is the features and a set of blanks to fill in:

Now we try to recreate those hidden scores ourselves. We have no idea yet what the right weight and bias are, so we just start with a more or less random guess, say weight = 4 and bias = 45, and run every feature through the line y = 4 × hours + 45 to predict a score. Then we uncover the real targets and set them beside our predictions to see how far off each one is:

Even by eye the fit is poor: the line runs below almost every point, and the gap grows the longer the study time. But "clearly off" is not something an algorithm can act on. To improve the line we first have to measure exactly how wrong it is. The value in that last column, error = prediction - actual, is the prediction error for a single data point.

That leaves us with one error per data point. Useful, but what we really want is a single number that sums up how the model is doing across the whole dataset. That is exactly what a loss function provides.

Mean Squared Error (MSE)

MSE = (1/n) × Σ (y_pred - y_actual)²

Square each error, then average. Squaring has two effects: it makes all errors positive (so negative and positive errors don't cancel out), and it penalises large errors more heavily: a prediction that is off by 10 contributes 100 to the sum, not 10.

Mean Absolute Error (MAE)

MAE = (1/n) × Σ |y_pred - y_actual|

Take the absolute value of each error, then average. Every error contributes proportionally to its size, no squaring penalty for outliers.

Trying different weights

To see the loss earn its keep, watch what happens to it as we change the weight. For this we will switch to a clean dataset whose perfect line is exactly y = 2x + 1, and to keep things simple we will hold the bias fixed at 1 and only move the weight. (A real model tunes the bias as well, but freezing it lets us watch one number change at a time.)

Start with a weight that is clearly too small, weight = 1. We run each feature through y = 1·x + 1, compare with the actual target, and note what each row contributes to MAE (the size of the error) and to MSE (the error squared):

Averaging the last two columns gives the two loss scores for this weight:

MAE = (1 + 2 + 3 + 4) / 4 = 2.5
MSE = (1 + 4 + 9 + 16) / 4 = 7.5

Both are large, because the line is far too shallow.

Now nudge the weight up to weight = 1.5 and redo the same table:

MAE = (0.5 + 1 + 1.5 + 2) / 4 = 1.25
MSE = (0.25 + 1 + 2.25 + 4) / 4 = 1.875

Both scores dropped (MAE from 2.5 to 1.25, MSE from 7.5 to 1.875), so weight 1.5 fits better than weight 1.

If we keep repeating this for a whole range of weights, a clear pattern emerges:

The loss falls as the weight climbs toward 2, touches zero exactly at the ideal weight (where the line matches the data perfectly), then climbs again as we overshoot. The closer the weight gets to its ideal value, the smaller the error. Finding that ideal weight is the whole game.

Try it yourself

Drag the slider to change the weight, with the bias fixed at 1. The line tilts, the gap at each point grows or shrinks, and the MAE and MSE at the bottom of the graph update as you go. See if you can settle on the weight that drives both losses to zero.

Loss curve

A single loss score only judges one guess. A loss curve tracks the loss as the weight changes: it sweeps through a whole range of weights and records how large the loss is at each one, turning a pile of separate scores into a single shape we can read at a glance.

That shape is what points us to the answer. Where the curve runs high, the weight there is a poor choice and the loss is large; where it dips, the weight fits better. The single lowest point of the curve marks the weight that gives the smallest loss of all, and that is almost certainly the best value to choose for our line. Finding the bottom of this curve is what the rest of training comes down to.

Plotting the MAE and MSE columns against the weight turns that table into a pair of curves:

Why the shape of the loss curve matters

The two curves above are not just different sizes, they are different shapes, and that shape turns out to be critical for gradient descent.

MSE gives a smooth U-shaped (parabolic) curve. Because errors are squared, the loss increases smoothly and symmetrically away from the minimum. Crucially, the curve has a well-defined gradient (slope) everywhere, steeper far from the minimum, nearly flat at the bottom. This gradient is exactly what gradient descent uses to know which direction to move and how far.

MAE gives a V-shaped curve. The slopes on either side are constant. There is no "get steeper as you move further away" behaviour. The gradient gives less information about how far you are from the minimum.

This is why MSE is the more commonly used loss function for regression: its smooth parabolic shape gives gradient descent a clear signal to follow all the way to the bottom.

Interactive demo: the loss curve

The animation below shows what happens as the weight changes from 0 to 4, with bias fixed at 1 (true line: y = 2x + 1). Watch how the predicted line (red) moves relative to the data, and how the MSE and MAE loss curves build up. The ideal weight is 2, that is the bottom of both curves.

Press Start to run the animation. Press Pause to freeze it at any point.

Key observations

• The minimum of both curves occurs at weight = 2, the true weight of the data-generating process (y = 2x + 1).
• The MSE curve is a smooth parabola. Far from the minimum the slope is steep; close to the minimum it flattens. This gives gradient descent a signal that scales with distance.
• The MAE curve has constant slopes on each side, and at the very bottom the two sides meet in a sharp corner. Right at that corner the slope jumps from one value to the other with nothing in between, so there is no gentle "you are nearly there" signal, exactly where precision matters most.
• Squaring errors in MSE means large outliers get penalised heavily. This is both a feature (outliers matter a lot) and a bug (outliers can dominate the training). The demo below shows this trade-off in action.

Outliers: where MSE and MAE disagree

Real datasets contain the occasional rogue point: a sensor glitch, a typo, the student who partied the night before the exam. Because MSE squares every error, one bad point can contribute more to the loss than all the good points combined, and the best-fit weight gets dragged towards it. MAE treats every unit of error equally, so a single outlier barely moves it.

Tick the box below to add one outlier, a student who studied 4 hours but scored as if they had studied none, and watch what happens to the two loss curves. The marker on each curve shows the weight at its lowest point.

Without the outlier, both losses agree: the best weight is 2. Add the outlier and the MSE minimum lurches away from 2, because that one squared error of around 49 outweighs everything else, while the MAE minimum barely moves. Neither behaviour is "right": if your outliers are genuine, rare-but-real events, you may want the model to respect them; if they are glitches, MAE's stubbornness is a virtue. Knowing which loss you are optimising, and what it does with your worst data points, is part of understanding what your model will learn.

Next: Gradient Descent: how to read the slope of this loss curve and use it to step the weight and bias toward the bottom, one iteration at a time.