Local vs Global Minima

So far we have been looking at a loss curve that is a clean U-shape, one minimum, no confusion about which direction to go. That is the ideal case for a simple linear model with a single weight.

Real models have many more parameters. And when you have multiple parameters, the loss is not a 2D curve. It is a high-dimensional surface.

From a curve to a surface

With one weight (w) and one bias (b), the loss is a 3D surface: a bowl. The x and y axes are w and b, and the height (z axis) is the MSE loss. Gradient descent now descends this bowl in two directions simultaneously, adjusting both w and b at each step.

For a simple linear model this bowl is smooth and has a single lowest point. But as models become more complex (more features, higher-order terms, layers of neurons), the surface develops multiple hills and valleys.

Local minima and global minima

Imagine a rough terrain with many valleys. Gradient descent is like a ball rolling downhill from wherever it is placed:

• It will always find the nearest valley bottom, the local minimum.
• It may not find the lowest point on the entire surface, the global minimum.

The starting position (the initial random weight and bias values) determines which valley the ball rolls into. Different random initialisations can lead to different results.

Is a local minimum bad?

Counterintuitively, the answer is often no.

Researchers studying deep neural networks found something surprising: the global minimum is frequently worse than a good local minimum in practice. A paper by Choromanska et al. (The Loss Surfaces of Multilayer Networks) put it this way:

"...we prove that recovering the global minimum becomes harder as the network size increases and that it is in practice irrelevant as global minimum often leads to overfitting."

This should ring a bell from the previous part: overfitting means the model fits the training data too perfectly, memorising it rather than learning general patterns. The global minimum is, by definition, the set of parameter values most precisely matched to the training set, noise, outliers, and all, so it is exactly where memorisation is at its worst.

A local minimum, by contrast, tends to capture the broader trends in the data without over-specialising. It generalises better to unseen inputs.

Interactive demo: loss surface and initialisation

The visualisation below shows a 2D loss surface (weight × bias). The white dot is where gradient descent starts (based on your initial weight). Click Run from here to watch it descend to the nearest valley. Try different starting points to see how the end result changes.

What to explore

Try clicking in different parts of the loss surface (the heatmap) and running the descent:

• Starting in the upper right (high w, high b) should roll into one valley.
• Starting in the lower left might roll into a different valley.
• Both valleys are "local minima", and neither may be the absolute lowest point on the surface.

The colour scale goes from dark blue (low loss) to red (high loss). The gradient descent path in white heads towards wherever the landscape slopes downward from the start.

The key takeaway

Because gradient descent follows the local slope, the result depends on where you start. For complex models this is unavoidable. The practical response is:

• Random restarts: train several times with different random initialisations and pick the best result.
• Accept local minima: for large neural networks, local minima are often good enough and generalise better than the global minimum.
• Batch training tricks (mini-batch SGD): randomly sampling batches of data introduces noise into the gradient, which helps the optimiser escape shallow local minima.

Beyond one feature: multivariate regression

One last piece of context before we leave plain linear regression behind. Throughout this series the target has depended on a single feature, one input x with one weight. Most real problems are not so tidy. Predicting a house price might draw on floor area, number of bedrooms, age, and location all at once. This is multivariate regression: several features, each with its own weight.

The model simply grows more terms, one weight per feature, plus the single shared bias:

y = w1·x1 + w2·x2 + w3·x3 + ... + b

For instance, to predict a house price from three features, its floor area, number of bedrooms, and age, the data might look like this:

Training finds one weight for each feature, plus a bias. For this data a fitted model comes out as:

price = 3 × area + 25 × bedrooms - 2 × age + 50

Each weight is a price sensitivity you can read straight off: about £3k for every extra square metre, £25k per bedroom, and £2k off for every year of age, on top of a £50k base. Feed a new house's three numbers into that equation and out comes its predicted price.

Nothing else about the recipe changes. You still make a prediction, measure the loss, and take a partial derivative with respect to every weight (and the bias) to see which way each one should move, then step them all downhill together. The only thing that grows is the number of dimensions: the loss surface now lives in a space with one axis per weight, far too many to draw. But gradient descent never needed the picture. It only ever needed the slope in each direction, and that it can always compute.

That is the engine behind everything from linear regression to the largest neural networks: measure how wrong you are, then roll downhill. What changes from here is not the engine but the shape of the model it drives, and that is where the next part comes in.

Next: Activation Functions: the trick that lets a model bend beyond straight lines to fit curves and other non-linear shapes.