Solve It or Search for It

The last part ended on the million dollar question: given a pile of data points, how do we find the weight and bias that make the best predictions?

Before meeting any answers, it is worth feeling the problem in your own hands. Below is a fresh dataset, and your job is to fit it: drag the two sliders to set a weight and a bias, and try to get the line as close to the points as you can. The dashed gaps show how far the line misses each point, and the readout keeps a running score: the average miss across all twelve points.

Two things become obvious after a minute of sliding. First, no straight line passes through all the points. Real data is noisy, measurements wobble, people are inconsistent, sensors drift, so the points scatter around the underlying trend rather than sitting on it. However you set the knobs, some dashed gaps remain. Second, some lines are clearly better than others: the misses shrink as you tune the knobs, and there is one particular weight and bias where the average miss is as small as it can get. That is the goal: not a line that touches every point (no such line exists), but the line whose overall miss is smallest.

There are two broad families of answer. We can solve for the best weight and bias directly with a formula, or we can search for them step by step. Both are worth understanding, because the reason the second one wins is the whole reason this series exists.

Both approaches rest on a single idea we should name before going further: the error. You have already met it: it is the "average miss" you were shrinking in the demo above, a single number measuring how wrong a particular guess at the weight and bias is, in other words, how far the line built from that guess falls from the data points. A good guess has a small error; a poor one has a large error. Whether we solve or search, the goal is identical: find the weight and bias with the smallest error. We will pin "error" down precisely in the next part on loss functions, where it turns out there is more than one reasonable way to keep score; the rough idea is enough for now.

And now the demo's small secret can be told: for that dataset, no straight line, however you set the sliders, can get the error below about 0.33. That number is this problem's floor, the smallest error any line can achieve, and the weight and bias that reach it are the answer to the million dollar question. Scroll back up and hunt for it; the readout turns green when you get there.

Closed-form solutions: solve it in one shot

A closed-form solution is a formula you feed your data into to get the answer straight back, no guessing, no repetition. For fitting a straight line, the best known is the Normal Equation (also called ordinary least squares):

w = (X^T X)^-1 X^T y

Do not worry if the notation looks alien. X is a table of your inputs (one row per data point, plus a column of 1s so the bias is included), and y is the column of targets. X^T means the table flipped on its diagonal, and ^-1 is a kind of "division" for tables of numbers. The important thing is not the symbols, it is that this is one calculation: data goes in, the best weight and bias come out. (You may also run into fancier-sounding routes to the same answer, names like QR decomposition, SVD, or ridge regression. They are refinements for numerical accuracy and stability, not different ideas. File the names away and move on.)

For a single feature, the whole thing collapses to two formulas you can do by hand:

w = Σ (xi - mean(x)) (yi - mean(y))  /  Σ (xi - mean(x))²
b = mean(y) - w × mean(x)

Worked on our study-hours data. Taking the six points from the table in the last part:

Σx = 21      Σy = 404      Σx² = 91      Σxy = 1510      (n = 6)

X^T X = [  6   21 ]       X^T y = [  404 ]
        [ 21   91 ]               [ 1510 ]

det(X^T X) = 6·91 - 21·21 = 105

b = ( 91·404 - 21·1510) / 105 = 5054 / 105 ≈ 48.13
w = (-21·404 +  6·1510) / 105 =  576 / 105 ≈  5.49

So the formula hands us w ≈ 5.49 and b ≈ 48.13, the line y ≈ 5.49x + 48.1, which is exactly the best-fit line drawn through the study-hours scatter in the last part. One calculation, no trial and error.

And it works just as well on the dataset you fitted by hand at the top of this page: feed those twelve points into the formula and it immediately returns w ≈ 0.60 and b ≈ 0.94, the very line you were hunting for with the sliders, found without a single try. That is the appeal of a closed-form solution: your minute of slider-fiddling, replaced by one calculation.

So why not always do this? Three reasons.

Reason 1: it does not scale

The expensive step is inverting X^T X, a matrix whose size is set by the number of features (weights), and building it means passing over all your data. The inversion cost grows with the cube of the feature count: double the features and you roughly octuple the work, and the matrix has to fit in memory. For a single straight line that is nothing. But modern models have thousands, millions, even billions of weights, and inverting a million-by-million matrix is hopeless.

Reason 2: curves make it messy

Real data often follows a curve rather than a line. To bend the model, you add higher powers of the input, each with its own weight:

straight line    y = w·x + b
parabola         y = w2·x² + w1·x + b
cubic            y = w3·x³ + w2·x² + w1·x + b

Here is the part that catches people out: a parabola is plainly not a straight line, so how can a linear method fit one? It comes down to what the word linear is describing. The Normal Equation does not care whether the curve bends; it only cares how the weights combine, and they combine in the simplest way possible, each one multiplied by a number and the results added together. Look again at y = w2·x² + w1·x + b: the weights w2, w1, and b are never squared, never multiplied by each other, never passed through a function. They are only scaled and summed. That is what "linear in the weights" means, and it is all the formula needs.

The way to picture it is to treat x² as a brand-new feature. You already know x for every data point, so you can work out x² for each one too and hand it to the formula as just another column of inputs, sitting beside x. The formula never realises that column is secretly x squared; it just fits a flat relationship across however many input columns you give it (x, x², x³, and so on). So the same one-shot calculation keeps working: you add a column for x², then x³, as far as you want to go.

Try it: bend the line with higher-order terms

The data points below follow an S-shaped curve, and no straight line can fit them. Start with just w1 and b (a straight line) and see how low you can push the error. Then bring in w2 and w3 to bend the curve. Watch the MSE readout: it measures how far the curve is from the points, and it is the "error" we have been talking about.

With only w1 and b, the best you can do is a compromise line through the middle, and the error stays high. Turn w3 up towards 1 and w1 down towards -2 and the curve snaps onto the points. Same fitting machinery, one extra column of inputs.

The catch is that it stops being free. The matrix grows with every term you add, it becomes increasingly delicate to invert, and you are now also guessing the right degree of curve. The neat one-shot formula starts to creak.

Reason 3: sometimes there is no formula at all

The closed-form trick rests on a single move: take the statement "these are the weights that make the error as small as it can possibly be," turn it into an equation, and solve that equation outright for the weights. That "smallest error" is just the bottom of the error curve, the point where nudging the weight or bias either way only makes the fit worse. For a straight line the resulting equation is simple enough to solve in one go, which is exactly why the Normal Equation works. But that move is not always available. Push to a genuinely non-linear model, and the equation you would need to solve can simply have no formula-style answer at all.

For the curious: this is a proven dead end, not a "we haven't found it yet." The Abel–Ruffini theorem shows there is no general formula (using only +, -, ×, ÷ and roots) for polynomial equations of degree five or higher. When your best-fit problem reduces to one of those, the closed-form door is mathematically shut. You do not need this theorem for anything else in the series, it is just a satisfying reason why the search-based approach below is not a workaround but a necessity.

Gradient descent: search for it step by step

Gradient descent gives up on solving the problem in one shot. Instead it starts with a guess for weight and bias, measures how wrong that guess is, and nudges the values a little in the direction that lowers the error, then repeats, again and again, until the result is good enough.

Notice the bar: good enough, not perfect. That sounds like a downgrade, but it is exactly what makes the method so powerful. It never inverts a giant matrix, so it scales to billions of weights. All it really needs is the slope of the error: a reading of which way is "downhill," meaning which small change to the weight and bias would make the error smaller. Keep stepping in that downhill direction and the error keeps shrinking. That slope can be worked out for almost any model (linear, polynomial, logistic, or a deep neural network), so gradient descent keeps working long after every closed-form formula has given up. Precisely how the slope is found, and how big each step should be, is what the next few parts of this series are about. And a fit that is merely good enough often generalises better to new data than one tuned to perfection (more on that in Parts 7 and 8). Just like in life, once you settle for good enough, it opens up a whole world of possibilities.

The rest of this series is about gradient descent.

Next: Loss Functions: how to turn "how wrong is this guess?" into a single number, the error we have been leaning on, and why the shape of its curve matters so much to gradient descent.