Opening your learning space
Starting the app …
Opening your learning space
Starting the app …
Linear least squares chooses model parameters that minimize the sum of squared deviations from an overdetermined set of measurements.
Free to browse · flashcards or a multiple-choice quiz · import from Essential
Formula
A·x and b must have the same unit · residuals have that output unit
| Symbol | Meaning | Unit |
|---|---|---|
| Design matrix containing measurement conditions or model features | unit of target / unit of model variable | |
| Vector of observed measurements | unit of the objective value | |
| x̂ | Parameter estimate with the smallest squared residual norm | unit of the corresponding parameter component |
| r = b − Ax̂ | Residual vector: observation minus fitted value | unit of the objective value |
\widehat{\mathbf{x}} = \arg\min_{\mathbf{x}} \|A\mathbf{x}-\mathbf{b}\|_2^2 \quad A^T A\widehat{\mathbf{x}}=A^T\mathbf{b}When a model has more measurement equations than unknown parameters, A·x = b usually has no exact solution. We seek x̂ that minimizes ‖A·x − b‖22. At the minimum, the residual vector is perpendicular to every column of A: Aᵀ(Ax̂ − b) = 0. This gives the normal equations AᵀA x̂ = Aᵀb. If A has full column rank, AᵀA is invertible and x̂ = (AᵀA)−1Aᵀb. This is the closed-form teaching result; for numerical work, QR or SVD methods are often more stable than explicitly forming AᵀA.
The minimization problem is defined for A·x ≈ b. The unique formula (AᵀA)−1Aᵀb requires full column rank. With deficient rank, there are generally multiple minimizing parameter vectors; QR or SVD also handle numerically difficult cases better.
At the minimum, the residual is perpendicular to the column space of A.
Normal equations
They also hold when the solution is not unique; the explicit inverse form requires full column rank.
Closed-form solution with full column rank
A textbook form, not a recommendation to compute the inverse explicitly.
Fit y=a+mt to (0,1), (1,2), (2,2) by least squares.
A=[[1,0],[1,1],[1,2]], b=[1,2,2].
The normal equations are [[3,3],[3,5]][a,m]ᵀ=[5,6]ᵀ, giving a=7/6 and m=1/2.
Thus ŷ=7/6+t/2.
Using the inverse formula when A is rank deficient or explicitly computing a numerical inverse.
Check the rank first. The normal equations express the minimum; compute stably with QR/SVD and check units and residuals.
Formula cluster
The matrix form generalizes linear regression to multiple features and measurement equations.
Fit y = a + m·t to the measurements (t,y) = (0,1), (1,2), (2,2). With A = [[1,0],[1,1],[1,2]] and b = [1,2,2], the normal equations give x̂ = [a,m]ᵀ = [7/6, 1/2]ᵀ. The fitted line is y = 7/6 + t/2; the residuals are −1/6, 1/3 and −1/6.
Fit measurement data to linear models, calibration, parameter estimation, trend analysis and numerical modelling
Here is how to work through a typical Linear Least Squares (x̂ = arg min ‖Ax − b‖² · AᵀAx̂ = Aᵀb) task step by step:
Task
Fit y=a+mt to (0,1), (1,2), (2,2) by least squares.
Solution path
A=[[1,0],[1,1],[1,2]], b=[1,2,2].
The normal equations are [[3,3],[3,5]][a,m]ᵀ=[5,6]ᵀ, giving a=7/6 and m=1/2.
Thus ŷ=7/6+t/2.
Use it when several measurements should be described by a model, but measurement noise means no parameters satisfy every equation exactly. The method chooses parameters that minimize the sum of squared residuals. A typical case is fitting a line through measurements: the points scatter, and the fitted line summarizes their trend. If measurements have very different uncertainties, ordinary least squares is not automatically appropriate; weighted residuals may be needed. Outliers also have a strong effect because residuals are squared. Check the model assumptions and data before interpreting the result as a reliable prediction.
The error measure is ‖Ax−b‖2, and we minimize its squared length. At the minimum, the change in this quantity in every parameter direction is zero; its gradient is 2Aᵀ(Ax−b). Setting it to zero gives AᵀAx=Aᵀb. Geometrically, the observation-minus-model vector b−Ax is perpendicular to every column of A, and therefore to the model space. This also explains why fitting is a projection of b onto that space. The normal equations are not a trick to memorize; they express the condition for minimizing the squared-error function.
The explicit formula assumes that A has full column rank. Then AᵀA is invertible and the minimization has exactly one solution. If columns are dependent, AᵀA can be singular; the best approximation Ax may still be unique even when the parameter vector is not. Check rank and model identifiability. For numerical calculations, also avoid simply computing a matrix inverse: QR decomposition or singular value decomposition is usually more stable. The formula is useful for understanding the solution mathematically, but it is not always the best computational route.
No. Forming AᵀA squares a matrix’s condition number and can amplify rounding errors, especially when columns of A are nearly dependent. QR decomposition usually solves the least-squares problem more stably, while SVD makes rank deficiency especially visible and supports a controlled pseudoinverse solution. In a foundational exercise, the normal equations remain valuable: they show the orthogonality condition and lead to a closed form. Keep theory separate from numerical implementation. For real data, report how the problem was solved and whether small data changes cause large parameter changes.
A residual is the observed value minus the model value; together, residuals show where and how the model misses. With an intercept, they sum to zero in unweighted linear least squares, but that does not prove the model is appropriate. Plot them against time, measurement or prediction: random scatter around zero is more consistent with a linear model, while a curve or funnel can indicate missing curvature or unequal variance. For the example ŷ=7/6+t/2, residuals are −1/6, 1/3, −1/6; they sum to zero but remain visible. Always inspect patterns, units and outliers.
Quanta exam set
Curated exam set for "Linear Least Squares":
Question (front)
Which formula describes Linear Least Squares?
Answer in your set
Question (front)
How do you rearrange Linear Least Squares for Normal equations?
Answer in your set
Question (front)
Which common mistake happens with Linear Least Squares?
Answer in your set
+ 7 more cards: units, variables, derivation, example, exam task
The formula set contains 10 cards. After importing, review them and start practicing; your ratings determine later review intervals.
This list names subject references and further reading. A page or section number appears when it is part of the reference; these are not verbatim quotations.
Create a curated FSRS exam set for x̂ = arg min ‖Ax − b‖² · AᵀAx̂ = Aᵀb: formula recall, variables, derivation, rearrangement, worked example, common mistakes and exam context.
Curated formula set · LaTeX · FSRS spaced repetition · import from Essential