Mathematics · Stochastics / Statistics

Linear Regression (Line of Best Fit)

Linear regression places the line of best fit $\hat y=a+bx$ through a scatter plot so that the sum of the squared deviations becomes minimal.

AdvancedExam-relevantUpper secondaryUniversity/collegeBachelor foundations

Create flashcards or a quiz I already have an account

Free to browse · flashcards or a multiple-choice quiz · import from Essential

Formula

b = \frac{\sum ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{\sum ( x _{i} - x ˉ ) ^{2}}, a = \overset{y}{ˉ} - b \overset{x}{ˉ}

b in y-unit per x-unit · a in y-unit · r dimensionless (−1 to 1)

Variables & units

Variables & units
Symbol	Meaning	Unit
$b$	Slope of the regression line	y unit/x unit
$a$	y-intercept of the regression line	y unit
x̄, ȳ	Means of the x and y data	same as the data
$r$	Correlation coefficient (strength of the linear relationship)	dimensionless

LaTeX

b = \frac{\sum (x_{i}-\bar{x})(y_{i}-\bar{y})}{\sum (x_{i}-\bar{x})^{2}}, \quad a = \bar{y} - b\bar{x}

QUANTA STUDYquanta-study.de

Seven data points with a highlighted regression line; dashed vertical lines show the deviations, and a ring marks the mean point (x̄ | ȳ) on the line. — The regression line ŷ = a + b·x minimises the sum of squared vertical deviations and passes through the mean point of the data (ring), whose coordinates are the means of x and y.

Derivation & background

Method of least squares, published by Legendre in 1805 and used by Gauss to determine the orbit of the dwarf planet Ceres. The regression line always passes through the centroid (x̄|ȳ) of the data. The correlation coefficient r measures only the strength of the linear relationship: r near ±1 means tight coupling, r near 0 no linear relationship. Correlation does not prove causation, and predictions far outside the data are unreliable.

Exam blueprint

Validity range

Meaningful only if the relationship is approximately linear (check the scatter plot); outliers distort the line strongly, and predictions are valid only within the data range.

Derivation steps

Minimize the sum of the squared vertical deviations from the line.

1For S(a,b) = Σ(y_i − a − bx_i)² the partial derivatives with respect to a and b are set to 0.
2The normal equations yield b = Σ(x_i−x̄)(y_i−ȳ)/Σ(x_i−x̄)² and a = ȳ $- b\cdot{}x$ ̄.

Rearrangements

Intercept

a = \bar{y} - b \cdot \bar{x}

Follows from the line passing through (x̄|ȳ).

Correlation coefficient

r = \frac{\sum (x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum (x_{i}-\bar{x})^{2} \cdot \sum (y_{i}-\bar{y})^{2}}}

Measures the strength of the linear relationship (−1 to 1).

Slope via r

b = r \cdot \frac{s_{y}}{s_{x}}

Connection between correlation and standard deviations.

Task variant

Determine the regression line for (0|1), (1|3), (2|5).

x̄ = 1, ȳ = 3.
Numerator: $(-1)(-2) + 0 + (1)(2) = 4$ , denominator: $1 + 0 + 1 = 2$ .
$b = 2$ , $a = 3 - 2\cdot{}1 = 1$ : ŷ = 2x + 1.
All points lie exactly on it $(r = 1)$ .

For (1|2), (2|3), (3|5), (4|6) we have ŷ = 1.4x + 0.5. Predict y for $x = 5$ and judge the fit.

ŷ(5) = 1.4·5 + 0.5 = 7.5.
Fit: $r =\frac{ 7}{\sqrt{(5\cdot{}10)}} =\frac{ 7}{7.07} \approx{} 0.99$ , a very tight linear relationship; the prediction is only slightly outside the data and acceptable.

Common mistakes

Interpreting correlation as causation.

r only measures co-movement; statistics does not say whether x causes y.

Extrapolating far outside the data range.

The linear trend holds only in the observed range; outside it the model can break.

Swapping the roles of x and y.

Regressing y on x minimizes vertical y-deviations; swapped roles give a different line.

Exam context

Statistics tasks with data tables and graphing calculators, trend description and prediction with assessment of model limits.

Formula cluster

Descriptive statistics

Connects means, spread and linear functions into a data model.

σ = √Var(X)

Standard Deviation

E(X) = Σ x_i·p_i

Expected Value

m = Δy/Δx, y = mx + b

Line Equation and Slope

Worked example

Points (1|2), (2|3), (3|5), (4|6): $\bar x=2.5$ and $\bar y=4$ . $b =\frac{ 7}{5} = 1.4$ and $a = 4 - 1.4\cdot{}2.5 = 0.5$ , so $\hat y=1.4x+0.5$ . Prediction for $x = 5$ : $\hat y=7.5$ .

Applications

Trend forecasts from measurement series (climate and sales data), statistics tasks with graphing calculators, calibration lines in the sciences, social research

How do you calculate with Linear Regression (Line of Best Fit)?

Here is how to work through a typical Linear Regression (Line of Best Fit) (ŷ = a + b·x) task step by step:

1
Task
Determine the regression line for (0|1), (1|3), (2|5).
Solution path
1. x̄ = 1, ȳ = 3.
2. Numerator: $(-1)(-2) + 0 + (1)(2) = 4$ , denominator: $1 + 0 + 1 = 2$ .
3. $b = 2$ , $a = 3 - 2\cdot{}1 = 1$ : ŷ = 2x + 1.
4. All points lie exactly on it $(r = 1)$ .
2
Task
For (1|2), (2|3), (3|5), (4|6) we have ŷ = 1.4x + 0.5. Predict y for $x = 5$ and judge the fit.
Solution path
1. ŷ(5) = 1.4·5 + 0.5 = 7.5.
2. Fit: $r =\frac{ 7}{\sqrt{(5\cdot{}10)}} =\frac{ 7}{7.07} \approx{} 0.99$ , a very tight linear relationship; the prediction is only slightly outside the data and acceptable.

Frequently asked questions about Linear Regression (Line of Best Fit)

How do you compute a regression line by hand?+

In four steps. First determine the means x̄ and ȳ. Second compute for each data point the deviation products (x_i − x̄)(y_i − ȳ) and the squares (x_i − x̄)². Third b = sum of products / sum of squares. Fourth a = ȳ $- b\cdot{}x$ ̄. Example with (0|1), (1|3), (2|5): x̄ = 1, ȳ = 3; products: $(-1)(-2) + 0 + (1)(2) = 4$ ; squares: $1 + 0 + 1 = 2$ ; so $b = 2$ , $a = 3 - 2 = 1$ and ŷ = 2x + 1. Check: the line must pass through (x̄|ȳ), here $2\cdot{}1 + 1 = 3$ ✓. A table with columns for x_i, y_i, deviations and products keeps the calculation tidy.

What does the correlation coefficient r tell you?+

r measures strength and direction of the linear relationship between two quantities and always lies between −1 and +1. $r = +1$ means all points lie exactly on a rising line; $r = -1$ exactly on a falling one. Values near 0 mean no linear relationship. Rough school reading: |r| from about 0.8 strong, around 0.5 moderate, below 0.3 weak. Two warnings: first, r measures only LINEAR coupling; a perfect parabola can yield $r \approx{} 0$ although a clear relationship exists (look at the scatter plot!). Second, even $r = 0.99$ says nothing about cause and effect. Example: for (1|2), (2|3), (3|5), (4|6), $r =\frac{ 7}{\sqrt{50}} \approx{} 0.99$ , an almost perfect linear trend.

Why are the deviations squared in regression?+

For three reasons. First the sign problem: positive and negative deviations would cancel each other when simply summed; a line could look perfect "on average" and still be far from all points. Squares are always positive. Second the weighting: squaring penalizes large outliers disproportionately, pulling the line towards points that would otherwise be missed badly. Third the mathematics: the sum of squares is differentiable, and setting the derivatives to zero yields unique closed formulas for a and b; with absolute values instead of squares there would be no such smooth solution. The price: single extreme outliers can tilt the line noticeably, so inspect the scatter plot first.

May you make predictions with the regression line?+

Within the data range (interpolation) yes, with judgement: for x-values between the observed data, ŷ = a + bx gives usable estimates when |r| is high. Example: ŷ = 1.4x + 0.5 from data with x from 1 to 4 may safely be evaluated at $x = 3.5$ . Extrapolation far beyond is critical: the linear trend is only supported within the observed range; outside it the relationship may flatten, tip over or break entirely. A classic: children's growth data extended linearly to age 30 yields absurd body heights. Exams expect exactly this assessment: compute the prediction AND judge its reliability based on data range and r.

Does a high correlation mean that x causes y?+

No, correlation is not causation. r only measures that two quantities vary together, not why. Often a third quantity is behind it (confounder): ice cream sales and sunburn cases correlate strongly, but the cause of both is sunny weather. The direction can also be unclear (does x influence y or vice versa?), and in small data sets high correlations even arise by chance. The sound approach: the regression describes the relationship and allows predictions; causal claims additionally require an experiment with controlled conditions or at least a plausible mechanism. In exam answers the sentence "a high r-value does not prove a causal relationship" is almost always a required assessment element.

Quanta exam set

Curated exam set for "Linear Regression (Line of Best Fit)":

Question (front)

Which formula describes Linear Regression (Line of Best Fit)?

Answer in your set

Question (front)

How do you rearrange Linear Regression (Line of Best Fit) for Intercept?

Answer in your set

Question (front)

Which common mistake happens with Linear Regression (Line of Best Fit)?

Answer in your set

+ 7 more cards: units, variables, derivation, example, exam task

The formula set contains 10 cards. After importing, review them and start practicing; your ratings determine later review intervals.

Create flashcards or a quiz I already have an account

Subject references and further reading

This list names subject references and further reading. A page or section number appears when it is part of the reference; these are not verbatim quotations.

[1]OpenStax, Principles of Data Science, §4.3: Correlation and Linear Regression Analysis

More formulas: Probability and statistics

P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}

Bayes' Theorem

f(x) = \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}

Normal Distribution (Gaussian Bell Curve)

P(X = k) = \binom{n}{k} \cdot p^{k} \cdot (1-p)^{n-k}

Binomial Distribution

P(A \mid B) = \frac{P(A \cap B)}{P(B)} \quad (P(B) > 0)

Conditional Probability

Spaced-repetition apps

Compare scheduling workflows

Study evidence

Read methods, results and limits

Compare flashcard apps

FSRS support and study features

Create a formula exam set

From Essential in Quanta

Retain Linear Regression (Line of Best Fit) for exams

Create a curated FSRS exam set for ŷ = a + b·x: formula recall, variables, derivation, rearrangement, worked example, common mistakes and exam context.

Create flashcards or a quiz Already registered? Continue directly

Curated formula set · LaTeX · FSRS spaced repetition · import from Essential

Quanta Study editorial team

Linear Regression (Line of Best Fit)

Linear regression places the line of best fit

\hat y=a+bx

through a scatter plot so that the sum of the squared deviations becomes minimal.

AdvancedExam-relevantUpper secondaryUniversity/collegeBachelor foundations

Free to browse · flashcards or a multiple-choice quiz · import from Essential

Symbol

Meaning

Unit

b

Slope of the regression line

y unit/x unit

a

y-intercept of the regression line

y unit

x̄, ȳ

Means of the x and y data

same as the data

r

Correlation coefficient (strength of the linear relationship)

dimensionless

Derivation & background

Exam blueprint

Validity range

Meaningful only if the relationship is approximately linear (check the scatter plot); outliers distort the line strongly, and predictions are valid only within the data range.

Derivation steps

Minimize the sum of the squared vertical deviations from the line.

1For S(a,b) = Σ(y_i − a − bx_i)² the partial derivatives with respect to a and b are set to 0.
2The normal equations yield b = Σ(x_i−x̄)(y_i−ȳ)/Σ(x_i−x̄)² and a = ȳ $- b\cdot{}x$ ̄.

How do you calculate with Linear Regression (Line of Best Fit)?

Here is how to work through a typical Linear Regression (Line of Best Fit) (ŷ = a + b·x) task step by step:

Task

Determine the regression line for (0|1), (1|3), (2|5).

Solution path

x̄ = 1, ȳ = 3.
Numerator: $(-1)(-2) + 0 + (1)(2) = 4$ , denominator: $1 + 0 + 1 = 2$ .
$b = 2$ , $a = 3 - 2\cdot{}1 = 1$ : ŷ = 2x + 1.
All points lie exactly on it $(r = 1)$ .

Task

For (1|2), (2|3), (3|5), (4|6) we have ŷ = 1.4x + 0.5. Predict y for $x = 5$ and judge the fit.

Solution path

ŷ(5) = 1.4·5 + 0.5 = 7.5.
Fit: $r =\frac{ 7}{\sqrt{(5\cdot{}10)}} =\frac{ 7}{7.07} \approx{} 0.99$ , a very tight linear relationship; the prediction is only slightly outside the data and acceptable.

Frequently asked questions about Linear Regression (Line of Best Fit)

How do you compute a regression line by hand?+

What does the correlation coefficient r tell you?+

Why are the deviations squared in regression?+

May you make predictions with the regression line?+

Does a high correlation mean that x causes y?+