Mathematics for Machine Learning
Deisenroth, Faisal & Ong · Freely readable textbook
Read §§10.2–10.3, printed pp.320–332, for PCA objectives; §4.5 for the SVD route.
Open the sourcePCA and the geometry of compression
The largest pattern is not necessarily the most meaningful one.
Prerequisites: 10 · Inner products and projection, 11 · Eigenvectors and the sign of a geometry, 25 · Hidden states and observable shadows
Understand
Objectives
- Compare reconstruction and classification on paired points.
- Calculate a retained-variance fraction with centered, unscaled coordinates.
- Exhibit a collision between different labels.
Start with something you can see
In the PCA lab choose Keep most variance. Predict where each circle and its paired square land. Switch to Keep label information and compare reconstruction with label separation. A closest-rank-one approximation answers a reconstruction question, not a classifier’s question.
Give the idea a precise name
Center the data by subtracting its mean. PCA selects a direction of largest projected variance, equivalently minimizing squared reconstruction error for a fixed projection dimension. It does not see the labels.
retained fraction = λ₁/(λ₁+λ₂)
For centered unscaled coordinates, the covariance eigenvectors give principal axes. The singular-value route to the closest rank-k approximation has the same squared-error objective. Scale the axes differently and PCA may choose another direction; a task-specific label objective must be evaluated separately.
Work one small world
Use points (t,+0.3) and (t,−0.3) for t∈{−3,−2,−1,1,2,3}. The horizontal variance is 14/3; the vertical variance is 0.09. Their cross-covariance is zero. The first component is horizontal and retains (14/3)/(14/3+0.09)=1400/1427, more than 98%. For every t the two opposite labels have the same horizontal projection.
| Projection | Retained variance | Label pairs |
|---|---|---|
| Horizontal (first PCA axis) | 1400/1427 | All six pairs collide |
| Vertical | 27/1427 | All pairs separate |
New collision case: take all twelve equally weighted points (t,±1/4), t∈{−4,−2,−1,1,2,4}, labeled by the sign of the second coordinate. The unscaled centered variances are 7 and 1/16 with zero cross-covariance. PCA retains 112/113 horizontally. A deterministic classifier receiving only that coordinate, not row identity, is correct on at most one of each opposite-label pair: best accuracy is 1/2.
THE BRIDGEThe next lesson separates uncertainty about an unseen state from geometric projection.
WHERE THIS IDEA STOPSExplained variance alone is not task information, predictive accuracy or intelligence.
Experiment
Choose Keep most variance (28°) and compare labeled pairs; then choose Keep label information (118°). Read the variance and reconstruction metrics at both angles. The lab rotates the cloud, so its on-screen principal axes are not the horizontal/vertical coordinates of the arithmetic example.
What useful distinction might be small in magnitude and therefore easy to throw away?
Check & explain
For the twelve-point ±1/4 case, what is the horizontal population variance?
What is the best deterministic horizontal-only classification accuracy on those balanced pairs?
- 1/2
- 112/113
- 1
Teach back
Construct a different paired cloud with a small label coordinate. State the scale, covariance, PCA direction, and what a classifier can and cannot observe.
- A missing centering/scale assumption: name the covariance being calculated.
- A missing collision proof: identify two differently labeled rows with the same projection.
- An objective mix-up: report reconstruction and label accuracy separately.
Read deeper
Lecture 7: Eckart–Young: The Closest Rank k Matrix to A
Gilbert Strang · MIT OpenCourseWare · Course lecture · MIT OCW
Use the closest-rank-k discussion for reconstruction, not a guarantee about labels.
Open the sourcePrincipal Component Analysis (PCA), Step-by-Step
Josh Starmer · StatQuest · Video · 22 min
Watch the full 22-minute PCA walkthrough, then solve the separate paired-label counterexample.
Open the source