Mathematics for Machine Learning
Deisenroth, Faisal & Ong · Freely readable textbook
Read §§6.3–6.4, printed pp.183–196, for conditional probability, Bayes and covariance.
Open the sourceProbability: uncertainty is not geometry alone
A prior can rank explanations without making them identifiable.
Prerequisites: 03 · Sets: the grammar of selection, 10 · Inner products and projection, 25 · Hidden states and observable shadows
Understand
Objectives
- Compute a posterior from a complete finite likelihood table.
- Name the conditioning event and verify its nonzero probability.
- Separate zero covariance from statistical independence and identification.
Start with something you can see
Two hidden switches each produce a sum of 1: H₁=(1,0) or H₂=(0,1). Before using a prior, predict whether the sum alone identifies which switch is on. Now give H₁ prior 0.9. The Bayes companion describes revising beliefs; our equal-likelihood example tests when the reading does not revise their relative odds. [1]
Give the idea a precise name
A conditional probability is defined for an event B with P(B)>0. It describes the fraction of B cases also in A. Here H₁ and H₂ exhaust our two-state model, and D is the observed sum 1.
P(A|B)=P(A∩B)/P(B); P(H₁|D)=P(D|H₁)P(H₁)/P(D)
Because both hypotheses predict D with probability 1, P(D)=0.9+0.1=1. The observation has two possible preimages. The 0.9 posterior reflects the prior rather than a distinguishing reading. Statistical independence asks whether joint probabilities factor; linear independence asks about vector combinations and is a different property.
Work one small world
Compute the posterior by the table:
| Hidden state | Prior | P(D|state) | Joint with D | Posterior given D |
|---|---|---|---|---|
| H₁=(1,0) | 0.9 | 1 | 0.9 | 0.9 |
| H₂=(0,1) | 0.1 | 1 | 0.1 | 0.1 |
Both states have the same likelihood. Measure the first switch directly to distinguish them; a stronger prior is not a new observation.
Counterexample to “zero covariance means independence”: X is uniform on {−1,0,1} and Y=X². P(Y=0 | X=0)=1 since P(X=0)=1/3>0, while P(Y=0)=1/3. Yet E[X]=E[XY]=0, E[Y]=2/3, and Cov(X,Y)=0. Y is determined by X despite zero covariance.
THE BRIDGEThe next lesson asks how a measurable prediction claim could be compared with a baseline.
WHERE THIS IDEA STOPSConditioning on a probability-zero event is not defined by this finite ratio.
Experiment
Before inspecting the table, calculate each prior times its likelihood. Check that the joint column sums to P(D)=1. Swap the priors and observe that the sensor still does not distinguish the switches.
Which part of your conclusion came from the observation, and which part came from what you already assumed?
Check & explain
If exhaustive H₁,H₂ have priors 0.8,0.2 and both guarantee D, what is P(H₁|D)?
For X uniform on {−1,0,1} and Y=X², what disproves independence?
- Their covariance is zero
- P(Y=0 | X=0)=1 differs from P(Y=0)=1/3
- They are both real-valued
Teach back
Replace the sum sensor by a reading of the first switch. Rebuild both likelihood rows and posteriors. Explain which conclusion comes from the new measurement.
- A missing denominator: compute P(D) and require it positive.
- A prior/measurement confusion: name the equal likelihoods.
- An independence confusion: use a conditional event rather than covariance alone.
Read deeper
Bayes theorem, the geometry of changing beliefs
3Blue1Brown · Grant Sanderson · Video · 15 min
Start at 4:09; watch the short 4:09–6:39 passage or the full 15-minute video. The two-switch table is a separate calculation.
Open the sourceLecture 2: Independence and Conditioning
Ankur Moitra · MIT OpenCourseWare · Lecture · 71 min
Watch the full 71-minute lecture for conditioning and independence; its OCW page is this same card.
Open the source