Mathematics for Machine Learning
Deisenroth, Faisal & Ong · Freely readable textbook
Read §8.6, printed pp.283–287, on evaluating and selecting models.
Open the sourceWhat would “intelligence in the patterns” mean?
Turn a compelling intuition into several precise research questions.
Prerequisites: 24 · The Hodge bridge: signature → inequality, 26 · Harmony as a path, not a frozen shape, 27 · PCA and the geometry of compression, 28 · Probability: uncertainty is not geometry alone
Understand
Objectives
- Derive a nonlinear toy XOR calculation on four inputs.
- Design a held-out prediction comparison with metric and baseline.
- Separate a mathematical result, empirical hypothesis and philosophical interpretation.
Start with something you can see
Pick one prediction task: given past sensor readings, predict the next reading on an unseen trajectory. State the withheld trajectories, a persistence baseline and a mean-squared-error score before choosing a representation. A clear definition of an activation, such as ReLU, is not evidence that a whole model predicts well. [1]
Give the idea a precise name
A representation h=φ(x) transforms inputs x; a readout g(h) predicts an output. An empirical hypothesis must name a measurable task, a baseline, a held-out split and a possible result that refutes the proposed improvement. A theorem about a specified map is a different kind of claim.
s=x₁+x₂; ReLU(u)=max(0,u); f=ReLU(s)−2ReLU(s−1)
On the four binary inputs, this derived toy formula computes XOR. It is not a theorem that a particular trained network will generalize. Composing linear maps without nonlinearities still yields one linear map; for XOR, the four required outputs contradict linearity (and even affine additivity at the corners).
Work one small world
The finite truth table is:
| (x₁,x₂) | s | ReLU(s)−2ReLU(s−1) |
|---|---|---|
| (0,0) | 0 | 0 |
| (0,1) | 1 | 1 |
| (1,0) | 1 | 1 |
| (1,1) | 2 | 0 |
For a separate empirical waveform claim, train on complete trajectories, hold out other complete trajectories, and compare next-step squared error with “predict the last reading.” Failure to beat that baseline on the held-out trajectories falsifies the stated improvement claim, not every possible definition of intelligence.
Counterexample to treating a pattern as intelligence evidence: a model that memorizes neighboring time points in a randomly split trajectory can score well while failing on a new trajectory. Splitting whole trajectories and fitting preprocessing only on training data removes that shortcut. XOR itself is a binary-domain calculation, not a real-valued truth rule.
THE BRIDGEThe capstones now ask for explicit translation maps and inspectable evidence rather than analogies.
WHERE THIS IDEA STOPSDo not call a high-dimensional picture, compressed representation or successful toy calculation a measurement of intelligence.
Experiment
Compute the four XOR rows before assessing a learning claim. For a waveform proposal, state a whole-trajectory train/test split, a fixed error metric, persistence baseline and a result that would change your mind.
What observable result would force you to revise this account of intelligence?
Check & explain
For (x₁,x₂)=(1/4,1/4), what does the displayed ReLU formula output?
Which observation would falsify the stated improvement claim?
- A compelling latent plot
- Higher error than persistence on held-out whole trajectories
- A larger number of model parameters
Teach back
Write a claim with columns for map, measurement, baseline and falsifier. Give a competing explanation for any score improvement, and say what a successful score would not establish about intelligence.
- A missing task: define input, output and withheld trajectories.
- A missing baseline/falsifier: compare mean squared error against persistence.
- An evidence leap: a toy XOR identity is not an empirical or philosophical conclusion.
Read deeper
Lecture 26: Structure of Neural Nets for Deep Learning
Gilbert Strang · MIT OpenCourseWare · Lecture · 53 min
Start at 5:00 for the 5:00–25:00 longer passage; the short core is 9:52–12:52. The full 53-minute lecture is available from this card.
Open the source