Optimization & Gradient Descent
First Principles Question
You’re blindfolded on a hilly landscape and want to reach the lowest point. How do you move?
The Core Idea
You feel the slope beneath your feet (the gradient) and step downhill. Repeat. That’s gradient descent. The loss function defines the landscape. The model’s parameters define where you are on it. Training is just walking downhill until you stop moving.
Key Concepts to Cover
- Loss function — what you’re trying to minimize; measures model wrongness
- Convex vs non-convex loss surfaces — one valley vs many
- Gradient — vector pointing in the direction of steepest ascent
- Gradient descent update: θ = θ − α·∇L(θ)
- Learning rate (α) — step size; too large overshoots, too small crawls
- Local vs global minima — why non-convex problems are hard
- Batch vs Stochastic vs Mini-batch gradient descent
- Momentum, Adam — why plain gradient descent is often not enough
- 1D minimization methods: Exhaustive, Fibonacci, Golden Section (deterministic)
- Stochastic search: GA, PSO — for non-differentiable or multimodal landscapes
Why It Matters for ML
- Linear Regression: closed-form solution is the minimum of MSE loss
- Logistic Regression: gradient descent on cross-entropy loss
- GBT: each new tree fits the negative gradient of the loss function
- CNN: backpropagation computes gradients; Adam updates weights
Prerequisites
Builds To
Linear Regression · Gradient Boosting Tree · Convolutional Neural Network
Content Ideas
Obsidian note: “Gradient descent is just walking downhill. Here’s why that’s all you need.”
X post: “Every ML model you’ve ever used was trained by walking downhill on a math landscape. That’s it.”
GitHub: gradient-descent-viz — animate convergence on a 2D loss surface with different learning rates