ML First Principles

Linear Regression

Properties1
tagsmodel, regression, first-principles, suryansh.co

2 min read

Linear Regression

First Principles Question

Given a set of points, what’s the line that comes closest to all of them at once?

The Core Idea

You define “closest” as minimizing the sum of squared vertical distances (MSE). That’s a quadratic function of the model’s parameters. A quadratic has exactly one minimum. Set the derivative to zero, solve — you get the line. One equation. That’s the entire algorithm.

The Math

y = β₀ + β₁x₁ + β₂x₂ + ... + ε

Loss = MSE = (1/n) Σ (yᵢ - ŷᵢ)²

Closed-form solution: β = (XᵀX)⁻¹ Xᵀy   ← pseudo-inverse

What Makes It Work

  • Assumes a linear relationship exists
  • Assumes errors are normally distributed with constant variance (homoscedasticity)
  • Minimizing MSE = maximizing likelihood under Gaussian error assumption (MLE)

What It Can’t Do

  • Non-linear relationships (need polynomial features or different model)
  • Classification (use Logistic Regression instead)
  • Works poorly with correlated features (multicollinearity)

Connection to Other Notes

The closed-form solution uses the pseudo-inverse from Linear Algebra. The gradient descent alternative uses Optimization & Gradient Descent. Regularized versions (Ridge, Lasso) address Statistics & Bias-Variance. LR is the simplest case of what Gradient Boosting Tree generalizes.

Prerequisites

Linear Algebra · Calculus · Optimization & Gradient Descent

Builds To

Gradient Boosting Tree · Convolutional Neural Network (loss functions)

Content Ideas

Obsidian note: “Linear regression is one derivative set to zero. Here’s the whole derivation.” X post: “Linear Regression has a closed-form solution. You don’t need gradient descent at all. Most people don’t know this.” GitHub: ml-from-scratch — LR in NumPy, both closed-form and gradient descent, side-by-side

Graph View

Backlinks