Probability & Distributions

First Principles Question

How do you reason about things you’re uncertain about? And why does the bell curve appear everywhere?

The Core Idea

Probability is a language for uncertainty. A distribution describes all the ways a random variable can behave. The Gaussian (bell curve) appears everywhere because of one deep theorem: when you average enough random things together, the result always tends toward a bell curve — regardless of where those things came from.

Key Concepts to Cover

• Probability as frequency vs degree of belief (frequentist vs Bayesian)
• Conditional probability: P(A|B) — probability of A given B is true
• Bayes’ theorem: how you update a belief when you see new evidence
• Joint probability vs marginal probability
• Independence — when knowing one thing tells you nothing about another
• Common distributions: Gaussian, Bernoulli, Categorical, Multinomial
• Central Limit Theorem — why Gaussian appears everywhere
• MLE vs MAP — two ways to fit a distribution to data
• Expected value and variance

Why It Matters for ML

• Naive Bayes: Bayes theorem + feature independence assumption
• LDA: assumes each class follows a Gaussian distribution
• HMM: observations emitted from probability distributions per hidden state
• Logistic Regression: models P(class | features) directly
• Softmax in CNN: outputs a categorical distribution over classes

Prerequisites

None. This is Layer 0.

Builds To

Naive Bayes · Linear Discriminant Analysis · Hidden Markov Model · Statistics & Bias-Variance

Content Ideas

Obsidian note: “Why is the bell curve everywhere? The Central Limit Theorem explained without calculus.” X post: “The Gaussian distribution appears in ML everywhere. It’s not a coincidence. It’s the Central Limit Theorem.” GitHub: Visualize CLT convergence: average N dice rolls, watch the histogram become a bell curve