Claude Shannon
Engineer who founded information theory, giving the field the mathematical vocabulary — bits, entropy, channel capacity — that loss functions and compression are still built on.
Claude Shannon (1916–2001) is best known for a 1948 paper, "A Mathematical Theory of Communication," that founded information theory from scratch. It answered a deceptively simple question: how much can a message be compressed, and how reliably can it be sent over a noisy channel? His answer introduced the bit as the fundamental unit of information and defined entropy as a precise measure of uncertainty — concepts borrowed directly from thermodynamics and repurposed for communication.
That vocabulary quietly underlies modern machine learningMachine Learning (ML)Machine learning is the practice of writing programs that learn a function from data, via a loss function and an optimizer, instead of following hand-written rules.. Cross-entropy, the loss functionLoss FunctionA loss function is a single number measuring how wrong a model's predictions are, which gradient descent minimizes during training. used to train almost every classifier and every LLMLLM (Large Language Model)An LLM is a large transformer trained to predict the next token on massive text corpora, then fine-tuned to follow instructions — the architecture behind GPT, Claude, Gemini, and Llama.'s next-token prediction, is Shannon's entropy applied to a model's predicted probability distributionProbability DistributionA probability distribution assigns a likelihood to each possible value of a random variable — the object every loss function is secretly built to measure the fit of.. TokenizationTokenizationTokenization converts raw text into a sequence of integers a model can process, typically via subword schemes like byte-pair encoding (BPE). schemes, model compression, and even the intuition that a model "learns" by reducing uncertainty about its data all trace back to concepts Shannon formalized before digital computers were common.
Shannon also had a hand in AIAI (Artificial Intelligence)AI is the field of building systems that perform tasks normally requiring human intelligence — reasoning, perception, language, and decision-making.'s founding directly — he co-organized the 1956 Dartmouth workshop alongside John McCarthyJohn McCarthyCoined the term "artificial intelligence" in 1955, organized the field's founding 1956 Dartmouth workshop, and invented Lisp., Marvin MinskyMarvin MinskyCo-founder of the MIT AI Lab and a founding figure of the field — his 1969 book with Seymour Papert exposed the perceptron's limits and helped trigger the first AI winter., and Nathaniel Rochester, the meeting that coined the term "artificial intelligence."
See also: Alan TuringAlan TuringMathematician who formalized what "computation" means and, in 1950, asked whether a machine could think — the question the entire field still answers to., John McCarthyJohn McCarthyCoined the term "artificial intelligence" in 1955, organized the field's founding 1956 Dartmouth workshop, and invented Lisp.
Learn more: ML Fundamentals · Wikipedia: Claude Shannon
Alan Turing
Mathematician who formalized what "computation" means and, in 1950, asked whether a machine could think — the question the entire field still answers to.
Walter Pitts
Self-taught logician who, with Warren McCulloch, co-authored the 1943 paper modeling neurons as logic gates — the mathematical starting point for every neural network.