Introduction to Information Theory

1. Probability and logs (the only math you need)

A probability is a number from 0 to 1 that says how likely something is: 0 means it never happens, 1 means it always happens. A power such as 2³ = 2 × 2 × 2 = 8 multiplies 2 by itself. A logarithm base 2, written log₂, asks the reverse question: which power of 2 gives this number? log₂(8) = 3, because 2³ = 8. These tools are all the math this course needs.

Think of light switches. One switch has 2 positions (off, on). Two switches give 2 × 2 = 4 patterns, three give 2³ = 8. The power answers "how many patterns can b switches make?" The log answers the reverse: "how many switches do I need to give each of n rooms its own pattern?" For 8 rooms, log₂(8) = 3 switches. A bit (binary digit) is exactly such a switch: a 0 or a 1. Probability works like slices of one pizza: each possible outcome gets a slice, and all the slices together make exactly one whole pizza.

The demo has two modes. Power and log: move the bits b slider to list every code you can write with b bits, and see the point (2^b, b) on the curve y = log₂(x). The zoomed plot shows numbers below 1, where log₂ turns negative; click a fraction to see it. Loaded die shows a probability for each face of a die (a cube with the numbers 1 to 6 on its faces); move the sliders to make some faces more likely and watch the probabilities always add up to 100%.

Probability in three rules

A distribution is the full list of outcomes with their probabilities, like the six bars in the loaded-die mode. Every later lesson starts from a distribution.

The average of a distribution

The expected value is a weighted average: multiply each outcome by its probability, then add the results. For a fair die: (1 + 2 + 3 + 4 + 5 + 6) × 1/6 = 21 ÷ 6 = 3.5. For the loaded die in the code panel, faces 1 to 5 have probability 0.1 each and face 6 has 0.5: (1 + 2 + 3 + 4 + 5) × 0.1 + 6 × 0.5 = 1.5 + 3 = 4.5. Entropy (lesson 4) is exactly this kind of weighted average.

Power and log: one question, two directions

Read backwards, log₂ also counts halvings: 16 → 8 → 4 → 2 → 1 takes 4 halvings, so log₂(16) = 4. Doubling 4 times goes up from 1 to 16; halving 4 times comes back down. Lesson 3 uses this to define the bit.

How to read the plot: the horizontal axis is x, the vertical axis is log₂(x). The curve passes through (1, 0), because 2⁰ = 1. It climbs fast at first and then slowly: going from 16 to 32 adds only one step, because doubling x always adds exactly 1. Numbers between powers land in between: 6 lies between 4 and 8, so log₂(6) is between 2 and 3 (about 2.58). Below 1 the curve drops under 0 and falls steeply as x gets close to 0, as the zoomed plot shows. Flip that negative part upside down and you get the surprise curve of lesson 3. Probabilities are numbers between 0 and 1, so their log₂ is 0 or negative: that is the part of the curve the rest of the course uses most.

Why base 2?

• It counts bits. With b bits you can write 2^b different codes, so giving n things each its own code needs log₂(n) bits. That is why information is measured in bits (lessons 2 and 3), and why computers, which store everything as 0s and 1s, fit it so well.
• Other bases only change the unit. Base 10 counts decimal digits; base e (the "natural" log, e ≈ 2.718) counts units called nats, which you will meet in AI libraries (lesson 9). The curve has the same shape; only the scale changes.
• It turns multiplying into adding, and dividing into subtracting. log₂(a × b) = log₂(a) + log₂(b), and log₂(a ÷ b) = log₂(a) − log₂(b). Example: log₂(2 × 8) = log₂(16) = 4 = 1 + 3. Independent probabilities multiply (rule 3 of probability, above), so this is why their information simply adds up (lesson 3). The dividing rule gives log₂(1/p) = log₂(1) − log₂(p) = −log₂(p), the key formula of lesson 3.

Where it shows up

• Weather forecasts: "70% chance of rain" is the probability 0.7.
• Colours on a screen: 24-bit colour can show 2²⁴ = 16,777,216 different colours.
• Games and draws: dice, cards and lotteries are all distributions.

2. What is information? (Shannon's big idea)

In 1948, the engineer Claude Shannon gave information a precise meaning: information is what reduces your uncertainty, that is, your not knowing which of several possibilities is true. It is about what you did not know before, not about what a message means.

Imagine two messages from a friend. "The sun rose this morning" tells you almost nothing: you were already sure of it. "It snowed in the desert last night" tells you a lot: you did not expect it at all. Both messages have meaning, but only the second one removes real uncertainty. Shannon's theory measures that second kind of value, and it works the same for text, sound, pictures or DNA.

The communication model

Shannon described every kind of communication with the same five parts. A message starts at a source. An encoder turns it into signals (for a computer, bits: binary digits, 0s and 1s). The signals travel through a channel (a wire, radio waves, a disk), where noise (random disturbance) can change them. A decoder turns the signals back into a message, and a receiver gets it.

In the demo, type a short message and press Send. The encoder writes each letter as 8 bits, using ASCII, the standard table that gives every letter a number (H is 72, which is 01001000 in bits). Turn on Noise to let the channel flip a few bits and see what reaches the receiver.

The five parts, in everyday examples

Information is not meaning

Shannon left meaning out on purpose. For an engineer, the job is to get the message across exactly, whatever it says. This one choice made the theory work for every kind of data at once.

Shannon's two big questions

1. How small can a message be? How few bits are enough to write it down? This is compression (lesson 6).
2. How fast can we send it reliably through noise? How many bits per second can a noisy channel carry without mistakes? This is channel capacity (lesson 7).

To answer both, we first need a way to measure information. The pattern in the table above, the less likely the message, the more information it carries, is exactly what lesson 3 turns into a number: an event with probability p carries −log₂(p) bits. Lesson 4 then averages it over a whole source.

Where it shows up

• Every digital device: phones, Wi-Fi, USB and streaming all follow the source → encoder → channel → decoder → receiver chain.
• Space probes: spacecraft send data across billions of kilometres through a very noisy channel, using codes built on Shannon's ideas.
• Storage: a disk or memory card is a channel through time: you write now and read later, and bits can still go wrong.

3. Surprise and the bit (measuring one event)

The bit is the unit of information: the amount you get from the answer to one fair yes/no question. The surprise (also called self-information) of an event with probability p is −log₂(p) bits: the rarer the event, the bigger the surprise. This is the form used in machine learning (ML) papers and libraries.

Think of news. "The sun rose today" surprises nobody, so it carries almost no information. A fair coin landing heads is a small surprise: exactly one bit, the same as the answer to one fair yes/no question. Winning a 1-in-a-million draw is a huge surprise, and it carries many bits.

In the demo, each event card is an everyday event with its probability. Click a card to turn that probability into bits: the meter shows its surprise, and its dot lights up on the surprise curve below. Press ▶ Show all to go through the cards from the most likely to the least likely.

From probability to bits

Both forms give the same number: lesson 1 showed that log₂ of a fraction is negative, and the minus sign makes the surprise positive. A fair coin has p = 1/2, so heads gives −log₂(1/2) = 1 bit: the same as the answer to one fair yes/no question.

Surprise is information. The surprise of an event is exactly the amount of information you get when it happens, which is why it is also called self-information or information content. Information (lesson 2) and surprise are two names for one number, −log₂(p): the less likely the event, the more you learn when it happens.

Why one yes/no answer is one bit: a fair yes/no answer halves the possibilities. To find one number out of 16 by asking "is it in the lower half?": 16 → 8 → 4 → 2 → 1, which is 4 answers, so 4 bits = log₂(16), the halving from lesson 1. The same rule appears in the demo's formula strip and as the dashed steps on the curve: halve p, and the surprise grows by exactly +1 bit.

The surprise curve

The plot under the event cards draws surprise for every probability: y = −log₂(p), for p between 0 and 1. It is the negative part of the log curve from lesson 1 (where x is below 1), flipped upside down by the minus sign. How to read it: at p = 1 the curve touches 0 (no surprise); as p shrinks, the curve rises, slowly at first and then very steeply near p = 0. The dashed purple steps show the key pattern: going from p = 1 to 1/2 to 1/4 to 1/8 adds exactly +1 bit each time. A 1-in-a-million event (19.93 bits) is far above the top of the chart.

You will meet this same curve again: entropy (lesson 4) is its average, and in lesson 9 it becomes the loss that trains AI classifiers.

Why −log₂(p)? Three rules of surprise

1. Certain events give 0 bits. If p = 1, then −log₂(1) = 0: no news.
2. Halving the probability adds 1 bit. 1/2 gives 1 bit, 1/4 gives 2 bits, 1/8 gives 3 bits.
3. Surprises of independent events add up. A coin showing heads and a die showing 6 has p = 1/2 × 1/6 = 1/12 (lesson 1). Its surprise is log₂(12) ≈ 3.58 bits, which equals 1 + 2.58: the two surprises added together.

Rule 3 is the reason Shannon used a logarithm: probabilities of independent events multiply, and the logarithm turns that multiplication into a simple sum.

Where it shows up

• Passwords: a random 4-digit PIN has p = 1/10,000, a surprise of about 13.3 bits. More bits mean harder guessing (lesson 9).
• Searching: finding one item among a million sorted items needs about 20 yes/no steps, because log₂(1,000,000) ≈ 19.93.
• News value: journalists know it without the formula: "dog bites man" is no story, "man bites dog" is.

4. Entropy (measuring a whole source)

Entropy, written H, is the average surprise of a source: how uncertain you are, on average, about the next outcome before you see it. It is the expected value (lesson 1) of the surprise (lesson 3), measured in bits.

Compare the weather in a desert city and in a city where the weather changes every day. In the desert, tomorrow is almost always sunny, so a forecast tells you very little: low entropy. In the changing city you really don't know what tomorrow brings, so the forecast is worth a lot: high entropy.

Notation: H(p) means the entropy of the whole distribution p (lesson 1); p(x) is the probability that p gives to one outcome x.

The demo has two modes. Coin lets you change how biased a coin is (how often it lands heads) and draws the entropy curve. Weather has four outcomes (sunny, cloudy, rainy, snowy) with a slider each. In both, the table under the bars shows every term of the formula: probability, surprise and their product.

How to read the coin curve

The horizontal axis is the probability of heads, from 0 to 1. The vertical axis is the entropy in bits. The curve is a hump: 0 at both ends, highest (1 bit) in the middle. A worked example for the 90/10 coin, using the formula above:

Heads is common, so its surprise is tiny (0.152 bits). Tails is rare and surprising (3.32 bits), but it happens only 10% of the time. On average you get less than half a bit per toss. The coloured bar under the demo shows the same split: the tails segment (0.332) is more than twice the heads segment (0.137), so the rare outcome carries most of the entropy.

Three properties of entropy

The "Mixed" weather preset (sunny 1/2, cloudy 1/4, rainy 1/8, snowy 1/8) gives a clean example: the surprises are 1, 2, 3 and 3 bits, so H = 1/2 × 1 + 1/4 × 2 + 1/8 × 3 + 1/8 × 3 = 0.5 + 0.5 + 0.375 + 0.375 = 1.75 bits. That is less than the 2-bit maximum, because sunny days are predictable.

Entropy of English

If all 26 letters were equally common, each letter would carry log₂(26) ≈ 4.7 bits. Real English is not equal: E, T and A are common, Q and Z are rare. Counting letter frequencies alone gives about 4.1 bits per letter. Using context lowers it a lot: after "informatio" you can be almost sure the next letter is "n". Shannon's experiments in 1951, with people guessing the next letter, estimated roughly 1 bit per letter. This gap between 4.7 and about 1 is why text compresses so well (lesson 6).

Where it shows up

• Compression: entropy is the smallest average number of bits per symbol a lossless code can reach (lesson 6).
• Decision trees in machine learning: they pick the question that lowers entropy the most (lesson 9).
• Passwords and keys: security people talk about "bits of entropy" to say how hard a secret is to guess.

5. Information shared between things (mutual information)

Two things can share information: knowing one tells you something about the other. Mutual information, written I(X;Y), measures how many bits of uncertainty about X disappear once you know Y. It is 0 when the two are unrelated, and as large as the entropy itself when one fully decides the other.

Think of the weather (X) and whether your neighbour carries an umbrella (Y). If you can't see the sky but you see the umbrella, you can guess the weather better. The umbrella "carries" some information about the weather. If your neighbour picks the umbrella at random, it tells you nothing.

Here X and Y are random variables: quantities whose value is decided by chance, each with its own distribution. H(X) means the entropy of X's distribution, the same quantity lesson 4 wrote as H(p).

Three new quantities

In the demo, the slider sets how often the umbrella matches the weather (umbrella on rainy days, no umbrella on dry days). To keep the numbers simple, half of the days are rainy, so the weather alone has H(X) = 1 bit, and the umbrella alone also has H(Y) = 1 bit. The joint table shows the probability of each of the 4 combinations.

How to read the picture

Each circle's area is the entropy of one thing: 1 bit for the weather and 1 bit for the umbrella. The overlap is what they share, I(X;Y). The blue part outside the overlap is what is still unknown about the weather after you see the umbrella, H(X|Y); the purple part outside the overlap is the same idea the other way round, H(Y|X). Both circles together (the whole coloured area) are the joint entropy H(X,Y). This gives the main formula:

A worked example for the 80% setting. The joint table has 0.4 for rain + umbrella, 0.4 for dry + no umbrella, and 0.1 for each of the two mismatches (0.4 + 0.4 + 0.1 + 0.1 = 1). Then:

Where it shows up

• Choosing useful data in machine learning: keep the inputs (features) that share the most information with what you want to predict.
• Decision trees: their "information gain" is mutual information between a question and the answer you want (lesson 9).
• Communication: the information that gets through a noisy channel is the mutual information between what was sent and what was received (lesson 7).

6. Compression (the intuition)

Compression means writing the same message with fewer bits. The trick is simple: give short codes to common symbols and long codes to rare ones. Shannon showed that the entropy of the source (lesson 4) is the limit: on average, no lossless code (one that gives back the exact original) can use fewer bits per symbol than the entropy.

Morse code, created for the telegraph in the 1800s, already used this idea. The most common English letter, E, is a single dot. A rare letter like Q is long: dash dash dot dash. Operators sent common letters quickly and paid the long price only for rare ones.

In the demo, pick a text and press Encode. The demo counts how often each letter appears, gives each letter a code (common = short), and writes the whole text in bits. The meters compare four ways to store the text, in bits per letter. The demo builds its codes with a method called Huffman coding (named after its inventor); you don't need its steps, only the result: common letters get short codes.

How to read the meters

Where the skewed numbers come from: the 32 letters are A ×16, B ×8, C ×4 and D ×4, so the probabilities are 1/2, 1/4, 1/8 and 1/8. The codes are A = 0, B = 10, C = 110, D = 111. Total length = 16 × 1 + 8 × 2 + 4 × 3 + 4 × 3 = 56 bits, and 56 ÷ 32 = 1.75 bits per letter. That is exactly the entropy of the "Mixed" weather in lesson 4, because the probabilities are the same. Plain text would need 32 × 8 = 256 bits.

Why random data won't compress

When every letter is equally likely, there are no "common" letters to make short. The entropy then equals the fixed length: 16 equally likely letters need log₂(16) = 4 bits each, whatever trick you use. Try the Random letters text: our code is close to 4 bits and can't beat the entropy. The same is true for files that are already compressed: their bits look random, so zipping a zip file saves almost nothing.

Lossless and lossy

Real tools such as ZIP go further than this demo: besides counting single letters, they also find repeated pieces ("THE " appears many times) and replace them with short references. The limit is still entropy, but measured on the text with all its patterns, which is much lower (about 1 bit per English letter, lesson 4).

Where it shows up

• Every file format: ZIP, PNG, JPEG, MP3 and video codecs all give short codes to common patterns.
• Messaging and web pages: text is compressed before it travels, so pages load faster.
• AI: a model that predicts the next word well can also compress text well; good prediction and good compression are the same skill.

7. Noisy channels (the intuition)

Every real channel has noise: now and then a 0 arrives as a 1, or a 1 as a 0 (lesson 2). Error-correcting codes fight back by adding smart redundancy: extra bits that let the receiver notice, or even fix, flipped bits. Shannon showed that every channel has a capacity: a speed limit below which messages can be sent almost without errors.

You already do this on a bad phone line. When you spell your name, you say "A as in Apple, B as in Banana". The extra words are redundancy: even if the line drops part of a sound, the listener can still recover the letter.

In the demo, a 12-bit message crosses a noisy channel. Set the noise (the chance that each bit flips) and pick a protection: None, a Parity bit (one extra bit per 4 bits that detects an error) or Repeat ×3 (send every bit three times and let the majority win). Press Send several times: the noise is random, so each send is different.

Three protections compared

How the parity bit works: the sender counts the 1s in a block of 4 bits and adds one extra bit so the total number of 1s is even. For 1011 (three 1s) the extra bit is 1, so it sends 10111. The receiver counts again: if the count is odd, some bit flipped. It cannot tell which one, so it asks for the block again.

How much Repeat ×3 helps: with 10% noise, a bit is decoded wrong only if 2 or all 3 copies flip. That chance is 3 × 0.1 × 0.1 × 0.9 + 0.1 × 0.1 × 0.1 = 0.027 + 0.001 = 0.028, about 2.8% instead of 10%. Better, but it costs three times the bits.

Capacity: the speed limit

Repetition is a clumsy code: it pays a lot and still makes mistakes. Shannon proved something surprising in 1948: for every channel there is a number, its capacity, such that smart codes can make errors as rare as you like, as long as you send information slower than the capacity. Above it, no code can work. In the words of lesson 5, capacity is the largest mutual information between what is sent and what is received: how many bits about the input survive the noise.

For a channel that flips each bit with 10% chance, the capacity is 1 − H(0.1), where H(0.1) is the entropy of a 90/10 coin from lesson 4: 1 − 0.47 = 0.53 bits of real information per bit sent. The noise "eats" 0.47 bits of every bit. Repeat ×3 carries only 1/3 ≈ 0.33 bits per bit sent and still has 2.8% errors, so it is far from the limit. Shannon's proof showed the limit exists; it took engineers about 45 years to build codes that come close.

Where it shows up

8. Cross-entropy and KL divergence (the cost of a wrong belief)

If you build your code, or your guesses, on the wrong probabilities, you pay extra bits. Cross-entropy H(p,q) is your average surprise when the world follows the true distribution p but you believe in a distribution q. KL divergence KL(p‖q) is the extra cost: cross-entropy minus the entropy of p. It is 0 only when your belief matches the truth.

Think of a traveller who packs for the wrong weather. If they expect sun every day but it often rains, each rainy day is a big, costly surprise. A traveller who knows the real weather is surprised less often. The difference between the two travellers' average surprise is the KL divergence.

Step by step: comparing two distributions

Cross-entropy is often used to measure how different two distributions are: the more q disagrees with p, the larger H(p,q). Take a random variable X (lesson 5) with three possible events: drawing a red, green or blue ball. Two distributions describe it:

Both columns add up to 1 (0.10 + 0.40 + 0.50 = 1 and 0.80 + 0.15 + 0.05 = 1). The demo computes the formula one event at a time. Pick a direction: H(p, q) takes the weights (how often each event happens) from p and the surprises from q; H(q, p) swaps the roles. The coloured bar at the bottom adds each event's term, so you can see which event costs the most.

Why H(p, q) is not equal to H(q, p)

The weight of each term comes from the first distribution and the surprise from the second, so swapping them changes which event dominates:

• H(p, q): the big cost is blue. Under p it happens half the time (0.50), but q gave it only 5%, so every blue ball costs 4.32 bits. 0.50 × 4.32 ≈ 2.16 bits, about two thirds of the total.
• H(q, p): the big cost is red. q expects red 80% of the time, but p gives it only 10%, so every red costs 3.32 bits. 0.80 × 3.32 ≈ 2.66 bits.

Different events dominate in each direction, with different weights and surprises, so the two totals differ: 3.288 ≠ 2.906. That is why cross-entropy is a measure of difference but not a true distance. A distance would be the same in both directions, and it would be 0 for two identical distributions. Cross-entropy is not 0 even then: H(p, p) is the entropy of p, about 1.36 bits here. KL divergence, next, removes that part.

Step by step: KL divergence

KL divergence keeps only the extra cost of the wrong belief. Using the same red / green / blue balls, it can be computed directly, one event at a time:

Each term is the weight p(x) times log₂(p/q): how many extra bits event x costs because q was used instead of p. A term is negative when q gives the event more probability than p does (that event is cheaper than it should be). The demo draws the running total as a waterfall: each event's term moves the total right (extra cost) or left (a saving). The savings never win: the total is always 0 or more, and it is 0 only when q = p.

Why KL(p‖q) is not equal to KL(q‖p)

Check with the earlier numbers: KL(p‖q) = H(p, q) − H(p) ≈ 3.288 − 1.361 = 1.927, and KL(q‖p) = H(q, p) − H(q) ≈ 2.906 − 0.884 = 2.022. (H(p) and H(q) are the entropies of p and q, lesson 4.)

• KL(p‖q) is mostly blue: under p blue is common, but q made it 10 times rarer (0.50 ÷ 0.05 = 10), so each blue costs log₂(10) ≈ 3.32 extra bits. Red gives a small saving (−0.300), because q over-prepared for red.
• KL(q‖p) is mostly red: under q red is common, but p made it 8 times rarer (0.80 ÷ 0.10 = 8), costing log₂(8) = 3 extra bits each time.

The weights come from the first distribution, so each direction cares most about the events it thinks are common. That is why the two totals differ, and why KL, like cross-entropy, is not a true distance. It does have one distance-like property cross-entropy lacks: KL(p‖p) = 0.

Your belief vs the real weather

The second demo uses the weather again. The true weather p is fixed: sunny 1/2, cloudy 1/4, rainy 1/8, snowy 1/8, the "Mixed" weather from lesson 4 (H = 1.75 bits). Drag the sliders to set your belief q. The stacked bar shows your cross-entropy split into two parts: the unavoidable entropy of the weather, and the extra KL cost of your wrong belief.

Link with compression

Lesson 6 gave each symbol a code of about −log₂(p) bits. If you build your code from a belief q instead of the truth p, each symbol costs −log₂(q) bits, and the average is exactly the cross-entropy. Example: with the "Uniform belief" (1/4 for each weather), every weather gets a 2-bit code, so you pay 2 bits per day instead of the best possible 1.75. The extra 2 − 1.75 = 0.25 bits per day is the KL divergence.

Where it shows up

• Training AI models: the standard loss for classifiers (models that pick a label, such as "cat" or "dog") and language models is cross-entropy; training makes the model's belief q closer to the data's truth p (lesson 9). The Learn the truth button shows this idea: as q moves toward p, the red KL part shrinks to 0.
• Comparing distributions: KL divergence measures how different two distributions are, for example a model before and after fine-tuning (extra training on new data).
• Compression: a compressor built on a poor model of the data wastes exactly KL bits per symbol.

9. Information theory in AI and beyond (putting it all together)

Modern AI runs on the ideas of this course. Classifiers and language models are trained by lowering cross-entropy (lesson 8). Decision trees choose their questions by information gain, the drop in entropy (lessons 4 and 5). Language models are scored by perplexity, which is just average surprise turned back into "how many choices".

An AI model is a guesser. It never knows the answer for sure; it gives probabilities. Information theory gives the scoreboard: a good guesser is rarely surprised by the truth.

The demo has three modes, one for each idea. Classifier loss: a model looks at a photo of a cat and gives probabilities to "cat", "dog" and "fox"; move its confidence in the correct answer and watch the loss. Decision tree: 16 days of "did we play outside?"; pick the question that splits the days best. LLM perplexity: a language model reads a sentence word by word; see how surprised it is by each real next word.

1 · Cross-entropy loss

When the correct answer is "cat", the truth p puts all its probability on "cat". The cross-entropy from lesson 8 then has only one term: the surprise of the correct answer under the model's belief. This number is called the loss, and training means changing the model to make it small.

This is the surprise curve from lesson 3, used as a scoreboard: the model is "surprised" by the right answer when it gave it a low probability, and training lowers that surprise.

The curve in the demo shows how hard it punishes confident mistakes: at q = 0.9 the loss is −log₂(0.9) ≈ 0.15 bits; at q = 0.5 it is 1 bit; at q = 0.01 it is −log₂(0.01) ≈ 6.64 bits.

2 · Information gain in decision trees

A decision tree is a model that predicts by asking yes/no questions, like the fair yes/no questions of lesson 3. At each step it picks the question with the highest information gain: the entropy before the question minus the average entropy after it. This is the mutual information between the question and the answer (lesson 5).

Worked example: 16 days, 8 played outside and 8 did not, so the entropy before is 1 bit. "Is it the weekend?" gives 4 weekend days (all 4 played: entropy 0) and 12 weekdays (4 played, 8 did not: entropy ≈ 0.918). Average after = 4/16 × 0 + 12/16 × 0.918 ≈ 0.69. Gain = 1 − 0.69 ≈ 0.31 bits, the best of the three questions. "Is it windy?" splits the days 4/4 and 4/4, leaves the entropy at 1 bit and gains nothing.

3 · Perplexity of a language model

A large language model (LLM) writes text by predicting the next word again and again. To score it, take real text, look at the probability the model gave each actual next word, and average the surprises. LLM APIs report these as log-probabilities ("logprobs"): for each token (a word or a piece of a word), log(q), using the natural log. Its negative, −log(q), is that token's surprise in nats. Perplexity = 2^{average surprise} turns those bits back into a number of choices (in ML libraries you will see the same number written as perplexity = exp(average NLL), using nats): a perplexity of 2.6 means the model was, on average, as unsure as if it were choosing between about 2.6 equally likely words.

In the demo, the easy sentence's words get probabilities 0.10, 0.30, 0.70, 0.80 and 0.50, with surprises 3.32 + 1.74 + 0.51 + 0.32 + 1.00 ≈ 6.90 bits (the rounded parts add to 6.89; the exact total is 6.895). Average = 6.90 ÷ 5 ≈ 1.38 bits, so perplexity = 2^1.38 ≈ 2.6. The odd sentence averages about 5.81 bits, a perplexity of about 56. (The probabilities are invented for the demo, but real models are scored exactly this way.) Lower perplexity = a better model of the language.

Beyond AI

Glossary

The key terms from the course on one page. Filter by topic to revise before the final quiz.

Final quiz & certificate

This is the last step of the course: one list of 12 questions on the main concepts of every lesson. After each answer you'll see the correct option and why it's right. Score 80% or more (10 of 12) to earn a certificate with your name.