Computer Science & Data Science · Information theory

Shannon entropy

Shannon entropy is the expected information of a discrete random experiment and measures uncertainty in its probability distribution in bits.

AdvancedBachelor foundationsSubject specialization

Create flashcards or a quiz I already have an account

Free to browse · flashcards or a multiple-choice quiz · import from Essential

Formula

Calculate

H (X) = - i = 1 \sum n p_{i} lo g_{2} p_{i}

H in bits; p_i dimensionless, 0 ≤ p_i ≤ 1 and Σp_i = 1

Variables & units

Variables & units
Symbol	Meaning	Unit
$H(X)$	Entropy of random variable X	bit
$pᵢ$	Probability of outcome i	dimensionless
$n$	Number of possible outcomes	count

LaTeX

H(X) = -\sum_{i=1}^{n} p_i \log_2 p_i

QUANTA STUDYquanta-study.de

Binary Shannon entropy curve from p = 0 to 1. It starts at zero, rises symmetrically to a maximum of 1 bit at 0.5, then falls to zero. — Binary entropy H(p): the maximum is 1 bit at p = 0.5. Certain outcomes at p = 0 or 1 have zero entropy.

Shannon entropy calculator

Find the entropy in bits from a complete list of probabilities that add up to exactly 1, or from counted frequencies.

Calculation model

Outcome 1

Outcome 2

Outcome 3

Filled with example values. Type your own and the result updates straight away.

Working · rearrange for H, substitute, evaluate

1. Rearrange for H
$H=-\sum_{p_i>0}p_i\log_2p_i$
2. Substitute values (in SI units)
$-\left(0.5\right)\cdot\log_2\left(0.5\right)+-\left(0.25\right)\cdot\log_2\left(0.25\right)+-\left(0.25\right)\cdot\log_2\left(0.25\right)=\left(1.5\right)$
3. Result
H = 1.5 bit

Supplied sum: 1; no normalisation. Maximum entropy of an exactly normalised list with 3 outcomes: 1.584962500721156 bits.

Result: Mean discrete information per output H

H = 1.5 bit

With 0.5, 0.25 and 0.25, H = 1.5 bit.

Quick tips

A uniform distribution gives the maximum: with n equally likely outcomes H = log₂ n, so 1 bit for a coin and 2 bits for four outcomes.
Mental maths with powers of two: p = 1/2ᵏ contributes k · p. For 1/2, 1/4 and 1/4 that is 0.5 + 0.5 + 0.5 = 1.5 bits.

Assumptions and limits

Finite complete discrete outcomes, not a continuous density or differential entropy. Zero probability contributes 0 and log(0) is not evaluated. Entropy is a probability-weighted mean, not semantic meaning/file size or an assumption of independent sequences. Result and contributions are in bits per output; displaying nats multiplies by ln 2. Maximum 100 rows is an interface limit, not a physical/information-theoretic limit. Each probability is in [0,1]. The compensated sum must differ from 1 by at most 10⁻¹² absolutely (numerical input tolerance). No silent normalisation; supplied sum is shown. This tolerance does not replace a complete scientifically appropriate distribution.

Derivation & background

The base-2 logarithm gives the unit bits. Each possible outcome contributes its surprise −log₂(p_i); entropy averages this surprise using probabilities p_i. Terms with p_i = 0 are defined as 0 by continuity. For n possible outcomes, H lies between 0 and log₂(n); the maximum occurs for a uniform distribution. Entropy describes the source distribution, not directly the length of every particular message.

Exam blueprint

Validity range

Finite discrete probability distribution with p_i ≥ 0 and Σp_i = 1; base 2 gives bits. Zero-probability terms contribute zero by a limit. This does not directly give file size or differential entropy.

Derivation steps

Entropy is the expected logarithmic information content.

1Specify discrete outcomes and probabilities $p_i\ge0$ with $\sum_i p_i=1$ .
2For $p_i>0$ , calculate information $I_i=-\log_2p_i$ . Base 2 gives bits.
3Weight each information value by its probability: $p_iI_i$ .
4Add the contributions: $H(X)=\sum_i p_iI_i=-\sum_i p_i\log_2p_i$ .
5For $p_i=0$ , set the whole contribution to zero by its limit; do not evaluate $\log_2(0)$ separately.
6Check $0\le H\le\log_2n$ ; the maximum occurs for n equally likely outcomes.

Rearrangements

Defining relation

H(X) = -\sum_{i=1}^{n} p_i \log_2 p_i

The equation states the defining relation within its stated domain.

Task variant

Find H for probabilities (1/2, 1/4, 1/8, 1/8).

Information values are 1, 2, 3, 3 bits.
Weighted: H = 1/2 + 2/4 + 3/8 + 3/8 = 1.75 bits.

Common mistakes

Adding information values without probability weights.

Multiply each −log₂ p_i by p_i; the quantity is an expectation.

Using ln p as a base-2 logarithm.

log₂ p = ln p/ln 2. Using ln alone gives nats rather than bits.

Directly evaluating log₂ 0 for p = 0.

The complete weighted zero term is defined as zero by its limit.

Exam context

Compare probability sources and verify the log₂ n bound.
Distinguish average information, codeword length and joint information of dependent sequences.

Formula cluster

Information and probability

Conditional probability describes relationships between events; entropy averages information across a whole distribution.

P(A|B) = P(A∩B)/P(B)

Conditional Probability

P(A|B) = P(B|A)·P(A) / P(B)

Bayes' Theorem

P(X=k) = (n über k)·pᵏ·(1−p)ⁿ⁻ᵏ

Binomial Distribution

Worked example

For a fair coin, H = −½log₂½ − ½log₂½ = 1 bit. For a coin with p(heads)=0.75 and p(tails)=0.25, H ≈ 0.811 bits per toss.

Applications

Understand lossless compression limits, compare information sources, split decision trees and quantify uncertainty.

How do you calculate with Shannon entropy?

Here is how to work through a typical Shannon entropy (H(X) = −Σ pᵢ log₂(pᵢ)) task step by step:

1
Task
Find H for probabilities (1/2, 1/4, 1/8, 1/8).
Solution path
1. Information values are 1, 2, 3, 3 bits.
2. Weighted: H = 1/2 + 2/4 + 3/8 + 3/8 = 1.75 bits.

Understand and apply the equation

Shannon entropy asks a concrete question: how much information does an outcome provide on average when its probability distribution is known? It averages the surprise of individual events. Original examples distinguish this quantity from the size of an actual file.

How do you calculate entropy for a complete discrete distribution?

Let X be a discrete random variable with finitely many possible outcomes x₁ through x_n. Each outcome has probability p_i = P(X = x_i). The probabilities are nonnegative and sum to one. Entropy depends on these probabilities; merely relabelling outcomes does not change it. For observed nonnegative counts $n_i$ with $N=\sum_i n_i>0$ , substitute $\hat p_i=\frac{n_i}{N}$ . The result describes the sample’s empirical distribution and is not automatically the exact source entropy or an unbiased estimate of it.[2][5]

H(X)=-\sum_{i=1}^n p_i\log_2 p_i

The formula takes the expected information per outcome.

A completely predictable output has zero entropy. That does not mean it is unimportant. Entropy measures statistical unpredictability under the chosen model, not the usefulness, truth or semantic meaning of a message.

Why does a base-two logarithm give bits?

I(x_i)=-\log_2p_i=\log_2\frac1{p_i}

A rare outcome has greater information content than a common one; p_i > 0.[1]

The logarithm matches the addition of independent information: independent probabilities multiply, while their negative logarithms add. The base determines the unit. Base 2 gives bits; the natural logarithm gives nats.

-\log_2(pq)=-\log_2p-\log_2q

Additivity for independent events with positive probabilities.

\mathbb E[I(X)]=\sum_i p_i I(x_i)=-\sum_i p_i\log_2p_i

Weight each information value by its probability and add all contributions.

\lim_{p\to0^+}p\log_2p=0

An impossible outcome contributes zero to the average. Set the whole term to zero rather than evaluating log₂(0).

Why uniform probabilities maximise entropy

For n possible outcomes, 0 ≤ H(X) ≤ log₂ n. A certain outcome gives H = 0. The upper bound is reached exactly when all n outcomes have probability 1/n.[3]

H=-\sum_{i=1}^n\frac1n\log_2\frac1n=\log_2 n

Under a uniform distribution every term is equal; n terms cancel the factor 1/n.

The lower bound follows directly from the terms: for 0 < p ≤ 1, log₂ p ≤ 0, so −p log₂ p ≥ 0. For the upper bound, use concavity of the logarithm. Jensen’s inequality gives the following calculation for m positive probabilities.

\sum_{p_i>0}p_i\log_2\frac1{p_i}\le\log_2\left(\sum_{p_i>0}p_i\frac1{p_i}\right)=\log_2m\le\log_2n

The weights p_i sum to one. Equal positive probabilities achieve equality in the first bound.

A fair binary source has maximum entropy

QUANTA STUDYquanta-study.de

For two outcomes with probabilities p and 1−p, maximum entropy is 1 bit. The curve describes the weighted mean of both information values. Endpoint values are limits; log₂(0) is not evaluated separately.[2][3]

Free with a Quanta account. Your download stays ready after registration.

Three examples: distribution, weighting and side information

Three source outputs

Given: A source outputs A with probability $\frac12$ , and B and C with probability $\frac14$ each.
Find: Entropy and an example of a matching prefix-free binary code.

Approach: The specified source has three discrete outcomes. Base-two logarithms give information in bits; probability weighting yields mean information and permits comparison with a prefix-free code.
$\begin{aligned}I_i&=-\log_2p_i\\{}H&=\sum_ip_iI_i\end{aligned}$
Check probabilities: $\frac12+\frac14+\frac14=1$ .
Information values are 1 bit for A and 2 bits for B and C.
$\begin{aligned}I(A)&=-\log_2\frac12=1\\{} I(B)&=I(C)=-\log_2\frac14=2\end{aligned}$
Add the weighted contributions.
$H=\frac12\cdot1+\frac14\cdot2+\frac14\cdot2=1.5\,\mathrm{bit}$

Result: H = 1.5 bits per output. Code A → 0, B → 10, C → 11 has the same average length.

Check: No codeword is a prefix of another. Average length is $\frac12\cdot1+\frac14\cdot2+\frac14\cdot2=1.5$ ; each individual output still uses an integer number of bits.

An uneven yes/no output

Given: P(yes) = 0.8, P(no) = 0.2.
Find: Find average information and compare it with a fair binary source.

Approach: The two complementary outcome probabilities define a Bernoulli source. Binary entropy weights both information values; a fair source provides an independent comparison at maximum binary uncertainty.
$H=-p\log_2p-(1-p)\log_2(1-p)$
Label yes as 1 and no as 0. Calculate a base-two logarithm for each outcome.
$\begin{aligned}I(1)&=-\log_2(0.8)\approx0.321928\\I(0)&=-\log_2(0.2)\approx2.321928\end{aligned}$
Do not take an unweighted average; multiply by 0.8 and 0.2.
$H\approx0.8\cdot0.321928+0.2\cdot2.321928=0.721928\,\mathrm{bit}$

Result: H ≈ 0.722 bits per output, below the maximum of 1 bit.

Check: An individual rare no contains more information but occurs less often. Swapping yes and no with their probabilities leaves H unchanged.

How much uncertainty does a clue remove?

Given: $X$ is uniform over four outcomes. Clue $Y$ reliably tells whether $X$ lies in $\{A,B\}$ or $\{C,D\}$ .
Find: Find original entropy, conditional entropy and mutual information.

Approach: four outcomes are possible before the clue. Either clue leaves two equally probable outcomes.
$H(X)=\log_2 4,\qquad H(X\mid Y)=\sum_yP(y)H(X\mid Y=y)$
Weight the remaining uncertainty for both clue values.
$H(X)=2\,\mathrm{bit},\qquad H(X\mid Y)=\tfrac12\cdot1+\tfrac12\cdot1=1\,\mathrm{bit}$
The mean reduction in uncertainty is mutual information.
$I(X;Y)=H(X)-H(X\mid Y)=1\,\mathrm{bit}$

Result: The clue provides $1\,\mathrm{bit}$ of information, leaving $1\,\mathrm{bit}$ of uncertainty about $X$ .

Check: $Y$ is itself a fair bit and is determined by $X$ . Thus $I(X;Y)=H(Y)=1\,\mathrm{bit}$ here, although the clue does not determine $X$ completely.

Entropy is not a universal file-size prediction

For lossless binary prefix codes of a discrete source, average codeword length is at least entropy. Suitable block coding of independent identically distributed outputs can approach H per output. Finite blocks, headers and communicating the model add overhead. Shannon–Hartley channel capacity instead concerns the achievable rate in a separate band-limited noisy-channel model; it is not another entropy formula for this source.[1]

Suppose a fair binary source independently produces 400 values. Their joint entropy is 400 bits. If one fair value is instead repeated 400 times, every position is individually fair, yet the whole sequence contains only 1 bit of random information. Dependence therefore changes the calculation fundamentally.

H(X_1,\ldots,X_k)=\sum_{j=1}^kH(X_j)

Individual entropies add directly only for independent outputs.

Five exercises with solutions

Calculate · Foundation

1. Eight outputs are equally likely. Find H.

Show hint

Use the uniform-distribution expression.

Learning goal: Calculate uniform entropy.

Show solution and calculation

Approach: $H=\log_2n$ .
H = log₂ 8 = 3 bits per output.

H=\log_2 8=3

Calculate · Application

2. Probabilities are $\left(\frac12,\frac14,\frac18,\frac18\right)$ . Calculate H.

Show hint

Weight information values 1, 2, 3, 3.

Learning goal: Calculate weighted distribution entropy.

Show solution and calculation

Approach: $H=-\sum_i p_i\log_2p_i$ .
H = 0.5 + 0.5 + 0.375 + 0.375 = 1.75 bits, below the four-outcome maximum of 2 bits.

H=\frac12+\frac24+\frac38+\frac38=1.75

Interpret · Application

3. One output has probability 1; all others have probability 0. What is the entropy?

Show hint

Use log₂ 1 = 0 and the zero-term convention.

Learning goal: Recognize zero entropy for a certain outcome.

Show solution and calculation

Approach: $H=-\sum_{p_i>0}p_i\log_2p_i$ .
H = 0.
The output is certain; zero probabilities contribute nothing under the limiting convention.

H=-1\log_2 1=0

Interpret · Application

A fair coin is tossed once. The source prints that result twice: HH or TT, each with probability one half. Find two-symbol block entropy and entropy per printed symbol. Is this equivalent to two independent tosses?

Show hint

Count possible blocks rather than doubling single-symbol entropy.

Learning goal: Account for dependence in block entropy.

Show solution and calculation

Approach: There are only two equally probable blocks, giving 1 bit per block and 0.5 bit per printed symbol.
Two independent tosses would produce four blocks and 2 bits.
Each symbol is marginally fair but the symbols are fully dependent.
Summing marginal entropies gives block entropy only under independence.

\begin{aligned}H(XX)&=-2\cdot\frac12\log_2\frac12=1\,\mathrm{bit}\\{}\frac{H(XX)}2&=0.5\,\mathrm{bit\ per\ symbol}\end{aligned}

[3]

Error analysis · Application

A binary source outputs A with probability 0.9 and B with 0.1. Someone calls the information of rare B the average entropy. Compute both and explain the error; use base-two logarithms.

Show hint

Entropy also weights the common event.

Learning goal: Distinguish event information from its expectation.

Show solution and calculation

Approach: Rare B provides about 3.32193 bits of information.
The probability-weighted mean across both events is instead about 0.468996 bits per output.
Common A provides little information and occurs nine times as often.
Entropy cannot exceed the binary maximum of 1 bit, immediately exposing the claimed equality.

\begin{aligned}I_B&=-\log_2(0.1)\approx3.32193\,\mathrm{bit}\\{}H&=-0.9\log_2(0.9)-0.1\log_2(0.1)\\{}&\approx0.468996\,\mathrm{bit}\end{aligned}

[1]

Avoid common calculation errors

Divide percentages by 100 first; 80% corresponds to p = 0.8.
Probabilities must describe a complete set of mutually exclusive outputs and sum to 1.
Use $\log_2 p=\frac{\ln p}{\ln2}$ on a calculator. Using ln alone gives nats.
Weight every logarithm by its matching p and keep the minus sign.
Sample frequencies are estimates, not automatically exact source probabilities.

Specify the complete discrete distribution and weight each information value by its probability. Base two gives bits. Entropy is an average; dependence, side information and actual coding require separate consideration.

Sources for this explanation

Quanta · Updated October 2, 2026

Editorial responsibility: Quanta Study editorial team

Review scope

This explanation, 3 worked examples and 5 exercises are available in German and English. Structure, source references and mathematical notation undergo automated checks. Calculations, model assumptions and both language versions receive a separate AI review.

An internal human subject review is documented for an earlier release of this page. That confirmation does not apply automatically to the revised explanation and new exercises.

An independent, named human subject review has not yet been documented. Our exercise numbers are worked examples, rather than measurements published by the cited sources.

Editorial responsibility, sources and methods

Change history

October 2, 2026Refined question headings and relevant paragraph links; added the full empirical-frequency substitution and a primary source on estimation limits. Preserved existing examples and exercises.
October 1, 2026Added two exercises; labelled all five by task type, difficulty and learning goal.
October 1, 2026Reviewed the learning sequence; added a third example, an original quantitative figure, symbolic exercise approaches and a short summary.

Cite this reference page

Quanta. Shannon entropy. Updated October 2, 2026. https://quanta-study.de/en/formula-sheet/shannon-entropie

When citing a specific scientific statement, also cite the specialist source linked to that statement.

Suggest a correction with a reference

Frequently asked questions about Shannon entropy

What does Shannon entropy measure?+

Shannon entropy measures the average information content of a discrete random variable under its specified probability distribution. For a positive probability, an individual outcome carries information $-\log_2p_i$ . Entropy weights each value by its probability and adds the contributions. Base two gives bits. A fair coin therefore has 1 bit per toss, while a certain outcome has 0 bits. This describes statistical unpredictability, not the meaning, truth or usefulness of a message. Entropy characterizes a source model; it does not directly specify the length of every individual text or particular file.

How do I calculate entropy from probabilities?+

First check that probabilities describe a complete set of mutually exclusive outcomes, are nonnegative and sum to one. Divide percentages by 100 before substitution. For each positive value, calculate information using a base-two logarithm, weight it by that same probability and sum. For $(\frac12,\frac14,\frac18,\frac18)$ , this gives $H=\frac12+\frac24+\frac38+\frac38=1.75$ bits. The upper check is $\log_2 4=2$ bits. A second check is that entropy must be nonnegative. Zero probabilities contribute zero; do not evaluate their logarithms separately. Round only the final result, and do not confuse an unweighted average with entropy.

When is entropy zero or maximal?+

For a finite discrete distribution, entropy is zero when exactly one outcome has probability one and all other outcomes are impossible. With n possible outcomes, entropy is at most $\log_2n$ bits; this maximum occurs when all n probabilities are equal. Four equally probable outcomes therefore give 2 bits. A binary source has a maximum of 1 bit at equal probabilities. An individual rare event can carry more than 1 bit of information, yet the weighted binary average remains at most 1 bit. These bounds do not transfer unchanged to differential entropy of continuous variables.

Why do bits and nats depend on the logarithm base?+

The logarithm base determines the information unit. Base two gives bits; the natural logarithm gives nats. Probabilities remain dimensionless and are unaffected by this choice. On a calculator without a log₂ function, use $\log_2p=\frac{\ln p}{\ln2}$ . Entropy in bits therefore equals entropy in nats divided by ln 2. A fair coin has 1 bit or approximately 0.693 nats. When comparing sources, keep both the base and distribution definitions consistent. In either unit, zero-probability terms are handled by the limit of the complete weighted term rather than by calculating a logarithm of zero.

How do dependence and side information change entropy?+

Individual output entropies add directly to joint entropy only under statistical independence. One fair coin toss printed twice produces two symbols but only 1 bit of joint uncertainty. Two independent fair tosses instead produce 2 bits. Side information is described by conditional entropy: $H(X\mid Y)$ averages remaining uncertainty over all possible observations of Y. If Y determines X, no uncertainty remains. If Y is independent, the original entropy remains. Their difference is mutual information, measuring statistical dependence or average information gain rather than causal direction. Thus counting printed symbols alone does not determine the information in an entire sequence.

Quanta exam set

Curated exam set for "Shannon entropy":

Question (front)

Which formula describes Shannon entropy?

Answer in your set

Question (front)

How do you rearrange Shannon entropy for Defining relation?

Answer in your set

Question (front)

Which common mistake happens with Shannon entropy?

Answer in your set

+ 7 more cards: units, variables, derivation, example, exam task

The formula set contains 10 cards. After importing, review them and start practicing; your ratings determine later review intervals.

Create flashcards or a quiz I already have an account

Subject references and further reading

This list names subject references and further reading. A page or section number appears when it is part of the reference; these are not verbatim quotations.

[1]MIT OpenCourseWare, 6.02 Introduction to EECS II: Digital Communication Systems, Chapter 2 “Information, Entropy, and the Motivation for Source Codes”.

Related formulas

D_{KL}(p\|q)=\sum_i p_i\ln\frac{p_i}{q_i}

Kullback-Leibler divergence (relative entropy)

More formulas: Information theory and coding

d_H(x,y)=\sum_{i=1}^{n}\mathbf{1}[x_i\ne y_i]

Hamming distance between bit strings

C=B\log_2(1+S/N)

Shannon-Hartley channel capacity

I(X;Y)=H(X)+H(Y)-H(X,Y)=H(X)-H(X|Y)

Mutual information of discrete variables

R(x)=(x^rM(x))\bmod G(x),\quad C(x)=x^rM(x)+R(x)

CRC checksum as a polynomial remainder over GF(2)

Spaced-repetition apps

Compare scheduling workflows

Study evidence

Read methods, results and limits

Compare flashcard apps

FSRS support and study features

Create a formula exam set

From Essential in Quanta

Retain Shannon entropy for exams

Create a curated FSRS exam set for H(X) = −Σ pᵢ log₂(pᵢ): formula recall, variables, derivation, rearrangement, worked example, common mistakes and exam context.

Create flashcards or a quiz Already registered? Continue directly

Curated formula set · LaTeX · FSRS spaced repetition · import from Essential

Quanta Study editorial team

Shannon entropy

Shannon entropy is the expected information of a discrete random experiment and measures uncertainty in its probability distribution in bits.

AdvancedBachelor foundationsSubject specialization

Free to browse · flashcards or a multiple-choice quiz · import from Essential

Symbol

Meaning

Unit

H(X)

Entropy of random variable X

bit

pᵢ

Probability of outcome i

dimensionless

n

Number of possible outcomes

count

Derivation & background

Exam blueprint

Validity range

Derivation steps

Entropy is the expected logarithmic information content.

1Specify discrete outcomes and probabilities $p_i\ge0$ with $\sum_i p_i=1$ .
2For $p_i>0$ , calculate information $I_i=-\log_2p_i$ . Base 2 gives bits.
3Weight each information value by its probability: $p_iI_i$ .
4Add the contributions: $H(X)=\sum_i p_iI_i=-\sum_i p_i\log_2p_i$ .
5For $p_i=0$ , set the whole contribution to zero by its limit; do not evaluate $\log_2(0)$ separately.
6Check $0\le H\le\log_2n$ ; the maximum occurs for n equally likely outcomes.

How do you calculate entropy for a complete discrete distribution?

H(X)=-\sum_{i=1}^n p_i\log_2 p_i

The formula takes the expected information per outcome.

Why does a base-two logarithm give bits?

I(x_i)=-\log_2p_i=\log_2\frac1{p_i}

A rare outcome has greater information content than a common one; p_i > 0.[1]

-\log_2(pq)=-\log_2p-\log_2q

Additivity for independent events with positive probabilities.

\mathbb E[I(X)]=\sum_i p_i I(x_i)=-\sum_i p_i\log_2p_i

Weight each information value by its probability and add all contributions.

\lim_{p\to0^+}p\log_2p=0

An impossible outcome contributes zero to the average. Set the whole term to zero rather than evaluating log₂(0).

Why uniform probabilities maximise entropy

For n possible outcomes, 0 ≤ H(X) ≤ log₂ n. A certain outcome gives H = 0. The upper bound is reached exactly when all n outcomes have probability 1/n.[3]

H=-\sum_{i=1}^n\frac1n\log_2\frac1n=\log_2 n

Under a uniform distribution every term is equal; n terms cancel the factor 1/n.

\sum_{p_i>0}p_i\log_2\frac1{p_i}\le\log_2\left(\sum_{p_i>0}p_i\frac1{p_i}\right)=\log_2m\le\log_2n

The weights p_i sum to one. Equal positive probabilities achieve equality in the first bound.

A fair binary source has maximum entropy

QUANTA STUDYquanta-study.de

Free with a Quanta account. Your download stays ready after registration.

Three examples: distribution, weighting and side information

Three source outputs

Given: A source outputs A with probability $\frac12$ , and B and C with probability $\frac14$ each.
Find: Entropy and an example of a matching prefix-free binary code.

Approach: The specified source has three discrete outcomes. Base-two logarithms give information in bits; probability weighting yields mean information and permits comparison with a prefix-free code.
$\begin{aligned}I_i&=-\log_2p_i\\{}H&=\sum_ip_iI_i\end{aligned}$
Check probabilities: $\frac12+\frac14+\frac14=1$ .
Information values are 1 bit for A and 2 bits for B and C.
$\begin{aligned}I(A)&=-\log_2\frac12=1\\{} I(B)&=I(C)=-\log_2\frac14=2\end{aligned}$
Add the weighted contributions.
$H=\frac12\cdot1+\frac14\cdot2+\frac14\cdot2=1.5\,\mathrm{bit}$

Result: H = 1.5 bits per output. Code A → 0, B → 10, C → 11 has the same average length.

Check: No codeword is a prefix of another. Average length is $\frac12\cdot1+\frac14\cdot2+\frac14\cdot2=1.5$ ; each individual output still uses an integer number of bits.

An uneven yes/no output

Given: P(yes) = 0.8, P(no) = 0.2.
Find: Find average information and compare it with a fair binary source.

Approach: The two complementary outcome probabilities define a Bernoulli source. Binary entropy weights both information values; a fair source provides an independent comparison at maximum binary uncertainty.
$H=-p\log_2p-(1-p)\log_2(1-p)$
Label yes as 1 and no as 0. Calculate a base-two logarithm for each outcome.
$\begin{aligned}I(1)&=-\log_2(0.8)\approx0.321928\\I(0)&=-\log_2(0.2)\approx2.321928\end{aligned}$
Do not take an unweighted average; multiply by 0.8 and 0.2.
$H\approx0.8\cdot0.321928+0.2\cdot2.321928=0.721928\,\mathrm{bit}$

Result: H ≈ 0.722 bits per output, below the maximum of 1 bit.

Check: An individual rare no contains more information but occurs less often. Swapping yes and no with their probabilities leaves H unchanged.

How much uncertainty does a clue remove?

Given: $X$ is uniform over four outcomes. Clue $Y$ reliably tells whether $X$ lies in $\{A,B\}$ or $\{C,D\}$ .
Find: Find original entropy, conditional entropy and mutual information.

Approach: four outcomes are possible before the clue. Either clue leaves two equally probable outcomes.
$H(X)=\log_2 4,\qquad H(X\mid Y)=\sum_yP(y)H(X\mid Y=y)$
Weight the remaining uncertainty for both clue values.
$H(X)=2\,\mathrm{bit},\qquad H(X\mid Y)=\tfrac12\cdot1+\tfrac12\cdot1=1\,\mathrm{bit}$
The mean reduction in uncertainty is mutual information.
$I(X;Y)=H(X)-H(X\mid Y)=1\,\mathrm{bit}$

Result: The clue provides $1\,\mathrm{bit}$ of information, leaving $1\,\mathrm{bit}$ of uncertainty about $X$ .

Check: $Y$ is itself a fair bit and is determined by $X$ . Thus $I(X;Y)=H(Y)=1\,\mathrm{bit}$ here, although the clue does not determine $X$ completely.

Entropy is not a universal file-size prediction

H(X_1,\ldots,X_k)=\sum_{j=1}^kH(X_j)

Individual entropies add directly only for independent outputs.

Five exercises with solutions

Calculate · Foundation

1. Eight outputs are equally likely. Find H.

Show hint

Use the uniform-distribution expression.

Learning goal: Calculate uniform entropy.

Show solution and calculation

Approach: $H=\log_2n$ .
H = log₂ 8 = 3 bits per output.

H=\log_2 8=3

Calculate · Application

2. Probabilities are $\left(\frac12,\frac14,\frac18,\frac18\right)$ . Calculate H.

Show hint

Weight information values 1, 2, 3, 3.

Learning goal: Calculate weighted distribution entropy.

Show solution and calculation

Approach: $H=-\sum_i p_i\log_2p_i$ .
H = 0.5 + 0.5 + 0.375 + 0.375 = 1.75 bits, below the four-outcome maximum of 2 bits.

H=\frac12+\frac24+\frac38+\frac38=1.75

Interpret · Application

3. One output has probability 1; all others have probability 0. What is the entropy?

Show hint

Use log₂ 1 = 0 and the zero-term convention.

Learning goal: Recognize zero entropy for a certain outcome.

Show solution and calculation

Approach: $H=-\sum_{p_i>0}p_i\log_2p_i$ .
H = 0.
The output is certain; zero probabilities contribute nothing under the limiting convention.

H=-1\log_2 1=0

Interpret · Application

A fair coin is tossed once. The source prints that result twice: HH or TT, each with probability one half. Find two-symbol block entropy and entropy per printed symbol. Is this equivalent to two independent tosses?

Show hint

Count possible blocks rather than doubling single-symbol entropy.

Learning goal: Account for dependence in block entropy.

Show solution and calculation

Approach: There are only two equally probable blocks, giving 1 bit per block and 0.5 bit per printed symbol.
Two independent tosses would produce four blocks and 2 bits.
Each symbol is marginally fair but the symbols are fully dependent.
Summing marginal entropies gives block entropy only under independence.

\begin{aligned}H(XX)&=-2\cdot\frac12\log_2\frac12=1\,\mathrm{bit}\\{}\frac{H(XX)}2&=0.5\,\mathrm{bit\ per\ symbol}\end{aligned}

[3]

Error analysis · Application

A binary source outputs A with probability 0.9 and B with 0.1. Someone calls the information of rare B the average entropy. Compute both and explain the error; use base-two logarithms.

Show hint

Entropy also weights the common event.

Learning goal: Distinguish event information from its expectation.

Show solution and calculation

Approach: Rare B provides about 3.32193 bits of information.
The probability-weighted mean across both events is instead about 0.468996 bits per output.
Common A provides little information and occurs nine times as often.
Entropy cannot exceed the binary maximum of 1 bit, immediately exposing the claimed equality.

\begin{aligned}I_B&=-\log_2(0.1)\approx3.32193\,\mathrm{bit}\\{}H&=-0.9\log_2(0.9)-0.1\log_2(0.1)\\{}&\approx0.468996\,\mathrm{bit}\end{aligned}

[1]

Avoid common calculation errors

Divide percentages by 100 first; 80% corresponds to p = 0.8.
Probabilities must describe a complete set of mutually exclusive outputs and sum to 1.
Use $\log_2 p=\frac{\ln p}{\ln2}$ on a calculator. Using ln alone gives nats.
Weight every logarithm by its matching p and keep the minus sign.
Sample frequencies are estimates, not automatically exact source probabilities.

Sources for this explanation

Quanta · Updated October 2, 2026

Editorial responsibility: Quanta Study editorial team

Review scope

An internal human subject review is documented for an earlier release of this page. That confirmation does not apply automatically to the revised explanation and new exercises.

An independent, named human subject review has not yet been documented. Our exercise numbers are worked examples, rather than measurements published by the cited sources.

Editorial responsibility, sources and methods

Change history

October 2, 2026Refined question headings and relevant paragraph links; added the full empirical-frequency substitution and a primary source on estimation limits. Preserved existing examples and exercises.
October 1, 2026Added two exercises; labelled all five by task type, difficulty and learning goal.
October 1, 2026Reviewed the learning sequence; added a third example, an original quantitative figure, symbolic exercise approaches and a short summary.

Cite this reference page

Quanta. Shannon entropy. Updated October 2, 2026. https://quanta-study.de/en/formula-sheet/shannon-entropie

When citing a specific scientific statement, also cite the specialist source linked to that statement.

Suggest a correction with a reference

Frequently asked questions about Shannon entropy

What does Shannon entropy measure?+

How do I calculate entropy from probabilities?+

When is entropy zero or maximal?+

Why do bits and nats depend on the logarithm base?+

How do dependence and side information change entropy?+

Retain Shannon entropy for exams

Create a curated FSRS exam set for H(X) = −Σ pᵢ log₂(pᵢ): formula recall, variables, derivation, rearrangement, worked example, common mistakes and exam context.

Curated formula set · LaTeX · FSRS spaced repetition · import from Essential

Shannon entropy

Variables & units

Shannon entropy calculator

Quick tips

Derivation & background

Exam blueprint

Validity range

Derivation steps

Rearrangements

Task variant

Common mistakes

Exam context

Information and probability

Worked example

Applications

How do you calculate with Shannon entropy?

How do you calculate entropy for a complete discrete distribution?

Why does a base-two logarithm give bits?

Why uniform probabilities maximise entropy

A fair binary source has maximum entropy

Three examples: distribution, weighting and side information

Three source outputs

An uneven yes/no output

How much uncertainty does a clue remove?

Entropy is not a universal file-size prediction

Distinguish conditional entropy and continuous quantities

Five exercises with solutions

1. Eight outputs are equally likely. Find H.

2. Probabilities are (12,14,18,18)\left(\frac12,\frac14,\frac18,\frac18\right)(21​,41​,81​,81​). Calculate H.

3. One output has probability 1; all others have probability 0. What is the entropy?

A fair coin is tossed once. The source prints that result twice: HH or TT, each with probability one half. Find two-symbol block entropy and entropy per printed symbol. Is this equivalent to two independent tosses?

A binary source outputs A with probability 0.9 and B with 0.1. Someone calls the information of rare B the average entropy. Compute both and explain the error; use base-two logarithms.

Avoid common calculation errors

Sources for this explanation

Frequently asked questions about Shannon entropy

Subject references and further reading

Related formulas

More formulas: Information theory and coding

Retain Shannon entropy for exams

Shannon entropy

Variables & units

Shannon entropy calculator

Quick tips

Derivation & background

Exam blueprint

Validity range

Derivation steps

Rearrangements

Task variant

Common mistakes

Exam context

Information and probability

Worked example

Applications

How do you calculate with Shannon entropy?

How do you calculate entropy for a complete discrete distribution?

Why does a base-two logarithm give bits?

Why uniform probabilities maximise entropy

A fair binary source has maximum entropy

Three examples: distribution, weighting and side information

Three source outputs

An uneven yes/no output

How much uncertainty does a clue remove?

Entropy is not a universal file-size prediction

Distinguish conditional entropy and continuous quantities

Five exercises with solutions

1. Eight outputs are equally likely. Find H.

2. Probabilities are (12,14,18,18)\left(\frac12,\frac14,\frac18,\frac18\right)(21​,41​,81​,81​). Calculate H.

3. One output has probability 1; all others have probability 0. What is the entropy?

A fair coin is tossed once. The source prints that result twice: HH or TT, each with probability one half. Find two-symbol block entropy and entropy per printed symbol. Is this equivalent to two independent tosses?

A binary source outputs A with probability 0.9 and B with 0.1. Someone calls the information of rare B the average entropy. Compute both and explain the error; use base-two logarithms.

Avoid common calculation errors

Sources for this explanation

Frequently asked questions about Shannon entropy

Subject references and further reading

Related formulas

More formulas: Information theory and coding

Retain Shannon entropy for exams

2. Probabilities are $\left(\frac12,\frac14,\frac18,\frac18\right)$ . Calculate H.

2. Probabilities are $\left(\frac12,\frac14,\frac18,\frac18\right)$ . Calculate H.