Opening your learning space
Starting the app …
Opening your learning space
Starting the app …
Mutual information measures the average reduction in uncertainty about X when Y is observed.
Free to browse · flashcards or a multiple-choice quiz · import from Essential
Formula
bits with base-2 logarithms; nonnegative for discrete variables
| Symbol | Meaning | Unit |
|---|---|---|
| I(X;Y) | Mutual information between X and Y | bit |
| H(X), H(Y) | Marginal entropies | bit |
| Joint entropy | bit | |
| H(X|Y) | Conditional entropy of X given Y | bit |
I(X;Y)=H(X)+H(Y)-H(X,Y)=H(X)-H(X|Y)For discrete random variables, I(X;Y)=Σx,ᵧ p(x,y)log2[p(x,y)/(p(x)p(y))]. Entropy identities are equivalent forms based on the same joint distribution. Mutual information is nonnegative and is zero exactly for independent variables. For identical variables I(X;X)=H(X). It measures statistical dependence, but does not encode its direction or establish causal influence.
For discrete random variables, I(X;Y)=Σx,ᵧ p(x,y)log2[p(x,y)/(p(x)p(y))]. Entropy identities are equivalent forms based on the same joint distribution. Mutual information is nonnegative and is zero exactly for independent variables. For identical variables I(X;X)=H(X). It measures statistical dependence, but does not encode its direction or establish causal influence.
Mutual information measures the average reduction in uncertainty about X when Y is observed.
Conditional entropy
Use one logarithm base and joint distribution consistently.
Use the example for Mutual information of discrete variables to explain which inputs determine the requested output.
The example connects concrete inputs to its result.
Mutual information measures the average reduction in uncertainty about X when Y is observed.
The relation applies only under its stated assumptions.
Applying the relation to a different case without checking its stated assumptions.
Check the validity range before substituting values: For discrete random variables, I(X;Y)=Σx,ᵧ p(x,y)log2[p(x,y)/(p(x)p(y))]. Entropy identities are equivalent forms based on the same joint distribution. Mutual information is nonnegative and is zero exactly for independent variables. For identical.
Formula cluster
Mutual information measures the average reduction in uncertainty about X when Y is observed.
If X and Y are identical fair bits, H(X)=H(Y)=H(X,Y)=1 bit and I=1+1−1=1 bit. Independent fair bits give I=0.
Assess dependencies in discrete data, feature selection and information gain.
Here is how to work through a typical Mutual information of discrete variables (I(X;Y) = H(X) − H(X|Y)) task step by step:
Task
Use the example for Mutual information of discrete variables to explain which inputs determine the requested output.
Solution path
The example connects concrete inputs to its result.
Mutual information measures the average reduction in uncertainty about X when Y is observed.
The relation applies only under its stated assumptions.
I(X;Y) measures how much observing Y tells us about X on average. Equivalently, it is uncertainty reduction H(X)−H(X|Y). If X and Y are independent, Y provides no information about X and I=0. The measure is symmetric, I(X;Y)=I(Y;X), and nonnegative for discrete variables. These identities apply to discrete variables and probabilities from the same joint distribution. Mutual information measures statistical dependence but does not prove causation. With empirical data, estimates depend on sample size and discretization; document these choices when comparing results. With unknown distributions, MI is estimated; small samples and discretization can bias the value.
Compute H(X), H(Y) and joint entropy H(X,Y) from the same joint table. Then I=H(X)+H(Y)−H(X,Y). For identical fair bits, all three entropies are one bit, so I=1+1−1=1 bit. Use one logarithm base consistently; base two gives bits. These identities apply to discrete variables and probabilities from the same joint distribution. Mutual information measures statistical dependence but does not prove causation. With empirical data, estimates depend on sample size and discretization; document these choices when comparing results. With unknown distributions, MI is estimated; small samples and discretization can bias the value.
For discrete variables, mutual information is zero exactly when X and Y are statistically independent. Knowing one value then does not change the distribution of the other. A finite-sample estimator can still be nonzero. Distinguish theoretical independence from an estimate based on observed data. These identities apply to discrete variables and probabilities from the same joint distribution. Mutual information measures statistical dependence but does not prove causation. With empirical data, estimates depend on sample size and discretization; document these choices when comparing results. With unknown distributions, MI is estimated; small samples and discretization can bias the value.
No. Although I(X;Y) can be read as information about X gained from Y, I(X;Y)=I(Y;X). It describes dependence without an effect direction. Directed questions need conditional measures, temporal models or causal design. A large MI alone does not establish a cause. These identities apply to discrete variables and probabilities from the same joint distribution. Mutual information measures statistical dependence but does not prove causation. With empirical data, estimates depend on sample size and discretization; document these choices when comparing results. With unknown distributions, MI is estimated; small samples and discretization can bias the value.
Shannon entropy H(X) measures uncertainty in one distribution. Mutual information compares joint and independent distributions to quantify what two variables share. For identical discrete variables, I(X;X)=H(X); for independent variables, MI is zero. Its maximum is bounded by the entropy of the variables involved. These identities apply to discrete variables and probabilities from the same joint distribution. Mutual information measures statistical dependence but does not prove causation. With empirical data, estimates depend on sample size and discretization; document these choices when comparing results. With unknown distributions, MI is estimated; small samples and discretization can bias the value.
Quanta exam set
Curated exam set for "Mutual information of discrete variables":
Question (front)
Which formula describes Mutual information of discrete variables?
Answer in your set
Question (front)
How do you rearrange Mutual information of discrete variables for Conditional entropy?
Answer in your set
Question (front)
Which common mistake happens with Mutual information of discrete variables?
Answer in your set
+ 7 more cards: units, variables, derivation, example, exam task
The formula set contains 10 cards. After importing, review them and start practicing; your ratings determine later review intervals.
This list names subject references and further reading. A page or section number appears when it is part of the reference; these are not verbatim quotations.
Create a curated FSRS exam set for I(X;Y) = H(X) − H(X|Y): formula recall, variables, derivation, rearrangement, worked example, common mistakes and exam context.
Curated formula set · LaTeX · FSRS spaced repetition · import from Essential