Opening your learning space
Starting the app …
Opening your learning space
Starting the app …
Information gain scores a candidate split at a tree node by the reduction in label entropy after partitioning the training examples.
Free to browse · flashcards or a multiple-choice quiz · import from Essential
Formula
Entropy and IG in bits with base-2 logarithms; probabilities dimensionless
| Symbol | Meaning | Unit |
|---|---|---|
| Class label of examples at the current node | discrete random variable | |
| Candidate split with child outcomes v | discrete partition | |
| Label entropy before the split | bit | |
| Fraction of samples in child v | dimensionless | |
| H(Y|S=v) | Label entropy within child v | bit |
IG(Y;S)=H(Y)-\sum_{v\in S}P(v)H(Y|S=v)A split has information gain H(Y)−H(Y|S), mathematically the mutual information between label Y and split outcome S in the empirical node distribution. A decision tree uses it as a local greedy selection rule; it is not causality or general predictive performance. For multiple children, weight each child entropy by its sample fraction. Small leaves can appear to have high gain; minimum leaf sizes, validation or regularization help limit overfitting. Base-2 logarithms give bits.
A split has information gain H(Y)−H(Y|S), mathematically the mutual information between label Y and split outcome S in the empirical node distribution. A decision tree uses it as a local greedy selection rule; it is not causality or general predictive performance. For multiple children, weight each child entropy by its sample fraction. Small leaves can appear to have high gain; minimum leaf sizes, validation or regularization help limit overfitting. Base-2 logarithms give bits.
Information gain scores a candidate split at a tree node by the reduction in label entropy after partitioning the training examples.
Conditional entropy from information gain
Use the same node sample and base-2 logarithms.
Use the example for Information gain of a decision-tree split to explain which inputs determine the requested output.
The example connects concrete inputs to its result.
Information gain scores a candidate split at a tree node by the reduction in label entropy after partitioning the training examples.
The relation applies only under its stated assumptions.
A parent has 6 positive and 4 negative cases. A split yields 4 positives on the left and 2 positives plus 4 negatives on the right. Calculate IG.
Hparent≈0.971 bits; Hleft=0; Hright≈0.918 bits.
Hchildren=0.4·0+0.6·0.918≈0.551 bits, hence IG≈0.420 bits.
Check: child counts total 6 positives and 4 negatives.
Applying the relation to a different case without checking its stated assumptions.
Check the validity range before substituting values: A split has information gain H(Y)−H(Y|S), mathematically the mutual information between label Y and split outcome S in the empirical node distribution. A decision tree uses it as a local.
Formula cluster
Information gain scores a candidate split at a tree node by the reduction in label entropy after partitioning the training examples.
The parent has 6 positive and 4 negative cases: H(Y)≈0.971 bits. A split creates a pure left child of 4 positives and a right child with 2 positives and 4 negatives (entropy≈0.918). Weighted child entropy is 0.4·0+0.6·0.918≈0.551, so IG≈0.420 bits. Check: the children together still contain 6 positives and 4 negatives.
Compare candidate partitions in classification trees and quantify reduction in label uncertainty.
Here is how to work through a typical Information gain of a decision-tree split (IG = H(Y) − gewichtete Kinderentropie) task step by step:
Task
Use the example for Information gain of a decision-tree split to explain which inputs determine the requested output.
Solution path
The example connects concrete inputs to its result.
Information gain scores a candidate split at a tree node by the reduction in label entropy after partitioning the training examples.
The relation applies only under its stated assumptions.
Task
A parent has 6 positive and 4 negative cases. A split yields 4 positives on the left and 2 positives plus 4 negatives on the right. Calculate IG.
Solution path
Hparent≈0.971 bits; Hleft=0; Hright≈0.918 bits.
Hchildren=0.4·0+0.6·0.918≈0.551 bits, hence IG≈0.420 bits.
Check: child counts total 6 positives and 4 negatives.
It compares label entropy before a split with the expected entropy in the child nodes afterward. Each child entropy is weighted by its share of examples. A higher value means that this split removes more label uncertainty in the current sample. The statement applies to data at this node; it proves neither causation nor better performance on new data. For a particular node and partition, IG is the reduction in label entropy; mathematically it is mutual information between the label and split outcome. It is a local greedy training rule, not a causal measure. Small leaves and many tested splits can inflate empirical gain, so check minimum leaf sizes and validation.
For a fixed label Y and variable S identifying the selected child, IG=H(Y)−H(Y|S)=I(Y;S). Thus information gain is mathematically mutual information. The term here emphasizes its use as a score for one tree split. General mutual information can study arbitrary variables and distributions; neither quantity establishes causal direction. For a particular node and partition, IG is the reduction in label entropy; mathematically it is mutual information between the label and split outcome. It is a local greedy training rule, not a causal measure. Small leaves and many tested splits can inflate empirical gain, so check minimum leaf sizes and validation.
A child’s size determines how often its conditional label uncertainty occurs within the parent node. A child containing 90% of cases must therefore weigh more than one containing 10%. Without this weighting, a tiny pure leaf would count as much as a large mixed leaf. The weights sum to one when every parent case is assigned to exactly one child. For a particular node and partition, IG is the reduction in label entropy; mathematically it is mutual information between the label and split outcome. It is a local greedy training rule, not a causal measure. Small leaves and many tested splits can inflate empirical gain, so check minimum leaf sizes and validation.
The parent has 6 positive and 4 negative cases, so H≈0.971 bits. The left child has 4 positive cases and entropy 0; the right has 2 positive and 4 negative cases, with H≈0.918 bits. Weighting gives 0.4·0+0.6·0.918≈0.551 bits and IG≈0.420 bits. Child counts add back to 6 positives and 4 negatives. For a particular node and partition, IG is the reduction in label entropy; mathematically it is mutual information between the label and split outcome. It is a local greedy training rule, not a causal measure. Small leaves and many tested splits can inflate empirical gain, so check minimum leaf sizes and validation.
A tree may test many candidate features and thresholds. Small leaves make random patterns easier to separate perfectly, creating high training gain that may not reproduce. Minimum leaf sizes, depth limits, pruning and a separate validation set help. Compare splits under the same rules and evaluate the fitted model outside its training sample. For a particular node and partition, IG is the reduction in label entropy; mathematically it is mutual information between the label and split outcome. It is a local greedy training rule, not a causal measure. Small leaves and many tested splits can inflate empirical gain, so check minimum leaf sizes and validation.
Quanta exam set
Curated exam set for "Information gain of a decision-tree split":
Question (front)
Which formula describes Information gain of a decision-tree split?
Answer in your set
Question (front)
How do you rearrange Information gain of a decision-tree split for Conditional entropy from information gain?
Answer in your set
Question (front)
Which common mistake happens with Information gain of a decision-tree split?
Answer in your set
+ 8 more cards: units, variables, derivation, example, exam task
The formula set contains 11 cards. After importing, review them and start practicing; your ratings determine later review intervals.
This list names subject references and further reading. A page or section number appears when it is part of the reference; these are not verbatim quotations.
Create a curated FSRS exam set for IG = H(Y) − gewichtete Kinderentropie: formula recall, variables, derivation, rearrangement, worked example, common mistakes and exam context.
Curated formula set · LaTeX · FSRS spaced repetition · import from Essential