Computer Science & Data Science · Machine learning

Cross-entropy (classification loss)

Cross-entropy scores how well a predicted probability distribution q matches a target distribution p over classes.

AdvancedBachelor foundationsSubject specialization

Create flashcards or a quiz I already have an account

Free to browse · flashcards or a multiple-choice quiz · import from Essential

Formula

H (p, q) = - i = 1 \sum K p_{i} ln q_{i}

Nats (the natural logarithm ln is used, as in scikit-learn log_loss)

Variables & units

Variables & units
Symbol	Meaning	Unit
$pᵢ$	Target probability for class i	dimensionless
$qᵢ$	Model-predicted probability for class i	dimensionless
$K$	Number of classes	count

LaTeX

H(p,q)=-\sum_{i=1}^{K}p_i\ln q_i

QUANTA STUDYquanta-study.de

Derivation & background

For one-hot labels, only the true class contributes, so the loss is −ln of its predicted probability. Confidently assigning a small probability to the true class produces a large penalty. If p_i>0 while q_i=0, the ideal loss is infinite; software typically clips probabilities for numerical stability. This entry consistently uses the natural logarithm, so the information unit is the nat. Cross-entropy differs from Shannon entropy because q represents the model prediction.

Exam blueprint

Validity range

Derivation steps

Cross-entropy scores how well a predicted probability distribution q matches a target distribution p over classes.

1Read the relation from its definition: H(p,q)=-\sum_{i=1}^{K}p_i\ln q_i.
2Identify the input, requested output and assumptions of the relation.
3If the true class receives probability 0.8, natural-log loss is −ln(0.8)≈0.223 nats; at probability 0.1 it rises to about 2.303 nats.

Rearrangements

Defining relation

H(p,q)=-\sum_{i=1}^{K}p_i\ln q_i

The equation states the defining relation within its stated domain.

Task variant

Use the example for Cross-entropy (classification loss) to explain which inputs determine the requested output.

The example connects concrete inputs to its result.
Cross-entropy scores how well a predicted probability distribution q matches a target distribution p over classes.
The relation applies only under its stated assumptions.

Common mistakes

Applying the relation to a different case without checking its stated assumptions.

Check the validity range before substituting values: For one-hot labels, only the true class contributes, so the loss is −ln of its predicted probability. Confidently assigning a small probability to the true class produces a large penalty..

Exam context

Train probabilistic classifiers and assess predicted distributions with a proper scoring loss.

Formula cluster

Machine learning

Cross-entropy scores how well a predicted probability distribution q matches a target distribution p over classes.

H(X) = −Σ p_i log₂(p_i)

Shannon entropy

F₁ = 2TP / (2TP + FP + FN)

F₁ score

θ_t+1 = θ_t − η∇J(θ_t)

Gradient descent parameter update

Worked example

If the true class receives probability 0.8, natural-log loss is −ln(0.8)≈0.223 nats; at probability 0.1 it rises to about 2.303 nats.

Applications

Train probabilistic classifiers and assess predicted distributions with a proper scoring loss.

How do you calculate with Cross-entropy (classification loss)?

Here is how to work through a typical Cross-entropy (classification loss) (H(p,q) = −Σ pᵢ ln(qᵢ)) task step by step:

1
Task
Use the example for Cross-entropy (classification loss) to explain which inputs determine the requested output.
Solution path
1. The example connects concrete inputs to its result.
2. Cross-entropy scores how well a predicted probability distribution q matches a target distribution p over classes.
3. The relation applies only under its stated assumptions.

Frequently asked questions about Cross-entropy (classification loss)

What does Cross-entropy (classification loss) describe?+

The relation H(p,q)=-\sum_{i=1}^{K}p_i\ln q_i summarizes Cross-entropy scores how well a predicted probability distribution q matches a target distribution p over classes. Read it as a model for defined quantities, not as an automatic claim about every real situation. In the example If the true class receives probability 0.8, natural-log loss is −ln(0.8)≈0.223 nats; at probability 0.1 it rises to about 2.303 nats., symbols are connected to concrete values and the result is interpreted in context. Validity limit: For one-hot labels, only the true class contributes, so the loss is −ln of its predicted probability. Confidently assigning a small probability to the true class produces a large penalty. Source: scikit-learn, `log_loss`: binary log loss and cross-entropy definition.

How should I work through a calculation?+

First write down the requested quantity and all given values with their symbols. Substitute them into the defining relation and check unit consistency. The example If the true class receives probability 0.8, natural-log loss is −ln(0.8)≈0.223 nats; at probability 0.1 it rises to about 2.303 nats. provides a worked check: formula and result must have compatible scale and meaning; round only at the end. Validity limit: For one-hot labels, only the true class contributes, so the loss is −ln of its predicted probability. Confidently assigning a small probability to the true class produces a large penalty. Source: scikit-learn, `log_loss`: binary log loss and cross-entropy definition.

Which assumption must I keep in mind?+

The formula applies only when quantities are defined as in the source and background. Check boundary cases, measurement conditions and whether an approximation rather than equality is intended. Substitution can be algebraically correct yet technically wrong when assumptions fail or the population or system boundary changes. Validity limit: For one-hot labels, only the true class contributes, so the loss is −ln of its predicted probability. Confidently assigning a small probability to the true class produces a large penalty. Source: scikit-learn, `log_loss`: binary log loss and cross-entropy definition.

Which units and quantities are involved?+

The stated units are: Nats (the natural logarithm ln is used, as in scikit-learn log_loss). Convert values to a consistent unit system before substitution; counts, percentages and dimensionless ratios are not interchangeable. The variable list states the role of each symbol. Dimensional analysis catches many errors but does not replace checking the formula's assumptions. Validity limit: For one-hot labels, only the true class contributes, so the loss is −ln of its predicted probability. Confidently assigning a small probability to the true class produces a large penalty. Source: scikit-learn, `log_loss`: binary log loss and cross-entropy definition.

When is comparison with shannon-entropie and f1-score useful?+

Comparison helps when both quantities address the same question but emphasize different properties. Report the definition, input data, threshold or averaging with the value. The source scikit-learn, `log_loss`: binary log loss and cross-entropy definition. supports the definition used here; other conventions can differ. Choose the measure to match the learning or analysis goal. Validity limit: For one-hot labels, only the true class contributes, so the loss is −ln of its predicted probability. Confidently assigning a small probability to the true class produces a large penalty. Source: scikit-learn, `log_loss`: binary log loss and cross-entropy definition.

Quanta exam set

Curated exam set for "Cross-entropy (classification loss)":

Question (front)

Which formula describes Cross-entropy (classification loss)?

Answer in your set

Question (front)

How do you rearrange Cross-entropy (classification loss) for Defining relation?

Answer in your set

Question (front)

Which common mistake happens with Cross-entropy (classification loss)?

Answer in your set

+ 7 more cards: units, variables, derivation, example, exam task

The formula set contains 10 cards. After importing, review them and start practicing; your ratings determine later review intervals.

Create flashcards or a quiz I already have an account

Subject references and further reading

This list names subject references and further reading. A page or section number appears when it is part of the reference; these are not verbatim quotations.

[1]scikit-learn, `log_loss`: binary log loss and cross-entropy definition.

More formulas: Data analysis and machine learning

\mathrm{BA}=\frac{1}{2}\left(\frac{TP}{TP+FN}+\frac{TN}{TN+FP}\right)

Balanced accuracy

\mathrm{Precision}=\frac{TP}{TP+FP}

Precision (positive predictive value)

\mathrm{Recall}=\frac{TP}{TP+FN}

Recall (sensitivity)

\mathrm{MCC}=\frac{TP\,TN-FP\,FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}

Matthews correlation coefficient (MCC)

Spaced-repetition apps

Compare scheduling workflows

Study evidence

Read methods, results and limits

Compare flashcard apps

FSRS support and study features

Create a formula exam set

From Essential in Quanta

Retain Cross-entropy (classification loss) for exams

Create a curated FSRS exam set for H(p,q) = −Σ pᵢ ln(qᵢ): formula recall, variables, derivation, rearrangement, worked example, common mistakes and exam context.

Create flashcards or a quiz Already registered? Continue directly

Curated formula set · LaTeX · FSRS spaced repetition · import from Essential

Quanta Study editorial team

Symbol

Meaning

Unit

pᵢ

Target probability for class i

dimensionless

qᵢ

Model-predicted probability for class i

dimensionless

K

Number of classes

count

Derivation & background

Exam blueprint

Validity range

Derivation steps

Cross-entropy scores how well a predicted probability distribution q matches a target distribution p over classes.

1Read the relation from its definition: H(p,q)=-\sum_{i=1}^{K}p_i\ln q_i.
2Identify the input, requested output and assumptions of the relation.
3If the true class receives probability 0.8, natural-log loss is −ln(0.8)≈0.223 nats; at probability 0.1 it rises to about 2.303 nats.

How do you calculate with Cross-entropy (classification loss)?

Here is how to work through a typical Cross-entropy (classification loss) (H(p,q) = −Σ pᵢ ln(qᵢ)) task step by step:

Task

Use the example for Cross-entropy (classification loss) to explain which inputs determine the requested output.

Solution path

The example connects concrete inputs to its result.
Cross-entropy scores how well a predicted probability distribution q matches a target distribution p over classes.
The relation applies only under its stated assumptions.

Frequently asked questions about Cross-entropy (classification loss)

What does Cross-entropy (classification loss) describe?+

How should I work through a calculation?+

Which assumption must I keep in mind?+

Which units and quantities are involved?+

When is comparison with shannon-entropie and f1-score useful?+

Retain Cross-entropy (classification loss) for exams

Create a curated FSRS exam set for H(p,q) = −Σ pᵢ ln(qᵢ): formula recall, variables, derivation, rearrangement, worked example, common mistakes and exam context.

Curated formula set · LaTeX · FSRS spaced repetition · import from Essential