Bayes' Theorem

Multiply the reverse conditional probability P(B|A) by the base probability P(A) and divide by the total probability P(B): $P(A|B) =\frac{ P(B|A)\cdot{}P(A)}{P}(B)$. The denominator P(B) is often computed via the law of total probability as $P(B|A)\cdot{}P(A) + P(B|\neg{}A)\cdot{}P(\neg{}A)$. Example medical test: disease with 1 percent prevalence, 95 percent sensitivity, 5 percent false-positive rate. Then $P(D|+) =\frac{ (0.95\cdot{}0.01)}{(0.95\cdot{}0.01 + 0.05\cdot{}0.99)} \approx{} 16$ percent. Despite a good test, a positive result is surprisingly often a false alarm, because the disease is rare and the many false positives outweigh the few true hits. Make sure to build the denominator fully from both contributions, otherwise the result is wrong.

The base rate P(A) scales the numerator in Bayes theorem and largely determines the result. If an event is very rare, it often remains unlikely even after a positive test result, because the many false-positive cases from the large healthy group outweigh the few true hits. This is exactly what the medical-test example shows: for a disease with only 1 percent prevalence, the probability of actually being ill is only about 16 percent despite 95 percent sensitivity. Anyone who ignores the base rate dramatically overestimates the significance of a test. This fallacy is called the base rate fallacy and is widespread in medicine and forensics.

P(A|B) and P(B|A) are two different conditional probabilities and must not be treated as equal. P(A|B) is the probability of A given that B has already occurred; P(B|A) is conversely the probability of B given that A has occurred. An example makes the difference clear: the probability of being ill given a positive test differs greatly from the probability of testing positive given that you are ill. This very confusion is a classic fallacy. Bayes theorem shows how to convert correctly between the two directions, namely by including the base rates P(A) and P(B).

Bayes theorem is the tool to turn an initial probability, the prior probability P(A), into an updated posterior probability P(A|B) in the light of new observations. You multiply the prior by the likelihood P(B|A), that is the probability of the observation under the assumption, and normalize with the total probability P(B). The result takes into account both the prior knowledge and the new evidence. If you obtain further data, you can use the posterior again as a prior and repeat the process. This stepwise learning from data is the basis of Bayesian statistics and of many modern methods of machine learning and diagnostics.

You usually determine the denominator P(B), also called the evidence, via the law of total probability. You consider all ways in which the event B can occur and add their probabilities: $P(B) = P(B|A)\cdot{}P(A) + P(B|\neg{}A)\cdot{}P(\neg{}A)$. The first term captures the cases where B occurs together with A, the second the cases with the complementary event $\neg{}A$. For a test these are the true positives plus the false positives. This denominator is often the decisive and easily overlooked step. Without it you do not get a valid probability, because otherwise the sum of all possibilities does not equal one.