Proof of Bayes' rule: Intro

[todo: This page has been orphaned, for being insufficiently different from /p/1xr (and not much easier to read). It should probably be deleted.]

In what follows, $~$H_i$~$ and $~$H_j$~$ are two competing hypotheses, and $~$e$~$ is the evidence. Bayes' rule (in the odds form) says:

$$~$ \dfrac{\mathbb P(H_i)}{\mathbb P(H_j)} \cdot \dfrac{\mathbb P(e\mid H_i)}{\mathbb P(e\mid H_j)} = \dfrac{\mathbb P(H_i\mid e)}{\mathbb P(H_j\mid e)} $~$$

The quantity $~$\frac{\mathbb P(H_i)}{\mathbb P(H_j)}$~$ is the prior odds. E.g., in the Diseasitis problem, dividing 20% by 80% gives us 1/4, which represents Odds of (1 : 4).

The quantity $~$\frac{\mathbb P(e\mid H_i)}{\mathbb P(e\mid H_j)}$~$ is the likelihood ratio. For example, in the diseasitis problem, we'd divide $~$\frac{\mathbb P({positive}\mid {sick})}{\mathbb P({positive}\mid \neg {sick})}$~$ to get the number 3, indicating "Sick patients are three times as likely as healthy patients to turn the tongue depressor black."

The quantity $~$\frac{\mathbb P(H_i\mid e)}{\mathbb P(H_j\mid e)}$~$ is the posterior odds ratio. E.g., in the Diseasitis problem, $~$\frac{\mathbb P({sick}\mid {positive})}{\mathbb P(\neg {sick}\mid {positive})}$~$ works out to an odds ratio of 3/4, which are odds of (3 : 4). We indeed have that 1/4 * 3 = 3/4, so the rule is numerically true in this case.

The proof of Bayes's Rule in its general form proceeds by repeatedly applying the definition of conditional probability, $~$\mathbb P(X\mid Y) = \frac{\mathbb P(X \wedge Y)}{\mathbb P(Y)}.$~$

$$~$ \dfrac{\mathbb P(H_i)}{\mathbb P(H_j)} \times \dfrac{\mathbb P(e\mid H_i)}{\mathbb P(e\mid H_j)} = \dfrac{\mathbb P(e \wedge H_i)}{\mathbb P(e \wedge H_j)} = \dfrac{\mathbb P(e \wedge H_i) / \mathbb P(e)}{\mathbb P(e \wedge H_j) / \mathbb P(e)} = \dfrac{\mathbb P(H_i\mid e)}{\mathbb P(H_j\mid e)} $~$$

In the Diseasitis problem, the corresponding quantities would be:

$$~$\dfrac{20\%}{80\%} \times \dfrac{90\%}{30\%} = \dfrac{18\%}{24\%} = \dfrac{0.18 / 0.42}{0.24 / 0.42} = \dfrac{43\%}{57\%}$~$$

Visualizing the proof graphically:

Which can be read as saying: The ratio of the initial sick population (red) to the initial healthy population (blue), times the ratio of positive results (+) in the sick population to positive results in the blue population, equals the ratio of the positive-and-red population to positive-and-blue population. Thus we can divide both into the proportion of the whole population (grey) which got positive results, yielding the posterior odds of sick (red) vs healthy (blue) among only those with positive results.