Why this mat­ters

Imag­ine a med­ical test comes back pos­i­tive. How likely is it that the per­son really has the con­di­tion? That ques­tion runs a con­di­tional prob­a­bil­ity back­wards, and Bayes' the­o­rem is the tool for it. To get there, we first need the total prob­a­bil­ity of an event, added up over all its pos­si­ble causes. So we will prove the the­o­rem of total prob­a­bil­ity, then Bayes' the­o­rem, and put both to work.

Par­ti­tions and total prob­a­bil­ity

We say events E1,E2,…,EnE_1, E_2, \ldots, E_n form a par­ti­tion of SS when no two of them over­lap, together they cover every­thing, and each one has non-zero prob­a­bil­ity. Then for any event AA,

P(A)=∑j=1nP(Ej) P(A∣Ej).\displaystyle P(A) = \sum_{j=1}^{n} P(E_j)\,P(A \mid E_j).

The idea is sim­ple. AA breaks up into sep­a­rate pieces A∩EjA \cap E_j, and each piece has prob­a­bil­ity P(Ej)P(A∣Ej)P(E_j)P(A \mid E_j). Add the pieces back together and you have the whole event.

Exam­ple 1. A fac­tory has three machines mak­ing 50%50\%, 30%30\% and 20%20\% of its bolts, and their defect rates are 2%2\%, 3%3\% and 5%5\%. Then P(defective)=0.5(0.02)+0.3(0.03)+0.2(0.05)=0.029P(\text{defective}) = 0.5(0.02) + 0.3(0.03) + 0.2(0.05) = 0.029.

Tree diagram: machines 1, 2, 3 make 0.5, 0.3 and 0.2 of the bolts, with defect rates 0.02, 0.03 and 0.05; the three branch products 0.010, 0.009 and 0.010 add to 0.029.
Total prob­a­bil­ity: add the prod­ucts along every branch that ends in a defec­tive bolt.

Bayes' the­o­rem

P(Ei∣A)=P(Ei) P(A∣Ei)∑jP(Ej) P(A∣Ej).\displaystyle P(E_i \mid A) = \frac{P(E_i)\,P(A \mid E_i)}{\sum_j P(E_j)\,P(A \mid E_j)}.

We call the P(Ei)P(E_i) the prior prob­a­bil­i­ties: what we believed before. The P(Ei∣A)P(E_i \mid A) are the pos­te­rior prob­a­bil­i­ties, our updated belief once we have seen AA.

Exam­ple 2. Go back to Exam­ple 1 and sup­pose a bolt turns out defec­tive. How likely is it that the third machine made it? The answer is 0.2×0.050.029=1029\displaystyle \tfrac{0.2 \times 0.05}{0.029} = \tfrac{10}{29}.

Exam­ple 3. Say a dis­ease affects 1%1\% of peo­ple. A test catches it in 95%95\% of those who have it, but it also wrongly says pos­i­tive for 4%4\% of those who do not. If your result is pos­i­tive, P(disease∣+)=0.01×0.950.01×0.95+0.99×0.04=0.00950.0491≈0.19\displaystyle P(\text{disease} \mid +) = \frac{0.01 \times 0.95}{0.01 \times 0.95 + 0.99 \times 0.04} = \frac{0.0095}{0.0491} \approx 0.19. Sur­prised? Because the dis­ease is rare, most pos­i­tive results are actu­ally false alarms.

Tree diagram: 0.01 have the disease and test positive with probability 0.95; 0.99 do not and test positive with probability 0.04; the positive branches give 0.0095 and 0.0396.
Most pos­i­tive results come from the large healthy group, so P(dis­ease given +) is only about 0.19.

Exam­ple 4. Bag I has 33 red and 22 blue balls, and bag II has 11 red and 44 blue. You pick a bag at ran­dom, draw a ball, and it is red. Then P(bag I∣red)=12⋅3512⋅35+12⋅15=34\displaystyle P(\text{bag I} \mid \text{red}) = \frac{\frac{1}{2}\cdot\frac{3}{5}}{\frac{1}{2}\cdot\frac{3}{5} + \frac{1}{2}\cdot\frac{1}{5}} = \tfrac{3}{4}.

Exam­ple 5. In an exam, a stu­dent knows the answer with prob­a­bil­ity 23\displaystyle \tfrac{2}{3}; oth­er­wise the stu­dent guesses from 44 options. If the answer is cor­rect, how likely is it that the stu­dent actu­ally knew it? We get 2323+13⋅14=89\displaystyle \frac{\frac{2}{3}}{\frac{2}{3} + \frac{1}{3}\cdot\frac{1}{4}} = \tfrac{8}{9}.

Prac­tice

  1. There are two boxes. Box A has 22 gold and 33 sil­ver coins; box B has 44 gold and 11 sil­ver. A box is picked at ran­dom and a coin is drawn. Find P(gold)P(\text{gold}).
  2. In ques­tion 1 the coin turns out gold. Find the prob­a­bil­ity that it came from box B.
  3. 40%40\% of a col­lege's stu­dents live in hos­tels. 30%30\% of hos­tel stu­dents and 20%20\% of day stu­dents get an A grade. A stu­dent with an A is cho­sen at ran­dom. Find P(hostel)P(\text{hostel}).
  4. A fac­tory has two plants mak­ing 60%60\% and 40%40\% of its out­put, and 3%3\% and 5%5\% of their items are faulty. A faulty item turns up. Which plant more likely made it?
  5. A test for a con­di­tion detects 98%98\% of cases and has a 3%3\% false-pos­i­tive rate; the con­di­tion affects 2%2\% of peo­ple. Find the prob­a­bil­ity that some­one who tests pos­i­tive really has it.
  6. A woman tells the truth 44 times in 55. She says a die showed a one. Find the prob­a­bil­ity that it really did.
  7. There are three coins: one has two heads, one is biased to show heads 75%75\% of the time, and one is fair. A coin is picked at ran­dom and shows heads. Find the prob­a­bil­ity that it is the two-headed coin.
  8. In one sen­tence, explain why a very accu­rate test can still give mostly false pos­i­tives.

Answers

Show answers
  1. 12⋅25+12⋅45=35\displaystyle \tfrac{1}{2}\cdot\tfrac{2}{5} + \tfrac{1}{2}\cdot\tfrac{4}{5} = \tfrac{3}{5}.
  2. 0.40.6=23\displaystyle \tfrac{0.4}{0.6} = \tfrac{2}{3}.
  3. 0.4×0.30.4×0.3+0.6×0.2=12\displaystyle \tfrac{0.4 \times 0.3}{0.4 \times 0.3 + 0.6 \times 0.2} = \tfrac{1}{2}.
  4. Plant 1 gives 0.0180.018 and plant 2 gives 0.0200.020, so it is plant 2 (P=2038=1019\displaystyle P = \tfrac{20}{38} = \tfrac{10}{19}).
  5. 0.02×0.980.02×0.98+0.98×0.03=0.01960.049=0.4\displaystyle \tfrac{0.02 \times 0.98}{0.02 \times 0.98 + 0.98 \times 0.03} = \tfrac{0.0196}{0.049} = 0.4.
  6. 16⋅4516⋅45+56⋅15=49\displaystyle \frac{\frac{1}{6}\cdot\frac{4}{5}}{\frac{1}{6}\cdot\frac{4}{5} + \frac{5}{6}\cdot\frac{1}{5}} = \tfrac{4}{9}.
  7. 11+0.75+0.5=49\displaystyle \frac{1}{1 + 0.75 + 0.5} = \tfrac{4}{9}.
  8. When the con­di­tion is rare, the few true pos­i­tives get swamped by the false pos­i­tives com­ing from the much larger healthy group.