Human Stupidity: Historical: Probability Thinking IV

One particular troublesome example of how disastrous probabilistic mistakes can be and one I use every time for my students is the classical example of testing for the existence of a rare but serious disease. Most of the time, texts refer to the disease as AIDS, but which one is not relevant. Let's just assume that there is a treatment to the disease that has serious side effects and any physician would prefer not to administer it unless really necessary. Luckily, most of the population has not contracted the disease; we actually know only one person in 1,000 has it. We also have a test for it that is reasonably reliable. Whenever someone is sick, it provides a positive result for detecting the virus 98% of the time. Whenever someone does not have the virus it gives a negative result 98% of the times as well (the two 98% chances are not necessarily equal and are often not). In other words, in both cases, it gives an erroneous result only 2% of the times. Assume the test is applied at a person you have no other information about and the result comes positive. This suggests that person might have the disease, but it is also possible that the test has failed. Given all the information above, if you were evaluating this patient, how likely would you say it is that this person is actually sick?

The importance of getting the result correctly can not be overstated here. If it is very likely that the person is sick, treatment should start immediately. If it is very unlikely, it might make sense to prepare additional tests, but, since the treatment has serious side effects, it should not be applied. And, if it is the case that we are not sure at all, for more central probabilities, close to 50%, an assessment of the risk involved in each decision must be made with the proper caution. But that all depends on getting the right evaluation. Ask yourself, just by reading the problem, how likely you think it is that the patient is sick. If you are like the majority of humankind, you will reason that, since the test is correct 98% of time, the probability that the patient would be sick should be around 98% as well. So, you would start the treatment immediately.

But the truth is not so simple. That reasoning simply ignores one extremely important information that you had. And that is the initial chance the patient were sick. I did tell you that was, before the test, 1 in 1,000. Reason a little. If it were the opposite, with 99,9% of the population sick, a positive result should mean extra evidence in favor of the disease and the chance should be even larger than 99,9%, not as small as 98%. By the same reasoning, if you knew for sure, at first, this patient was healthy, you would continue to know it and simply conclude this was one of the 2% cases where the test goes wrong. So, that 1 in 1,000 has important information and you ignored it. And, as a matter of fact, the chance that this patient is sick is not 98%, it is actually smaller than 5%. If the side effects of the treatment are severe, a person that was had a 95% chance of being healthy would have to suffer it, without enough evidence to support the need for it.

What is going on? It is actually not so hard to understand when we look at the whole picture. What we know is that the test gave a positive result. Two things might have happened to cause that result. The patient could actually be sick and that would have happened initially with one chance in 1,000. Or the test could have failed, providing a positive answer for a healthy person. This case had a chance of 20 in 1,000. Clearly, the failure of the test is much more likely than the hypothesis that the patient is sick, 20 times more likely, actually. The chance of the patient being sick actually increases from the initial 1 in 1,000 to a posterior probability a little less than 50 in 1,000. That is a huge increase and this large change is due to the fact the test is reasonably reliable. But, by ignoring one very important piece of information in the problem, completely wrong decisions about our health can happen (and almost certainly do happen) every day (the correct way to calculate the final probability is using Bayes Theorem. Its use will be explained later here).

As a matter of fact, the simple misuse of basic probabilistic and statistical concepts in health problems is so widespread and so serious that efforts already exist to better educate physicians and patients. Gerd Gigerenzer  has been championing this urgent need for better education, with good reason. He has also collected a number of stories on how that misuse can cause bad health decisions and bad policies, leading governments to spend money on non-existant problems. One of those stories I particularly like I heard from him in a conference. The problem started when  Rudi Giuliani said in a campaign advertisement in 2007 that: ``I had prostate cancer, 5, 6 years ago. My chance of surviving prostate cancer -- and thank God I was cured of it -- in the United States? 82%. My chance of surviving prostate cancer in England? Only 44% under socialized medicine''(original story in the Washington Post is here).

This certainly looks as if there were a very serious difference in the quality of the treatments in USA and England. But the actual mortality rates in both countries are basically the same. And yet, Giuliani was not lying, just using numbers without all the competence required to analyse them. Those figures he cited were 5-year survival rates. What was actually happening was that in the USA men are under a lot of pressure to screen for prostate cancer and many actually do participate in the prostate-specific antigen (PSA) screening. That does not happen in England. Even forgetting the important and not clear question of whether screening actually saves lives or not, there is one other extremely important effect here. The percentages do not refer to the groups treated equally and, as such, they just are NOT comparable. Gigerenzer has the perfect example to explain why.

Imagine two men, one American and one English, who will die of prostate cancer at age 70, regardless of treatment. In both cases, the cancer could be detected by screening as early as they were 60. But only the American does the screening and he discovers the disease at age 60. Even if the treatment fails and he dies at 70, his 5-year survival rate is still 100%, he does get to 65 without dying. The English man, on the other hand, only finds out he is sick when the disease is advanced and there are clear symptoms, at age 67. He also dies at the same age, but he does not pass the 5-year survival period. Not because of any difference in the health systems, both men contracted the disease at the same age and died the same age as well. The only difference is that one knew it much earlier. And, since American men are screened, they do know about it earlier and the almost double death rate in England is nothing more than a trick of measuring things differently.