header
QM logo in partnership with agena logo
Providing Outreach  in Computer Science Bringing Bayesian solutions to real-world risk problems

Probability: Fallacies, Myths and Puzzles: Main Page

The fallacy of reasoning about evidence in Court

courtroomThe examples here and here expose some classic fallacies of legal reasoning. These fallacies have, unfortunately, often been introduced in real criminal trials.

These, and other similar fallacies of probabilistic reasoning, would be avoided if judges, lawyers and juries were better informed of the basics of probability theory. In fact, the entire legal process could be simplified if those involved in the system were aware of a simple 250-year-old theorem - Bayes Theorem - that explicitly tells us how to revise our beliefs in the light of new evidence.

For reasons that are explained below, the legal system does not currently support using Bayes. In years to come our descendants will reflect in astonishment that the single most obvious tool to help people reason about evidence in a rational way was deliberately ignored in the very arena where this type of reasoning is most crucial.  One of the problems about Bayes Theorem is that the mathematical and statistical community have done little to explain it to 'lay' people.  Try googling on Bayes Theorem and you will see what I mean.

But the explanation is quite simple and is ideally stated in a legal context. And the mathematical formula for Bayes Theorem and other probability laws (even though they are actually quite straightforward) do not even need to be understood or known because there are tools which do all the calculations once you supply the necessary inputs to it.

Basically we start with some hypothesis (let's call it H) which in the legal context is usually the statement "Defendant is innocent". In this case the hypothesis is either true or false (that is not always the case but you do not need to assume anything else in order to understand Bayes).

Now, people will have a belief about whether H is true or not.  If you know absolutely nothing about the defendant or the crime your belief about H will be different to that of, say, the police officer who first charged the defendant, or the prosecuting lawyer who has seen all of the evidence. In each of these cases the belief will be uncertain and so can be expressed as a probability. (Please note that If the idea of quantifying such a belief makes you uncomfortable then do not worry because, as we will, see Bayes Theorem can also be used to determine the extent to which each piece of evidence increases or decreases the probability of guilt without you ever having to quantify your initial belief). The police officer who first charged the defendant might believe that H is true with a probability of 0.1 (and hence false with a probability of 0.9), whereas you may have no reason to believe that the defendent is any more likely to be guilty than any other able-bodied person in the country. So, if there are say 10 million such people, your belief will be that H is false with a probability of  1 in 10 million) and hence that H is true with a probability of 9,999,999 in 10 million). If there were no witnesses to the crime then the only person whose belief about H is not uncertain is the defendent himself (notwithstanding esoteric cases such as the defendant suffering amnesia or being under a hypnotic trance). If he really is innocent then his belief about H being true is 1 and if he is not innocent his belief about H being true is 0.

So any person has what we call a prior belief about H. This is their own (subjective) probability about the truth of H at the start.  So, if you are prepared to quantify this belief, then the first key input to Bayes Theorem is the answer to the following question:

Question 1: "What is your prior belief about the probability the defendant is innocent". (Mathematicians write this as P(H) to stand for "the probability of H").. 

What now happens is that you start to find out evidence. For every piece of evidence E you find out you will naturally want to revise your (prior) belief about H.

If the evidence 'favours' the belief that H is true (for example, suppose the evidence is that the defendant has an alibi) then your revised belief (also called your posterior belief) about H being true should increase; whereas if the evidence favours the belief that H is false (for example, a witness claims to have seen the defendant at the scene of the crime) then your revised belief about H being false should increase.

All of that is simple common sense. The question that Bayes Theorem answers precisely is the following :
Question 2: "what is the revised (posterior) belief about the defendent being inncocent given the evidence".   (Mathematicians write this as P(H | E) meaning "the probability of the hypothesis H given the evidence E")


It turns out that, whereas answering Question 2 directly is normally difficult, it is easier to answer the following question:
Question 3: "what is the probability of seeing the evidence given that the defendent is innocent."  (Mathematicians write this as P(E | H) meaning "the probability of the evidence E given the hypothesis H" and they call it the likelihood).


 (the fact that some people wrongly think that questions 2 and 3 really are the same is exactly the prosecution fallacy) 
 
Pictorially we can represent the situation for Question 3 as:


h implies e

Suppose, for example, the evidence is "a blood sample of the criminal found at the scene matches the blood type of the defendent". If that blood type is found in 1 in every 10 people,  then the answer to question 3 is simply 0.1. Once we have an answer to Questions 1 and 3 then we have all we need to use Bayes Theorem. (the fact that you do not actually need to know Bayes Theorem in order to 'use it' and appreciate the result is emphasized here by giving you a separate link to the theorem itself).  So suppose that your answer  to Question 1 is 0.4. Then, using one of many Bayesian calculation tools that allow you to answer these questions (here we are using AgenaRisk) you will see the following result:

bayesresult


So, having observed the evidence is true (namely that the blood type matches the defendant's), with the answers to Questions 1 and 3 as above the revised belief that the defendant is innocent drops to 0.0625 (i.e. 6.25%). In other words the answer to Question 2 is 0.0625 and this answer is automatically calculated (using Bayes Theorem). Anybody who arrives at a different answer with these same assumptions is demonstrably irrational.

It is also worth noting that the answers to questions 1 and 3 are also used by Bayes Theorem to make the following calculations:

bayesmarg
This is what mathematicians call the 'marginal' probabilities. For H the marginal probability is just the prior probability, but for E the marginal is the probability of the evidence before we have seen any. So, even before seeing evidence about the blood, there is a 0.64 probability that the blood type would match the defendant's. Many jurors would be surprised at how high the probability is in this case. In fact, most lay people assume the probability is 0.1 because they fall into the trap of the base rate fallacy, whereby they think of the (unconditional) probability of seeing the evidence (in this case a random person having the matching blood type) and they ignore the prior probability of inncocence.

I stress that you can see the details of how the calculations work here, but lawyers and jurors for example should never have to know any of that. They simply have to answer questions 1 and 3 and trust a Bayesian tool to do the calculations. In fact, since normally the answer to Question 3 would be provided by a subject expert (such as a DNA expert) this should make the job of the jury even easier still.

So what are the objections to this approach in the legal system?

There are two objections. One rational and one irrational.

The irrational objection concerns the calculation of the answer to Question 2. According to the British legal system, finding the answer to Question 2 must be left to the individual jurors and it is a matter for them (and only them) as to how they arrive at a conclusion. In other words once they have their answers to questions 1 and 3 they must work out for themselves the answer to question 2.  Nobody forbids them from using Bayes theorem but, given their likely ignorance of the theorem, they will almost certainly come to a conclusion that is computationally wrong. That is because, whereas Bayes theorem provides the only rational answer to the question,  people who do not 'apply' the theorem consistently and demonstrably arrive at incorrect answers for their given assumptions about questions 1 and 3. Given the established and (mathematically) universally agreed pedigree of Bayes Theorem, how is it possible that the legal system arrived at the situation whereby it is ignored? Sadly, it is because on the rare occasions where Bayes has been introduced into court, experts have attempted to explain the calculations from first principles rather than simply presenting the results of the calucations as above. In doing so they confused the jury, judge and lawyers. A reasonable analogy to this situation would  be the following:

An expert in court presents the result of dividing 38659 by 27104 as 1.42632. He is asked how he arrived at this figure. Instead of simply saying he used a calculator (and demonstrating the use of the calculator to check that he entered the numbers correctly) he attempts to explain the calculation in terms of the sequence of binary arithmetic functions taking place at the hardware circuit level. 

If an expert really did this then he would also inevitably confuse the jury. But you would  conclude that what the expert did was unnecessary. You would surely not reject the use of calculators to perform long division on the basis that it is too difficult for lay people to understand the underlying sequence of actions that take place at the hardware circuit level. Yet this is exactly what has happened in the case of Bayes Theorem.

The more rational objection to Bayes (though interestingly less prominently articulated within the legal profession) comes from the crucial dependence on the subjective prior belief about H.  It is obvious that people with vastly different 'prior' beliefs about H will (even using Bayes Theorem) end up with different revised beliefs when they are both presented with the same piece of evidence. This observation underpins the legal view that Bayes in 'inappropriate'. But even in this case the legal sensitivities are somewhat spurious for two reasons:
  1. There is a form of Bayes Theorem (see the bottom of the page here) that, for each piece of evidence, enables you to determine the extent to which the evidence supports or rejects the hypothesis H, without you even needing to quantify your prior belief about H. Specificifically,  you only need to consider what is called the likelihood ratio. This is the probability of seeing the evidence given that the defendant is innocent divided by the probability of seeing the evidence given the defendant is guilty. Bayes theorem tells us how the likelihood ratio determines the extent to which the evidence should change our belief about H. If the likelihood ratio is less than 1 then if follows that the odds on the posterior hypothesis of innocence is less than the odds on the prior hypothesis of innocence by a ratio equal to the likelihood ratio. For example, suppose the evidence is matching blood type where the blood type is prevalent in 1 in a 1000 people; then the likelihood ratio is 1/1000 and so the probability of innocence decreases by a factor of 1000 (and the probabuility of guilt increases by a factor of 1000).  Conversely a likelihood ratio of greater than 1 supports the defence case since it increases the probability of innocence by a factor equal to the likelihood ratio.
  2. Jurors will (whether they are asked explicitly or not) have their own prior beliefs about innocence. Just because there is disagreement about the 'starting' position this does not justfify rejecting the only rational way to reason from that starting position. Moreover, if the evidence clearly favours one outcome then (if Bayes is applied)  any two revised beliefs will certainly be closer than the corresponding two prior beliefs. And if the two people are subsequently presented with another piece of evidence then their revised beliefs will get closer still. In fact, the more (common) evidence they see the more their beliefs will converge. With enough consistent evidence the priors become almost irrelevant. That does not help us in the case where there may be few pieces of truly convincing evidence, but it nevertheless is an important counter to those who reject Bayes on the grounds that it is 'all too dependent on subjective priors'.

Two strongly recommended books that cover these issues comprehensively are:
 For more information on fallacies in legal reasoning read this article:

Fenton NE and Neil M, ''The Jury Observation Fallacy and the use of Bayesian Networks to present Probabilistic Legal Arguments'', Mathematics Today ( Bulletin of the IMA, 36(6)), 180-187, 2000.

which is available here:

http://www.agenarisk.com/resources/white_papers/jury_fallacy_revised.pdf

These (and related) issues also arise in the case of Shirley McKie. For an excellent account of this see Steve Horn's dedicated web pages.

Here is the material on Bayes Theorem itself..

For an excellent web page that explains Bayes Theorem interactively try Yuri Yudkowsky's An Intuitive Explanation of Bayesian Reasoning

  footer
Norman Fenton


 Return to Main Page Making Sense of Probability: Fallacies, Myths and Puzzles