Likelihood Ratios are a convenient way to measure the weight of evidence. They allow us to view Bayes Theorem as a simple linear relationship in odds form, thus neatly depicting what we have called the evidential tug of war – the iterative, multiplicative accumulation of evidence that drives the search for Truth. They also give us a useful graphical representation of the relationship between PP and BR as a function of LR that summarises all the properties of Bayes Theorem we have discussed so far.
In Othello’s case, we have TPR=50% and FPR=5%, hence LR=10: the handkerchief evidence weighs 10 times more under the hypothesis of infidelity than under its complement. In Bob’s case, we have TPR=95% and FPR=5%, hence LR=19: a positive virus test is 19 times weightier if Bob has the virus than if he doesn’t. In both instances, evidence is strongly confirming (LR>1) and quite accurate. In fact, we assumed the virus test is 95% symmetrically accurate: TPR=TNR=95%. By the same token, the handkerchief evidence has TPR=50% and TNR=1-FPR=95%, hence we say it has 72.5% asymmetric accuracy – the average of the two true rates.
This is the natural definition of Accuracy:
where the three equivalent measures follow from the definitions.
In general, think of evidence E as a signal about hypothesis H. If the signal is present, we call the evidence positive. If it is absent, we call it negative.
The signal can produce two right calls, or Hits:
E is positive when H is true, with probability TPR, also known as Sensitivity.
E is negative when H is false, with probability TNR, also known as Specificity.
And two wrong calls, or Errors:
E is positive when H is false, with probability FPR=1-TNR, also called Type I Error, or False Alarm.
E is negative when H is true, with probability FNR=1-TPR, also called Type II Error, or Miss.
Like probability, Accuracy ranges from 0 to 1:
Maximum Accuracy (A=1) denotes Perfect Evidence: TPR=TNR=1, hence FPR=FNR=0. Perfect Evidence is perfectly accurate about H: it always makes Hits and never makes Errors. It has Maximum Sensitivity: it is always positive when H is true. And Maximum Specificity: it is always negative when H is false.
Minimum Accuracy (A=0) denotes Perfect Counterevidence: TPR=TNR=0, hence FPR=FNR=1. Perfect Counterevidence is the mirror image of Perfect Evidence. It is perfectly accurate about H̄: it is always positive when H is false and always negative when H is true.
In the middle between the two extremes, A=0.5 denotes Neutral Evidence: TPR=FPR, hence TNR=FNR. Neutral Evidence is weightless about H: it is as likely to be right as to be wrong, whether it is positive or negative. Hence another name for it is Coin Toss Evidence.
A third measure of the weight of evidence is the Likelihood gap itself: TPR-FPR, also known in meteorology1 as Youden’s J statistic.
Notice J = 2A-1, hence we have Perfect Evidence: J=1, Perfect Counterevidence: J=-1 and Neutral Evidence: J=0. In Othello’s case: J=45%. In Bob’s case: J=90%.
The three measures are linked by the following relationship:
which lends itself to a resonant interpretation:
The tilt of the PO line away from the neutral 45° line is equal to the ratio between Net Signal – how often is E present under H rather than under H̄ – and Noise – how often does E produce False Alarms.
Putting it all together, we notice three more crucial properties of Bayes Theorem:
First, a signal that is always wrong about H is as useful as a signal that is always right. Perfect Counterevidence is as accurate about H̄ as Perfect Evidence is about H. Imagine a virus test that is always positive when you don’t have the virus and always negative when you have it. That would be as informative as a perfect test. Or, to revisit our earliest example, imagine a weather forecast that always predicts Rain when it doesn’t rain and No Rain when it does. That would be as useful, if one draws the right conclusion.
Second, a perfectly useless signal (LR=1, A=0.5, J=0) is not one that is always wrong. It is one that has no weight: it is as likely to be right as to be wrong. Consider an extreme example: a bogus virus test that is always positive: TPR=FPR=1. It would have Maximum Sensitivity – it would never miss an infected individual, producing no False Negatives (FNR=0). However, it would simultaneously have Minimum Specificity – it would classify every health individual as infected, thus producing a False Alarm for all of them (FPR=1).
Third, there is an important difference between perfect and conclusive evidence. Conclusive evidence is not perfect:
The presence of a Smoking Gun proves H: FPR=0, hence PP=1. But its absence does not prove H̄: if no smoking gun is found, the suspect may still be guilty: FNR>0. Hence the Accuracy of a Smoking Gun is smaller than 1: A=1-FNR/2, J=1-FPR.
The presence of an Alibi proves H̄: TPR=0, hence PP=0. But its absence does not prove H: if the suspect does not have an alibi, he may still be innocent: FPR>0. Hence the Accuracy of an Alibi is greater than 0: A=(1-FPR)/2, J=-FPR.
In other words:
With both Smoking Gun and Perfect Evidence, a positive signal proves H: PP=1. But only with Perfect Evidence does a negative signal disprove H: PP=0.
With both Alibi and Perfect Counterevidence, a positive signal proves H̄: PP=0. But only with Perfect Counterevidence does a negative signal disprove H̄: PP=1.
Simply put: Conclusive and Perfect Evidence both provide a sufficient reason for certainty. But only Perfect Evidence also provides a necessary reason.
I know this all sounds abstract and theoretical. But, like everything we encounter along the Bayes Trail, it has many profound, concrete and practical consequences.
As we shall soon see.