We have taken a good look at the Base Rate BR.

Let’s now examine the other component of Bayes Theorem: the Likelihood pair TPR and FPR.

The True Positive Rate TPR – the probability of the evidence if the hypothesis is true – and the False Positive Rate FPR – the probability of the evidence if the hypothesis is false – provide the weight of evidence that drives the update from BR to the posterior probability PP.

Remember: Evidence is confirming if it is more likely when H is true: TPR>FPR, hence PP>BR. It is disconfirming if it is more likely when H is false: TPR<FPR, hence PP<BR. And it is neutral – uninformative, irrelevant – if it as likely when H is true as when H is false: TPR=FPR, hence PP=BR.

The gap between TPR and FPR determines the size of the update. This can be measured in several ways. One of them is the Likelihood Ratio LR=TPR/FPR, where we have:

Confirming evidence: LR>1

Disconfirming evidence: LR<1

Neutral evidence: LR=1.

The dynamics of the updating process is best seen when Bayes Theorem is written in odds form (mathophobes – you can do it if you try):

PO = LR ∙ BO

where PO=PP/(1-PP) are Posterior Odds and BO=BR/(1-BR) are Prior Odds.1

This says that Posterior Odds are a linear function of Prior Odds, with slope LR. The formula helps us to visualise what we have called the evidential tug of war – the iterative accumulation of evidence that drives the search for Truth:

The updating process unfolds through multiplicative accumulation. Starting from initial prior odds BO₀, evidence E₁ generates Likelihood Ratio LR₁, thus producing posterior odds PO₁. These then become the new prior odds for the next step: PO₁=BO₁. Then a second, conditionally independent piece of evidence E₂ provides LR₂, leading to PO₂, which become BO₂, and so on.

As we saw at the beginning of our journey, the tug of war can lead to distinct outcomes, which we are now equipped to describe more precisely.

Interestingly, they are closely echoed in the note on Locke which opens Section VI of Hume’s Enquiries Concerning Human Understanding, entitled Of Probability:2

Mr. Locke divides all arguments into demonstrative and probable. In this view, we must say, that it is only probable that all men must die, or that the sun will rise to-morrow. But to conform our language more to common use, we ought to divide arguments into demonstrations, proofs, and probabilities. By proofs meaning such arguments from experience as leave no room for doubt or opposition.

Demonstrations are based on what we have called Faith: a prior certainty in the truth or falsity of a hypothesis. Faith requires no tug of war as no amount of evidence can change it.

Proofs, on the other hand, are entirely based on evidence, and occur when the tug of war has a winner. This can happen in two ways:

1. A single piece of conclusive evidence yields de jure certainty

Multiplicative accumulation implies that even a single piece of conclusive evidence can stop the update and immediately drive Posterior Odds all the way to infinity or to zero. Namely, a Smoking Gun (FPR=0) is sufficient to yield an infinite LR, thus driving PO to infinity (a vertical line) and PP to 1. Conversely, a single Alibi (TPR=0) is all we need to yield a LR of 0, thus collapsing PO to 0 (a horizontal line), and PP to 0. At any stage of the update, the acquisition of conclusive evidence implies that the hypothesis is certainly true or certainly false. A Smoking Gun is enough to prove that the hypothesis “The suspect is guilty” must be true. An Alibi is enough to prove that the hypothesis must be false. Or, to use another famous image, one black swan is sufficient to prove that the hypothesis “All swans are white” must be false. Such is the allure of conclusive evidence: it provides irrefutable, de jure certainty, thus freeing our beliefs from subjective priors.

2. Accumulation of inconclusive evidence yields de facto certainty

Multiplicative accumulation also implies that, if Likelihood Ratios are consistently confirming (LR>1) or disconfirming (LR<1), Posterior Odds tend to infinity or to zero. Hence, posterior probabilities converge towards certainty. While each piece of evidence falls short of proof, they collectively carry overwhelming weight in one direction. PP does not reach 0 or 1 exactly, yet it approximates one or the two extremes, thus allowing us to consider the hypothesis true or false beyond reasonable doubt. We cannot demonstrate that all men must die, or that the sun must rise tomorrow. We can only expect it, based on an overwhelming accumulation of confirming evidence. As they converge to one of the two boundaries of the probability spectrum, posterior probabilities again cease to depend on Base Rates: whatever the initial non-zero priors (remember Cromwell’s Rule), convergence proves that the hypothesis is true or false. This happens to everyone’s satisfaction, leaving no room for doubt or opposition. However, such certainty is not the inescapable consequence of conclusive evidence, but merely the limit of a convergent accumulation of inconclusive evidence. It is virtual, de facto certainty, which again frees us from subjective priors but remains open to refutation.

Finally, Probabilities are the result of a tug of war where evidence remains in the balance, as neither side manages to prevail on the other. Evidence may shift probabilities away from initial priors, but neither conclusively nor overwhelmingly. As competing LR offset one another, the tug of war produces no winner, leaving us with the belief that the hypothesis is probably true and, by complement, also probably false.

Crucially – as we have seen – unlike proofs, probabilities fail to free our beliefs from priors. Hence the illegitimate temptation, driven by our craving for certainty, to escape the burden of priors by falling into Prior indifference.

In fact, notice that, under Prior indifference, we have BO=1, and therefore PO=LR, i.e. PP=LR/(1+LR): confirming evidence=supportive evidence, and neutral evidence=coin toss evidence.

Simple algebra3 gives us the relationship between PP and BR as a function of LR:

Graphically:

The graph is a visual compendium of all the Bayes Theorem properties we have encountered so far:

Faith: BR=0 hence PP=0, BR=1 hence PP=1, regardless of LR.

Cromwell’s Rule: 0<BR<1.

Confirming evidence: LR>1, hence PP>BR. Notice the increasing concavity of the curve as the weight of evidence increases.

Disconfirming evidence: LR<1, hence PP<BR. Notice the increasing convexity of the curve as the weight of evidence increases.

Neutral evidence: LR=1, hence PP=BR. No update.

Conclusive evidence: De jure certainty

• Smoking Gun: FPR=0, hence LR=∞. Maximum concavity (coinciding with the vertical axis), yielding PP=1 for all values of BR.

• Alibi: TPR=0, hence LR=0. Maximum convexity (coinciding with the horizontal axis), yielding PP=0 for all values of BR.

Converging evidence: De facto certainty

Iterative accumulation of confirming evidence pushing LR towards the vertical axis, or disconfirming evidence pushing LR towards the horizontal axis.

Laplace’s dictum: The smaller BR, the greater the LR required to achieve a given PP.

Prior Indifference: Shifting a small BR to Prior indifference (BR=50%) is a faster but illegitimate way to boost PP in lieu of increasing the weight of evidence.

Amazingly, this is David Hume 15 years before the appearance of Bayes Theorem:

A wise man, therefore, proportions his belief to the evidence. In such conclusions as are founded on an infallible experience, he expects the event with the last degree of assurance, and regards his past experience as a full proof of the future existence of that event. In other cases, he proceeds with more caution: He weighs the opposite experiments: He considers which side is supported by the greater number of experiments: To that side he inclines, with doubt and hesitation; and when at last he fixes his judgment, the evidence exceeds not what we properly call probability. All probability, then, supposes an opposition of experiments and observations, where the one side is found to overbalance the other, and to produce a degree of evidence, proportioned to the superiority. A hundred instances or experiments on one side, and fifty on another, afford a doubtful expectation of any event; though a hundred uniform experiments, with only one that is contradictory, reasonably beget a pretty strong degree of assurance. In all cases, we must balance the opposite experiments, where they are opposite, and deduct the smaller number from the greater, in order to know the exact force of the superior evidence.4

Share

Othello and Bob: two unlikely bedfellows in the grip of the Prior indifference fallacy.

Othello badly misjudged the probability of his beloved wife’s infidelity, and paid the ultimate price. Bob made the same mistake with the probability of carrying a rare virus, and had the scare of his life.

Read more

Let’s ask Bob.

Bob hears on television that forty people have recently died from a lethal virus. He actually knew one of the deceased. Although he is not the impressionable type, just to be sure he calls his doctor. The doctor says he shouldn’t worry but, to be safe, he can take a test that is 95% accurate at spotting the virus. Well, let’s do it then, says Bob, and fixes an appointment for the next day.

That night he can’t sleep. His mind keeps going back to that poor fellow who died, just 32, leaving a wife and two children. He has no reason to worry, but what if he has the virus? And, much worse, what if he has it, but happens to be in the 5% of cases for which the test is wrong? He thinks he doesn’t have the virus but he actually has it: that would be just horrible.

The next morning Bob rushes to the doctor’s clinic to get the test done and put an end to his mental torture. The result comes in the afternoon. Immediately he notices there is something wrong on the doctor’s face. In a highly embarrassed tone, the doctor gives him the verdict: you tested positive – the test said you have the virus. Remember, there is still a chance that the test is wrong, but I am afraid it is not very high. Sorry.

Desperate, Bob staggers back home and start writing his will, when his friend Tom, the statistician, calls him on the phone. Bob tells him the awful news but, to his dismay, Tom starts laughing. So what? – he says –do you want to know the probability that you have the virus? Yes, 95% – Bob cries. Think again – says Tom – the probability is less than 2%. I’ll come to your place and explain – if you offer me a beer or two.

Read more

Back to Bayes Theorem.

Remember: The Posterior Probability PP of hypothesis H in the light of evidence E depends on three parameters:

1. BR: The prior probability of H, or Base Rate

and the Likelihood Pair:

2. TPR: The probability of observing E if H is true, or True Positive Rate

3. FPR: The probability of observing E if H is false, or False Positive Rate.

As we have seen, there are two ways to be certain that H is true (PP=1):

100% Faith – A priori certainty, irrespective of any evidence: BR=1

Smoking Gun – A posteriori certainty, because of conclusive evidence showing that E is incompatible with H being false: FPR=0.

And there are two ways to be certain that H is false (PP=0):

0% Faith – A priori certainty, irrespective of any evidence: BR=0.

Alibi – A posteriori certainty, because of conclusive evidence showing that E is incompatible with H being true: TPR=0.

Both Faith and conclusive evidence deliver certainty. But they do so in opposite ways. Faith defies Cromwell’s Rule, thus disabling Bayes Theorem. Conclusive evidence relies on Cromwell’s Rule, thus enabling Bayes Theorem. Faith relies on extreme priors. Conclusive evidence renders priors irrelevant.A priori certainty is impermeable to evidence. A posteriori certainty is entirely built on it.

A good Bayesian adheres to Cromwell’s Rule and looks for conclusive evidence. If he can find it, fine: he reaches certainty – the Gold Standard of Science. But if he can’t, he is fine as well: he is comfortable with uncertainty and enjoys the tug of war between confirming and disconfirming evidence, where H is probably true and, by complement, also probably false.

Inconclusive evidence leaves truth in the balance, placing Posterior Probability PP somewhere along the spectrum between 0 and 1, where its position will depend on the starting point BR and on the weight of evidence provided by the Likelihood Pair TPR and FPR.

Let’s first look at BR.

Read more

My life upon her faith. (Act 1, Scene 3)

In other words: BR=0. But as soon as the seed settles in, Faith gives way to sober reflection (Act 3, Scene 3):

Othello: I do not think but Desdemona’s honest.

Iago: Long live she so, and long live you to think so!

Othello: And yet how nature, erring from itself –

Iago: Ay, there’s the point;

Infidelity may be unlikely but, by Cromwell’s Rule, is not impossible: BR>0. Iago starts in a low key. He describes himself as a rather poor forecaster, prone to calling too many False Positives:

Though I perchance am vicious in my guess,

As I confess it is my nature’s plague

To spy into abuses, and oft my jealousy

Shapes faults that are not –

He knows Othello is starting from a very low BR, and that he will require very accurate evidence to be moved away from it. Othello is very clear about this:

No, Iago,

I’ll see before I doubt; when I doubt, prove;

And on the proof, there is no more but this:

Away at once with love or jealousy.

Othello is willing to change his mind, but only if he sees full-proof evidence: TPR=1 or FPR=0. Iago knows this will be impossible. But he also knows that, if he manages to increase Othello’s priors, imperfect evidence may suffice. So he gets to work:

I speak not yet of proof.

Look to your wife. Observe her well with Cassio.

Wear your eyes thus: not jealous, nor secure.

I would not have your free and noble nature

Out of self-bounty be abused. Look to ’t.

I know our country disposition well.

In Venice they do let God see the pranks

They dare not show their husbands; their best conscience

Is not to leave ’t undone, but keep ’t unknown.

Forget about this BR=0 nonsense, says Iago. Open your mind and face reality: the probability that a Venetian woman is unfaithful is higher than you like to think.

Othello trusts Iago and thinks he is an expert judge of human foibles:

This fellow’s of exceeding honesty,

And knows all quantities, with a learnèd spirit

Of human dealings.

But he is desperately trying to hang on to his Faith:

If she be false, O then heaven mocks itself!

I’ll not believe’t.

Then Iago gets hold of the handkerchief and orchestrates his plot by planting it in Cassio’s lodgings. He knows the evidence is far from conclusive, but he is counting on Othello’s expanded BR:

I will in Cassio’s lodging lose this napkin,

And let him find it. Trifles light as air

Are to the jealous confirmations strong

As proofs of holy writ. This may do something.

The Moor already changes with my poison.

Dangerous conceits are in their natures poisons,

Which at the first are scarce found to distaste,

But with a little act upon the blood,

Burn like the mines of sulphur.

Othello is in pain. He feels his BR expanding. What he would love to see is an Alibi: evidence that is logically inconsistent with the hypothesis of infidelity: TPR=0. This would imply PP=0, regardless of BR. He could ask Desdemona, or Cassio, or Iago’s wife Emilia, Desdemona’s confidant. But he doesn’t trust them. Instead, he asks Iago for a Smoking Gun: evidence that is logically inconsistent with the hypothesis of fidelity: FPR=0. He hopes Iago is wrong, of course. But in case he is right, he will at least end the torment of an expanding BR. If that’s the only way, so be it, even if it implies PP=1.

Othello threatens Iago to provide no less than a Smoking Gun:

Othello: Villain, be sure thou prove my love a whore.

Be sure of it. Give me the ocular proof,

Or, by the worth of mine eternal soul,

Thou hadst been better have been born a dog

Than answer my waked wrath.

Iago: Is’t come to this?

Othello: Make me to see’t, or at the least so prove it

That the probation bear no hinge nor loop

To hang a doubt on, or woe upon thy life.

But it’s too late. Othello’s BR has already expanded to the 50% indifference level:

By the world,

I think my wife be honest, and think she is not.

I think that thou art just, and think thou art not.

I’ll have some proof.

This is Prior Indifference precisely. Othello is done: he doesn’t need FPR=0. Some proof will do. He just wants to be satisfied:

Iago: And may. But how, how satisfied, my lord?

Would you, the supervisor, grossly gape on,

Behold her topped?

Othello: Death and damnation! O!

Iago: It were a tedious difficulty, I think,

To bring them to that prospect. Damn them then

If ever mortal eyes do see them bolster

More than their own! What then? How then?

What shall I say? Where’s satisfaction?

It is impossible you should see this,

Were they as prime as goats, as hot as monkeys,

As salt as wolves in pride, and fools as gross

As ignorance made drunk. But yet, I say,

If imputation, and strong circumstances

Which lead directly to the door of truth,

Will give you satisfaction, you may ha’t.

Othello: Give me a living reason she’s disloyal.

By now Iago is teasing Othello. Sure – he says – you should try to surprise your wife in bed with Cassio – that would be a perfect proof! Unfortunately, it is rather tricky. So you’d better be happy with circumstantial evidence. Othello is beaten. He no longer requires a perfect proof – barely a living reason will be enough. Iago obliges, by telling him that he heard Cassio talking in his sleep about his affair with Desdemona, and that Cassio has the handkerchief. So what? – Othello would have said, had he kept a small BR. Instead:

O that the slave had forty thousand lives!

One is too poor, too weak for my revenge.

Now do I see ’tis true. Look here, Iago,

All my fond love thus do I blow to heaven – ‘tis gone.

Arise, black vengeance, from the hollow hell.

Yield up, O love, thy crown and hearted throne

To tyrannous hate! Swell, bosom, with thy freight,

For ’tis of aspics’ tongues.

To Othello’s Prior Indifference, Iago’s evidence is virtually as conclusive as a Smoking Gun.

By Act 4, Scene 1, Iago has turned the poor Othello into the hapless victim of his perfidious irony:

Iago: Will you think so?

Othello: Think so, Iago?

Iago: What, to kiss in private?

Othello: An unauthorized kiss.

Iago: Or to be naked with her friend in bed

An hour or more, not meaning any harm?

Othello: Naked in bed, Iago, and not mean harm?

It is hypocrisy against the devil.

They that mean virtuously and yet do so,

The devil their virtue tempts, and they tempt heaven.

Iago: So they do nothing, ’tis a venial slip.

But if I give my wife a handkerchief—

Finally, the coup de grace. Othello had started with BR=0; then he admitted BR>0; then Iago manoeuvred him into Prior Indifference: BR=50%. What next?

Othello: Dost thou mock me?

Iago: I mock you not, by heaven.

Would you would bear your fortune like a man.

Othello: A hornèd man’s a monster and a beast.

Iago: There’s many a beast then in a populous city,

And many a civil monster.

Othello: Did he confess it?

Iago: Good sir, be a man.

Think every bearded fellow that’s but yoked

May draw with you. There’s millions now alive

That nightly lie in those unproper beds

Which they dare swear peculiar. Your case is better.

Oh, ’tis the spite of hell, the fiend’s arch-mock,

To lip a wanton in a secure couch,

And to suppose her chaste! No, let me know,

And knowing what I am, I know what she shall be.

Face reality, says Iago. Being cheated by their wives is the plight of most married men: BR≈1. You might as well start with that assumption. In fact, worse than the small risk of a False Positive – a wrong belief of being betrayed by a faithful wife – is the much larger risk of a False Negative: to sleep unknowingly with an unfaithful one. So cheer up: now that you know you’ve been cheated, you are no longer at risk. You are the lucky one.

Leave a comment

How could this have happened? And how should he begin to make sense of such a troubling discovery?

Othello wants to know: What is the probability of

H: Desdemona has been unfaithful

in the light of

E: Cassio has the handkerchief

Read more

I hope you have now got a good grip on Bayes formula. I encourage you to write it down on a spreadsheet and play around with it: tweak the three inputs and watch how the posterior probability responds.

Be liberal in your experiments. The three parameters are independent, so as long as you stay between 0 and 1, you are free to explore all sorts of combinations.

We have already started in the previous post. We shall do more here, where we begin by deriving Bayes Theorem from its most basic elements: the observations.

Let’s use again the rain example. Assume we take 100 observations and place them in the four quadrants of the following 2x2 table:

The two columns represent the possible states of hypothesis H: True – Rain tomorrow; False – No Rain tomorrow. The two rows represent the possible states of Evidence E: Positive – The weather report forecasts Rain; Negative – The weather report forecasts No Rain. Starting from the first quadrant on the top left, and moving clockwise using the numbers assumed in the example, we place the 100 observations according to their joint outcomes:

True Positives: It rained and the report rightly forecast it would.

False Positives: It did not rain but the report wrongly forecast it would.

True Negatives: It did not rain and the report rightly forecast it would not.

False Negatives: It rained but the report wrongly forecast it would not.

We can have as many observations as we want. We then normalise the result by dividing the four numbers by their row totals, thus deriving the four conditional probabilities of the Evidence given the Hypothesis:

where:

P(E|H)=1/1=100%. We call this the True Positive Rate (TPR)

P(E|H̄)=10/99=10.1%. We call this the False Positive Rate (FPR)

P(Ē|H̄)=89/99=89.9%. We call this the True Negative Rate (TNR)

P(Ē|H)=0/1=0%. We call this the False Negative Rate (FNR)

In general, we have:

We call them Likelihoods. Notice that, by definition, TPR=1-FNR and FPR=1-TNR. But there is no direct relationship between TPR and FPR – that’s why you can freely experiment with them.

One last definition. Notice that the prior probability P(H) is equal to the total number of observations in the H column over the total number of observations in the table: 1/100=1% in our example. P(H) is the probability of H regardless of evidence E. It is therefore also known as the Base Rate (BR).

Putting all together, and denoting the Posterior Probability P(H|E) as PP, we can rewrite Bayes formula as:

Here we are – we got to the final destination. I hope this version of the formula does not look as foreboding as the original.

We are now ready to start looking at the many remarkable features of Bayes Theorem and to analyse its crucial, fruitful and far-reaching properties.

Read more

It’s a beautiful summer day in Rome, 30 degrees, blue sky. Question: What is P(H), the probability that it will rain tomorrow? Answer: Very low – let’s say 1%. Rain is highly unlikely.

But then suppose that in the evening we start seeing some grey clouds on the horizon. The TV weather reporter says there is a slow-moving precipitation band 50 km upwind, aligned with prevailing winds, and warns that, as a result, it is probable that we are going to see some rain tomorrow. That’s confirmed by the cloud icons we see on our phone weather app.

Question: What is P(H|E), the probability that it will rain tomorrow in the light of this new evidence E? Clearly, it is going to be higher that 1%. But how much higher? Let’s see.

Before we do that, I warn you: you will see some equations. Famously, in the preface to A Brief History of Time, Steven Hawking wrote: Someone told me that each equation I included in the book would halve the sales. I therefore resolved not to have any equations at all. But I will not heed his advice. Instead, I will ask you to make an investment. None of what follows is complicated – you just need to sit down with pencil and paper and understand the main concepts. To the mathophobes among you: please don’t be afraid. Think of maths as nothing more than a shorthand language. Instead of saying: the probability of the hypothesis is one in a hundred, we write P(H)=1%. This has three key advantages:

1. It is compact

2. It is universal – the same in all languages

3. It makes it easier to draw logical consequences that would mostly be lost in a natural language discourse.

So here we go. Write to me if you need help.

H: It will rain tomorrow

E: The weather report predicts probable rain

Read more

We start with P.

P stands for Probability. We all know what that means. When we say that something is probable, we mean that it is possible, it may happen, it may be true. P measures the degree to which that is the case. It ranges from 0 to 1. P=0 (or 0%) means that the thing is impossible, it cannot happen, it can’t be true. P=1 (or 100%) means that it is certain, it must certainly happen, it is certainly true. In the middle between the two extremes, P=0.5 (or 50%) means that the thing is equally probable, as likely to happen or not happen, as likely to be true or false.

Along the scale from 0 to 1, we use a variety of terms to describe our perceptions of different degrees of probability. Building on a CIA memo written in 1964, the following graph (available here) lists many of them:

We are all familiar with these expressions. Yet, as the graph – known as a joyplot – shows, we do not attach the same numerical value to them. When we say, for instance, that something is “almost certainly” true, we clearly have a high probability in mind. But whether that means 80% or 90%, 99%, or some other value in the upper range, varies from person to person and depends on context and circumstances. This is entirely normal in ordinary language, where words – especially those referring to abstract concepts – exist within a halo of vagueness. What is a chair? What is love? What is green?

What is probability?

If you studied Probability Theory in any form, you have a ready answer: probability is the relative frequency with which something happens or is true in a large number of identical, repeatable trials. For example, if we toss a fair coin many times under identical conditions, the fraction of Heads converges to 0.5. Probability Theory started in the 17^th century as a study of hard evidence, resulting from controlled, replicable experiments, leading to the measurement of undisputable objective probabilities, grounded on empirical frequencies.1

But this is not the kind of evidence we will encounter in Bayesland. So please take your coins, dice and card decks and put them aside.

Most of the evidence we will encounter in Bayesland is soft. Soft evidence does not arise from measurable regularities. Rather, it is shaped by the observer’s perception and filtered through his judgement and interpretation, giving rise to subjective probabilities such as those depicted in the joyplot. These probabilities are grounded in beliefs, confidence, and trust.

Probability in Bayesland is a primal concept which, since the dawn of civilisation, has been inherently tied to the evaluation of soft evidence.2 The Law is the original domain in which the concept of probability has been moulded through time. Law needs to reach a decision, i.e. separate true from false, guilt from innocence. To reach a verdict is, literally, verum dicere: to speak the truth. Over the centuries, different legal systems defined different rules of evidence but relied on the same underlying relationship between beliefs and trust. Now as then, our confidence rests on the judgement of reputable authorities. Indeed, the very word probability comes from the Latin probabilis, which means ‘worthy of approbation‘ 3. Approbation comes from the recognized probity of honest, honourable people, who are thus capable of influencing our beliefs by approving a hypothesis as true or false.

A tug of war between probi viri: this is probability in Bayesland.

Before we begin, let’s give some structure to what has so far been a deliberately vague introduction.

Read more