Posted on | January 28, 2013 | 5 Comments
In an earlier blog post (here) we wrote that, given the evidence available to us at the time, Lance Armstrong was probably not guilty of doping. The main line of our argument was that hundreds of doping tests from certified laboratories using accepted procedures had not found dope. We restricted the evidence to this, in the same way a court of law restricts the evidence that can be used to determine the outcome of a trial.
We restricted the evidence for two reasons. First, the definition of “doping” is that an athlete fails the test on both halves of a split sample. If an athlete passes, anti-doping officials should leave him or her alone or athletes will never be able to rise above suspicion. Second, the evidence we excluded was weak anyway (the testimony of other athletes, and EPO tests that could not be reproduced).
We believe that we made the right inference—a restricted inference—about Armstrong at the time. In other words, it was the right inference to make with the information we had and chose to use. Then Armstrong confessed to doping. So we were inferentially right and factually wrong. There is no contradiction here—it happens all the time, from flipping coins to rating bonds.
Suppose someone hands you a bent coin and tells you that he just flipped it 1,000 times. It came up 1/3 of the time one way (either heads or tails) and 2/3 the other way (either tails or heads). But, crucially, he does not tell you which. Now, he asks, what probability would you assign to getting a head in the next flip? (We’re assuming for the sake of this argument that there is no attempt to control or direct the outcome of the coin flipping: that is, it is flipped in the usual, vigorous way.)
What would you answer? You are missing some key information. You have no choice but to answer ½—even though you know that if you flip the coin another 1,000 times you will not get some number of heads close to 500 but closer to either 333 or 667! The assignment of a ½ probability is inferentially right even though you know that it does not correspond to the outcome you will get if you flip the coin many times.
There is no need to flip the bent coin another 1,000 times before updating your probability assignment of ½. After each flip you can—should—update your assignment of probability. It won’t take long for you to reach a new probability assignment of either 1/3 or 2/3. That does not make the original ½ assignment wrong—it was the right conclusion at the time.
When rating agencies rate bonds they use similar reasoning. When they give an initial rating they use all the information available to them at the time and come up with a rating such as AAA. As the conditions change for the bond issuer and new information comes to light—financial problems, economic problems—the rating agency updates the rating.
Suppose the agency downgrades the bond to AA+. Then two weeks later the bond issuer defaults and declares bankruptcy. Was the rating wrong? (Assume there is no corruption or collusion and that it is an honest rating). No—it was the right inference at the time.
There is a quotation attributed a variety of people, including Keynes, Samuelson and Churchill that sums things up: “When the Facts Change, I Change My Mind. What Do You Do, Sir?”
By Phil Green and George Gabor, co-authors of misLeading Indicators: How to reliably measure your business. © 2013 Greenbridge Management Inc.