misLeading Indicators: How to reliably measure your business

Lance Armstrong doping case and bond defaults show challenges of probabilistic reasoning

Posted on | January 28, 2013 | 5 Comments

In an earlier blog post (here) we wrote that, given the evidence available to us at the time, Lance Armstrong was probably not guilty of doping. The main line of our argument was that hundreds of doping tests from certified laboratories using accepted procedures had not found dope.  We restricted the evidence to this, in the same way a court of law restricts the evidence that can be used to determine the outcome of a trial.

Lance Armstong

Lance Armstong. Photo: Brad Cambert/Shutterstock.com

We restricted the evidence for two reasons. First, the definition of “doping” is that an athlete fails the test on both halves of a split sample. If an athlete passes, anti-doping officials should leave him or her alone or athletes will never be able to rise above suspicion. Second, the evidence we excluded was weak anyway (the testimony of other athletes, and EPO tests that could not be reproduced).

We believe that we made the right inference—a restricted inference—about Armstrong at the time. In other words, it was the right inference to make with the information we had and chose to use. Then Armstrong confessed to doping. So we were inferentially right and factually wrong. There is no contradiction here—it happens all the time, from flipping coins to rating bonds.

Suppose someone hands you a bent coin and tells you that he just flipped it 1,000 times. It came up 1/3 of the time one way (either heads or tails) and 2/3 the other way (either tails or heads). But, crucially, he does not tell you which. Now, he asks, what probability would you assign to getting a head in the next flip? (We’re assuming for the sake of this argument that there is no attempt to control or direct the outcome of the coin flipping: that is, it is flipped in the usual, vigorous way.)

What would you answer? You are missing some key information. You have no choice but to answer ½—even though you know that if you flip the coin another 1,000 times you will not get some number of heads close to 500 but closer to either 333 or 667!  The assignment of a ½ probability is inferentially right even though you know that it does not correspond to the outcome you will get if you flip the coin many times.

There is no need to flip the bent coin another 1,000 times before updating your probability assignment of ½. After each flip you can—should—update your assignment of probability. It won’t take long for you to reach a new probability assignment of either 1/3 or 2/3. That does not make the original ½ assignment wrong—it was the right conclusion at the time.

When rating agencies rate bonds they use similar reasoning. When they give an initial rating they use all the information available to them at the time and come up with a rating such as AAA. As the conditions change for the bond issuer and new information comes to light—financial problems, economic problems—the rating agency updates the rating.

Suppose the agency downgrades the bond to AA+.  Then two weeks later the bond issuer defaults and declares bankruptcy.  Was the rating wrong? (Assume there is no corruption or collusion and that it is an honest rating). No—it was the right inference at the time.

There is a quotation attributed a variety of people, including Keynes, Samuelson and Churchill that sums things up: “When the Facts Change, I Change My Mind. What Do You Do, Sir?”

 

By Phil Green and George Gabor,  co-authors of misLeading Indicators: How to reliably measure your business. © 2013 Greenbridge Management Inc.

Comments

5 Responses to “Lance Armstrong doping case and bond defaults show challenges of probabilistic reasoning”

  1. The bond trader’s fallacy : misLeading Indicators: How to reliably measure your business
    January 28th, 2013 @ 6:25 pm

    [...] Lance Armstrong doping case and bond defaults show challenges of probabilistic reasoning [...]

  2. Timo Elliott
    January 29th, 2013 @ 1:34 pm

    Phil,

    The challenge of probabilistic reasoning is very real, and it’s a common problem, so it’s great that you discuss it here.

    But I’m not sure it applies in this case…

    I remember being surprised at your original post, because I didn’t think Armstrong having passed all the tests was a strong data point for innocence.

    Part of your reasoning in the original post was that “the tests were done by certified labs and were therefore, without evidence to the contrary, reliable”

    However, his teammates had passed those same tests, only to later admit that they had been doping.

    The logical conclusion was that the tests were indeed unreliable, and shouldn’t be heavily relied upon when weighing the evidence.

    I think therefore that this falls into “inferentially wrong” in this case — no?

  3. Phil Green
    January 29th, 2013 @ 2:18 pm

    Timo: You say “the logical conclusion was that the tests were indeed unreliable, and shouldn’t be heavily relied upon when weighing the evidence.”

    The logical conclusion…when? Now, when we know they were unreliable? Back then we didn’t know that, and we made a decision to exclude the testimony of other athletes as (1) not part of the dope testing procedure and (2) weak evidence. We had an accepted procedure that was followed repeatedly, always yielding negative test results. The USDA went outside of this procedure, in effect redefined it in an ad hoc manner. Hindsight is 20/20. That was the point of the blog post.

    So it was the right inference with the test procedure and information available at the time, even though it turned out it was factually incorrect.

  4. BI in Barcelona, Scary Big Data, Cool 3D Analytics and More | Business Analytics
    February 2nd, 2013 @ 1:16 pm

    [...] fascinating post by Phil Green, co-author of Misleading Indicators, shows that there can be lots of debate about why decisions go wrong (see the [...]

  5. BI in Barcelona, Scary Big Data, Cool 3D Analytics and More
    February 4th, 2013 @ 12:44 am

    [...] fascinating post by Phil Green, co-author of Misleading Indicators, shows that there can be lots of debate about why decisions go wrong (see the [...]

Leave a Reply







  • Buy Your Copy

    On sale Feb 29, 2012

    Price $48

    Buy from Barnes and Noble

    (In stock now).

    Buy from Praeger (Publisher).

    (In stock now)

    Order ebooks 1-800-368-6868 (7am to 430pm PST, Mon-Fri) or 805-968-1911 ext 312

    Order by email


    Buy Amazon US

    Buy from Amazon.com
    In stock.

    Buy Amazon Canada

    Buy from Amazon.ca
    In stock

    Read the first two chapters: Click here.

  • Tag Cloud

    advsertising baseball bonds cancer crime rate customer service dashboard debt definitions doping earnings exception reporting Facebook fashion industry fraud gambler's fallacy gas pump GDP gold government statistics happiness hurricane investing in measurement jobs leading indicator management medical studies mineral resources Obama on-line p-value poverty probability productivity profit public health randomization Romney safety science silver six sigma stock market US Election wealth
  • Archives