misLeading Indicators: How to reliably measure your business

How to measure compliance?

Posted on | January 24, 2011 | No Comments

A Google search of the title of this post turns up web pages on all sorts of different types of “compliance” people measure: security compliance, regulatory compliance, Sarbanes-Oxley compliance, environmental compliance, safety compliance, quality control compliance, privacy compliance, tax compliance and even hand-washing compliance.  When many people measure compliance they are actual counting violations of some rule or standard. Technically they are counts, not measurements.  So it is slightly misleading to say that such counts are compliance measurements, if they are not qualified.

Counting violations can lead to an interesting conundrum.  Suppose someone in a municipal water quality plant wants to measure compliance with water quality standards. For arguments’ sake, let’s say there are ten standards for things such as bacteria, turbidity, acidity, colour, lead concentration, and so on.  Many of these things are measured on-line at extremely high frequencies. But some are measured in a laboratory at a much lower frequency.   The water plant sends ten samples to the lab for analysis. What is the percentage compliance?

 The test results that come back from the lab are shown below. The ten samples are labelled 1, 2 and so on. The ten standards are labelled A, B, C and so on.  A “one” in the table below indicates compliance and a “zero” indicates non-compliance.  What is the percent compliance?



Sample  A  B  C  D  E  F  G  H  I  J
1 1 1 1 1 1 1 1 1 1 0
2 1 1 1 1 1 1 1 1 0 1
3 1 1 1 1 1 1 1 0 1 1
4 1 1 1 1 1 1 0 1 1 1
5 1 1 1 1 1 0 1 1 1 1
6 1 1 1 1 1 1 1 1 1 1
7 1 1 1 1 1 1 1 1 1 1
8 1 1 1 1 1 1 1 1 1 1
9 1 1 1 1 1 1 1 1 1 1
10 1 1 1 1 1 1 1 1 1 1


Many people would say there is 95% compliance from these results, based on 100 tests and 5 non-compliance results. Other people would say they had 100% compliance on standards A, B, C, D and E and 90% compliance on F, G, H, I and J. There may be situations where these sorts of compliance measurements make sense, and there is often a pretty strong incentive to argue that they do, especially if there are consequences for low compliance numbers. But often such compliance indicators are misleading.

If each of those samples were a glass of water you would look at the numbers differently, especially if you were the one drinking the water. There were five good glasses, and five glasses that failed at least one test. In other words, 50% of glasses of water complied with all water quality standards—and 50% did not.  Which would you drink?

You can measure compliance without counting if the raw data used to  determine compliance is a measurement; for example pH is used to measure acidity. If you have such measurements it is more informative to use the original measurements rather than “measuring” compliance by turning them into counts of violations.

When the police use radar to measure your vehicle’s speed they are ultimately trying to determine if you comply with the law. But they record your actual speed on your traffic ticket too, and in many jurisdictions the speed determines the amount of your penalty.


Leave a Reply

  • Buy Your Copy

    On sale Feb 29, 2012

    Price $48

    Buy from Barnes and Noble

    (In stock now).

    Buy from Praeger (Publisher).

    (In stock now)

    Order ebooks 1-800-368-6868 (7am to 430pm PST, Mon-Fri) or 805-968-1911 ext 312

    Order by email

    Buy Amazon US

    Buy from Amazon.com
    In stock.

    Buy Amazon Canada

    Buy from Amazon.ca
    In stock

    Read the first two chapters: Click here.

  • Tag Cloud

    advsertising bonds compliance cooking data credit rating customer complaints customer service debt definitions deflation earnings election errors experimentation fraud GDP gold government statistics happiness investment key performance indicator Kord Kelvin leading indicator media medical studies mineral resources Obama Olympics on-line p-value polls probability profit public health quality of life randomization risk Romney safety science statistical process control stimulus unmeasurable wealth website traffic
  • Archives