Posted on | January 24, 2011 | No Comments
A Google search of the title of this post turns up web pages on all sorts of different types of “compliance” people measure: security compliance, regulatory compliance, Sarbanes-Oxley compliance, environmental compliance, safety compliance, quality control compliance, privacy compliance, tax compliance and even hand-washing compliance. When many people measure compliance they are actual counting violations of some rule or standard. Technically they are counts, not measurements. So it is slightly misleading to say that such counts are compliance measurements, if they are not qualified.
Counting violations can lead to an interesting conundrum. Suppose someone in a municipal water quality plant wants to measure compliance with water quality standards. For arguments’ sake, let’s say there are ten standards for things such as bacteria, turbidity, acidity, colour, lead concentration, and so on. Many of these things are measured on-line at extremely high frequencies. But some are measured in a laboratory at a much lower frequency. The water plant sends ten samples to the lab for analysis. What is the percentage compliance?
The test results that come back from the lab are shown below. The ten samples are labelled 1, 2 and so on. The ten standards are labelled A, B, C and so on. A “one” in the table below indicates compliance and a “zero” indicates non-compliance. What is the percent compliance?
Many people would say there is 95% compliance from these results, based on 100 tests and 5 non-compliance results. Other people would say they had 100% compliance on standards A, B, C, D and E and 90% compliance on F, G, H, I and J. There may be situations where these sorts of compliance measurements make sense, and there is often a pretty strong incentive to argue that they do, especially if there are consequences for low compliance numbers. But often such compliance indicators are misleading.
If each of those samples were a glass of water you would look at the numbers differently, especially if you were the one drinking the water. There were five good glasses, and five glasses that failed at least one test. In other words, 50% of glasses of water complied with all water quality standards—and 50% did not. Which would you drink?
You can measure compliance without counting if the raw data used to determine compliance is a measurement; for example pH is used to measure acidity. If you have such measurements it is more informative to use the original measurements rather than “measuring” compliance by turning them into counts of violations.
When the police use radar to measure your vehicle’s speed they are ultimately trying to determine if you comply with the law. But they record your actual speed on your traffic ticket too, and in many jurisdictions the speed determines the amount of your penalty.