I immediately saw that there was a significant correlation between the independently collected error rates and the measures of team effectiveness from my survey. But then I looked closely and noticed something wrong. The direction of the correlation was exactly the opposite of what I had predicted. Better teams were apparently making more – not fewer – mistakes than less strong teams. Worse, the correlation was statistically significant. I briefly wondered how I could tell my dissertation chair the bad news. This was a problem.

No, it was a puzzle.

We see similar patterns on all eight of the questions—we see, in other words, that when we zero in on the critical aspects of our experience at work, they vary more team-to-team than they do company-to-company. Any ideas—like the idea of culture—that rest on the assumption that our experience of a company is uniform, no matter where we sit, don’t hold up. Any ideas—again, like the idea of culture—that rest on the assumption that our experience will vary company to company are incomplete, because our experience will vary more within a company than between companies.

And because competencies are unmeasurable, it is impossible to prove or disprove the assertion that everyone who excels in a particular job possesses a particular set of competencies. It is equally impossible to show that people who acquired the competencies they lacked outperformed those who did not—that, in other words, well-rounded people are better. These two statements together are the foundation for most of what companies do to develop the talents of their people, yet each of them is unfalsifiable—you will find no academic papers in any peer-reviewed journal proving the necessity of possessing certain competencies, and no proof that acquiring the ones you lack nets you any increase in performance. Both of these assertions, despite the good intentions that created them, are conjured from thin air—and we can never know if they are correct.

And, as we’ve seen in this chapter, the problem with almost all data relating to people—including you—is that it isn’t reliable. Goals data that reports your “percent complete”; competency data comparing you to abstractions; ratings data measuring your performance and your potential through the eyes of unreliable witnesses: it wobbles by itself, and fails to measure what it says it’s measuring.

There are limits to data, however, and some people rely on it too heavily. Analyzing it correctly is difficult, and it is dangerous to assume that you always know what it means. It is very easy to find false patterns in data. Instead, I prefer to think of data as one way of seeing, one of many tools we can use to look for what’s hidden. If we think data alone provides answers, then we have misapplied the tool. It is important to get this right. Some people swing to the extremes of either having no interest in the data or believing that the facts of measurement alone should drive our management. Either extreme can lead to false conclusions.

“You can’t manage what you can’t measure” is a maxim that is taught and believed by many in both the business and education sectors. But in fact, the phrase is ridiculous - something said by people who are unaware of how much is hidden. A large portion of what we manage can’t be measured, and not realizing this has unintended consequences. The problem comes when people think that data paints a full picture, leading them to ignore what they can’t see. Here’s my approach: Measure what you can, evaluate what you measure, and appreciate that you cannot measure the vast majority of what you do. And at least every once in a while, make time to take a step back and think about what you are doing.