Has this ever happened to you? It happened to a friend of mine. You open a performance report, one that's been sent to your boss and all the top executives in your company, and it states that your group is failing to meet company mandated quotas and therefore its performance is poor. Your heart sinks and you think, I don't need this.
Then, knowing how hard you and your team have been working in this particular area, you take a closer look at the numbers and realize that all the data points in the report are wrong. In some instances, the real-life numbers are multiples of what's stated in the report. The creators of the report are not willing to make any changes to their data sourcing or methodology. You understand the need for quotas and the drive to "move the needle," but the inaccuracy and apparent unfairness of the reports leave you feeling helpless and betrayed.
It's a side effect of data-driven decisionmaking. Data-based decisions sound so good, so democratic and fair. But much depends on what is being measured and how, and the soundness of the underlying data. Without solid and well identified data sources, numbers can easily be miscounted, misallocated and misattributed.
And methodologies are often flawed. In a recent New York Times article, Steve Lohr wrote, "The problem is that a math model, like a metaphor, is a simplification. This type of modeling came out of the sciences, where the behavior of particles in a fluid, for example, is predictable according to the laws of physics. In so many Big Data applications, a math model attaches a crisp number to human behavior, interests and preferences." The article quotes Claudia Perlich, chief scientist at online ad company Media6Degrees, saying, "You can fool yourself with data like you can't with anything else. I fear a Big Data bubble." That bubble, she says, would involve a rush of people calling themselves "data scientists," doing poor work and giving the field a bad name.
Four efforts would improve data-driven decisions.
1. Vet the performance metrics. Are your key performance indicators the right ones? Say you measure employee performance by number of tweets posted per day. One person might meet the quota with irrelevant or poorly craft tweets that fail to advance the company's goals, or worse, an automated feed. Another might tweet three times a day but create engagement, generate followers and boost the brand.
2. Check the accuracy of the underlying data. If the data are wrong, people will sooner or later begin to notice and ignore the reports. Have an unbiased, independent party conduct a reality check against original sources before you take a report live.
3. Use your intuition. If numbers feel wrong, they probably are and at the very least are worth a closer look.
4. Be the squeaky wheel. Someone has to speak up for the truth, or an ugly precedent will be set. Ideally - and this is the hard part - one does this with discretion and good will.
Without such efforts, false reports become a cultural illness that can breed misunderstandings and poor decisions.
