Measurement 101

I love measuring things.

Maybe it’s because I was deprived of good quantitative data during my formative years studying a liberal arts degree.

Or maybe it’s because the first thing I ever seriously tried to measure was my own work performance.

I was designing a marketing campaign and I wanted to find a way to measure whether my campaign was working. After all, how else could I possibly know if I was doing a good job? And if I didn’t know that, how could I possibly improve?

It seemed so self-evident, I naively assumed that was how business worked and everyone would think the same way. You’re probably not surprised to learn this initial enthusiasm was challenged very quickly as I ran headlong into all sorts of difficulties.

Maintaining the passion to navigate through these difficulties can teach a lot about obstacles and limitations facing measurement in complex organisations, and how to do it well.

People resist measurement

People present the main obstacle to organisational measurement. Sometimes they actively go out of their way to thwart, subvert or manipulate measurement systems for perceived protection or personal gain. Sometimes people are just hard to measure because they’re so inherently complex and changeable.

One of the lessons that emerged from my very earliest experiments was how much people tended to view the whole idea of measurement with fear and suspicion. The most common question, fired from beneath knitted brows, was, “Why do you want to do that?”

Even if the thing I was trying to measure bore no direct relation to the person I was speaking to, they were still quite likely to perceive my efforts as setting a dangerous precedent, best nipped in the bud.

How to deal with it
One of the many reasons people behave this way is because measurement is almost always a precursor to some sort of change. There’s a huge amount written about why people resist change, and what to do about that. So let’s say that’s adequately covered elsewhere, and park it as a subject for future posts.

Where are you going with all that data?

Organisations struggling with measurement are often focusing at the wrong end, on the data collection itself. This can result in complex, unwieldy processes to amass piles of very prescriptively defined data that aren’t being used for anything very important.

These piles of data are often nurtured and protected from disturbance or exploitation by gate keepers. Like sphinxes, these guardians seek to confuse the unwary and will gleefully waste hours discussing statistical validity, non-random samples, longitudinal issues, and other ‘statistrivia’, to scare people away from using the data pile to make decisions.

This sort of behaviour gives measurement a bad name and, as indicated above, people often feel threatened by the whole concept of measurement anyway, so this makes it all too convenient to just forget about the whole thing and go back to comfortable fumbling around in the dark.

How to deal with it
It’s usually better to start at the other end: what are we trying to achieve? Then see if any metrics already captured in the financials, or HR, or IT systems are pointing in the right direction to enable better decision making. When dealing with gate keepers, I’ve tried garlic and crucifixes, with limited success – let me know if you have a better solution.

Rules of thumb for measurement

  1. There is no such thing as 100% certainty.
  2. Human beings always make decisions based on imperfect or incomplete data – so get used to it.
  3. Some data is better than no data.
  4. Approximate data is better than no data.
  5. The quality of the decision making process is more important than making sure the data is accurate to 15 decimal places.
  6. It’s easier, faster and cheaper to use data you already have than to create a new measurement system.
  7. Combine measures of quantity with measures of quality, to make sure you know what you’re measuring.
IT service desk flowchart

This is why 'closed jobs' on its own is an inadequate measure of IT service desk performance

This last point may seem obvious, but it’s not. For example, lots of IT service desks use the number of jobs closed as a measure of performance. This would be great if ‘closed job’ = ‘problem solved’ = ‘satisfied customer’.

But this is not true, because the IT service desk staff usually determine when a job is closed, not the customer. So this provides an incentive for staff to close off all jobs as fast as possible, even though the problem isn’t solved.

When the customer complains, the service desk just opens another job, the cycle repeats, and the service desk staff look good, because they’re smashing the ‘closed jobs’ KPI.

So measuring quantity by itself is not enough. You also need an accompanying measure of quality to understand what’s actually being measured.

About Jonathan Smith

Turning strategy into reality
This entry was posted in Decreasing costs, Increasing revenue, Positioning for the future, Quality, Two-way communication. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s