On Metrics and KPIs

Metrics, also referred to as Key Performance Indicators (KPIs) in more limited contexts, are a staple of the modern business world.

It’s considered common wisdom and “best practice” that “you can’t manage what you can’t measure”, and therefore that you should measure it. It is also fairly well accepted that the measurement in an of itself contains little insight, and that the insight is generated from comparing the actual measurement, to a forecasted target that was set ahead of time. This is roughly where common wisdom stops, leaving out several critical aspects of using metrics effectively. I’d like to cover two of these aspects today.

1. Targets must be relative and dynamic

The Beyond Budgeting community wrote a lovely whitepaper on this topic, and Niels Pflaeging wrote a more succinct post about this. Here’s the gist:

[I]t is impossible to set a target in advance that represents ‘good performance’. The modern world is complex and dynamic making prediction impossible, and we know that we will never face the same set of circumstances twice. The economy changes, as do our customers and competitors, so we can do no more than hazard a guess at ‘what good looks like’

[A]ny measure of reality will always contain noise: the impact of an unknowable number of random or irrelevant events, which distort and disguise ‘real’ performance (the signal)… [I]f we do not have the ability to measure the level of noise, we have no scientific basis to distinguish between something that is safe to ignore and something that we should be acting upon.

…so most of what passes as performance analysis is the result of comparing a guess with noise!

Worry not. This does not mean that any metric and any measurement is useless. The folks at BB also offer a solution:

The only way that we can assess performance in a truly meaningful way is to compare ourselves to peers that have faced the same set of conditions.

Targets, should therefore be relative: at the business level — compared to external competition; Internally —  compared to other teams/departments if feasible. Here’s a good example from StatOil, defining its shareholder-return target to be “above peer average” and its return-on-capital target to be “in the top-3 of peer group”.

2. Metrics should be paired

“If you give a manager a numerical target, he’ll make it even if he has to destroy the company in the process.” -W. Edwards Deming

Leaving the “gaming the system” challenge for a different post, on a less malicious level, a key challenge with metrics is their reductionist nature. Over-orienting behavior towards a view that looks at the world through a single number is unlikely to lead to a positive outcome. Here as well, “not measuring” is not the only solution. And you don’t have to go full-on “balanced scorecard” either. Keith Rabois offers a simple idea that’s more lightweight to implement:

One important concept is pairing indicators. Which is, if you measure one thing and only one thing, the company tends []to optimize to that. And often at the expense of something that is important… It’s really easy to give the risk team the objective and say, we want to lower our fraud rate. It sounds great. Until they start treating every user in this audience as a suspect because they want to lower the fraud rate. So they require each of you to call them up on the phone and give them more supplemental information and fax in things. Then you have the lowest fraud rate in the world, you also have the lowest level of customer satisfaction score.

What you want to measure at the same rate as your fraud rate, is your false positive rate. That forces the team to actually innovate. Similarly, you can give recruiters [volume] metrics around hiring. And guess what? You will have a lot of people come in for interviews. But if you are not tracking the quality of interviewers, you may be very unhappy about the quality of people you are hiring and giving interviews to. So you always want to create the opposite and measure both. And the people responsible for that team need to be measured on both.

On Metrics and KPIs

Rethinking Uncertainty [Harbinger]

AoC Toolbox: Becoming Friendly with Uncertainty by Jordan Harbinger

One of the downsides of a super-short commute, and choosing to exercise without a pair of earphones, is that podcasts are not part of my daily routine. By they are certainly part of my vacation routine, usually on the long drives to/from my vacation destinations.

I’m not sure how I first came across The Art of Charm podcast, but deciding to withhold judgement on the dubious branding was a wise decision, since it’s a great resources for psychology and personal growth nerds like myself.

Recently, Jordan and team did a great piece about uncertainty.

They started by providing a great definition for what uncertainty is:

Uncertainty is a function of the availability of information — how much we want vs. how much is available to us. A gap in information creates uncertainty and makes it harder for us to understand and control the world around us. The less control we feel, the more stability we crave, and the fewer new experiences and stimuli we seek out. [Furthermore,] Uncertainty has an informational aspect (the data gap leading to uncertainty) and a subjective experience (how it feels on a gut level to be uncertain).

They then shared an interesting study, suggesting that uncertainty is neither good or bad, but rather uncertainty acts as an emotional amplifier: good events feel better, while bad events feel worse.

Which led them to a novel insight:

Because not only is uncertainty a fundamental constant in life, it’s actually one of the most helpful and productive environments available to us. After years of studying this stuff, I’m convinced that it’s not uncertainty we need to move beyond, but our aversion to it. That’s what most self-help approaches don’t understand — that by trying to avoid uncertainty, we’re only increasing it, and missing out on a huge opportunity.

Once we stop turning uncertainty into the enemy, we can begin to look at it, understand it, even start to enjoy it. And rather than fleeing from it, we can actually invite it in, and use it to our advantage.

This insight suggests a practical technique for dealing with situations in which we are emotionally overwhelmed by uncertainty:

What the brain doesn’t care about is whether that information is true, useful, or even important at this moment. The brain is simply wired to consume data in any form. When it knows that there’s more data out there, it throws itself into a cognitive panic.

The information gap is real, but that doesn’t mean it’s important. The fact is, we live in a world that never gives us enough information, and we operate (very well, in fact) despite it.

So once you catch your brain obsessing over the information gap, ask yourself these two questions.

Can I actually get this information?

Do I actually need to know this information right now?

You’ll be amazed how often your brain will hunger for information it can’t obtain, to answer questions it doesn’t need to ask.

With that perspective, you’re free — free to stop obsessing, and free to start focusing only on the information that can actually serve you right now.

On a deeper level, the team also suggests a powerful mindset shift — testing and eventually changing a deeper belief/assumption that uncertainly exists to throw you off your stable progression through life. And instead choosing to trust that uncertainty exists to serve you:

Even for those of us who are naturally anxious about change, knowing that change will ultimately fuel our growth makes it easier to take in stride. Trusting that uncertainty is designed not just to throw you, but to make you a better person, is an essential step in embracing it.

Over time, that trust will turn into excitement. After a few cycles of uncertainty leading to personal growth, new challenges will carry a hidden promise: a new problem, a new set of skills, and a new identity waiting for you on the other side.

Rethinking Uncertainty [Harbinger]