In the months following my most recent post about performance, I’ve been noodling on one key aspect of the challenge: if we take a more outcome (rather than output)-based approach to evaluating performance, how do we separate outcomes caused by luck and outcomes caused by skill?
I started off reading Annie Duke’s “Thinking in Bets” which have been sitting in my queue and received good feedback from colleagues. I finished the book somewhat disappointed having gained some good insights on pursuing the truth and having a stronger decision-making process, but not a lot that pertained to the assessment of performance and untangling luck and skill. However, Duke did mention another book with a rather promising title by Michael Mauboussin:
Weary of committing another big chunk of time to a book on this topic, I decided to look for more lightweight mediums and was able to find this Talks at Google video for the more audio-visual inclined and this 25iq blog post for the more textual inclined. Both provide good summaries of the major themes covered in the book. There’s a lot there, including lots of interesting tangents in their own right, but I’ll try to focus on one arc that’s relevant to my own area of inquiry.
Different domains of performance fall on a spectrum between pure luck and pure skill, but all of them have a combination of some luck and some skill.
As a domain evolves, it becomes more dependent on luck than skill. That’s not because skill matters less, but because knowledge dissemination happens more quickly and cheaply, causing skill to be distributed more uniformly. Mauboussin refers to that phenomenon as “The Paradox of Skill”.
The strategy for “how to get better” also varies depending on where the performance domain falls on the luck-skill spectrum. The closer it is to the skill edge of the spectrum, a deliberate practice strategy will yield better outcomes. The closer it is to the luck edge of the spectrum, the emphasis needs to be on a strong decision-making process. The latter helps frame Duke’s book more clearly: since poker falls closer to the luck edge of the spectrum, the heavy emphasis on the decision-making process makes a lot of sense.
It’s worth noting, however, that this point does not seem to be corroborated with a lot of evidence, at least in the resources that I reviewed (it may be treated differently in the book).
Mauboussin offers the following criteria for evaluating the process:
- Analytical — finding edge and figuring how much to bet on that edge.
- Behavioral — understanding the common biases we all tend to fall for and weaving into the process methods to mitigate and manage those.
- Organizational — avoiding “agency costs” (misalignment of incentives). Is the organization helping or impeding the quality of the decision?
So where does all of this leave us with regards to performance management?
- It supports the claim that variance in outcomes may have more to do with luck than skill.
- This gets compounded in more mature domains where the “Paradox of Skill” is in full effect.
- It supports the shift from focusing on the outcome to focusing on the process when evaluating performance.
Did it solve our problem? No. Did it get us closer to a solution? Yes. Baby steps…