Long time readers of this blog know that I hold some pretty strong opinions about the ill-use of surveys. More recently I wrote about it in “Working on Work” and my guest contribution in The Ready magazine, “Ending the Tyranny of the Measurable”. At a more tactical level, the overuse and abuse of the Likert scale have a prominent spot on my rather short pet peeves list.
I had a chance to put my $$$ where my mouth is, working on a recent project where surveying was essential to getting the insights I was looking for. It was a short (17 questions) survey about remote work where I wanted to learn from long-time practitioners about their perspective on the key advantages, challenges, and essential practices of remote work. This was in early Feb 2020, mind you, pre -COVID-19 .
I was looking for relative insights: which advantage do practitioners find to be the biggest advantage? which challenges do practitioners find to be the biggest challenges? etc. Rather than use a series of Likert scale questions where respondents would have to rate each advantage on a 1-to-5 scale from “not important” to “very important”, I used a single question asking respondents to sort/rank the advantages from the one most meaningful to them, to the one least meaningful to them.
The end results, courtesy of SurveyGizmo’s beautiful reporting feature looked like this:
In addition to immediately getting the relative ranking, which was what I was looking for, it was also super easy to glean some high-level statistical insights on the responses: the advantages had more differentiated winners (differences in the overall score between #1,2, and 3) and responses were more consistent — most people chose the same advantages to be the top ones, as indicated by the overly positive “rank distribution”. The top challenges, on the other hand, won by a very small margin and responses were more polarized — some chose the winners as the top challenges while others didn’t, as indicated by the rather neutral/even “rank distribution”.
Top-3 mastered practices:
Top-3 critical practices:
The gap between the practices practitioners have mastered and the practices they find most critical for remote work, was the core insight I was looking for, as it highlights the areas where tools, programs, and support can be most beneficial. While the #1 item is shared across both lists, the #2 and #3 most critical practices, didn’t make the top-3 most mastered practices.
Full survey results can be found here for those curious.
Conclusion and important disclaimer
While this survey was an awesome methodological validation — using a sort/rank question, rather than a series of Likert scale questions to surface the insight I was looking for — it did not provide me with the clarity and directional confidence I was hoping for. This was due to the low number of total respondents (<100) leading to results that are not very robust/statistically significant, and preventing any further analysis using the demographic/psychographic data I collected.