Surveys: exploring statistical significance

WARNING: Some stats and math ahead. Mostly based on this lovely post: T-test explained: what they mean for survey analysis

Who doesn’t like surveys?

Well, most people. And yet, we love using them in organizational contexts for various purposes.

One big challenge in using them in that context is that they are a one-sided exchange of information. And while that makes sense in many other contexts, for example, when asking customers for feedback about a product; inside the organization, what we’re really trying to create is dialogue, since survey “takers” have a big part to play in addressing any insights that may come up from the survey. But that’s a topic for a different post. Today, I want to focus on something a lot more concrete.

We like using surveys because they can provide us with a quantitative assessment of a situation. For example, to measure “how are we doing?” in a particular area and to track it over time or across different organizational demographics. But sometimes, if we’re not analyzing the data carefully enough, over-reliance on surveys can lead us to over-react.

Let’s say that we ran an inclusion survey in which participants were asked to respond to the following statement using our beloved 5-point Likert scale: “When I speak up, my opinion is valued”. When analyzing the survey results we discovered that women, on average scored a 4.5, while men, on average scored a 4.3. Can we say based on the survey data that men and women in our organization are not given an equal voice?

The answer, as always, is: “it depends”. Depends on what? Glad you asked! It depends on the following things:

  1. The size of our organization and the participation rate in our survey
  2. The confidence level we want to have in our answer. The standard 95% confidence level means that if we ran the survey again, we’ll reach the same conclusion 95% of the times.
  3. The difference in the means between the two groups
  4. The standard deviation of the responses in each of the group

1–3 are fairly straight forward. The standard deviations is the least intuitive of the bunch so we’ll focus on it and say that: assuming an organization of a certain size, in order for a certain difference in means to be statistically significant at a certain confidence level, the standard deviation of the results in each group must fall below a certain maximal threshold.

More so: the smaller the org (or the lower the participation rate), the smaller the difference in means and the higher the confidence level required— the lower the standard deviation threshold will be.

Let’s make this a bit more concrete: assuming a best case scenario in which there’s full participation in the survey and the groups are of equal size — these would be the standard deviation thresholds for various combinations of org size (n), confidence levels, and difference in means:

Standard deviation thresholds for statistical significance of difference in means at varying confidence levels and org sizes

So in a 100-person organization, in order for a 0.1 difference in means to be statistically significant at a 95% confidence level, the standard deviation of both groups must be below 0.25. Keep in mind that this is the best case scenario, so if participation was lower or the groups were not equal in size, that threshold will be even lower.

Which leads us to the next question: what does a 0.25 standard deviation look like? Sure we can do the math and crunch the numbers, but for those of us (yours truly included) who don’t have a strong statistical intuition this may help:

Distribution of n=100 results on a 1–5 scale with standard deviations of 0.3, 0.6, 0.9, 1.2

The next time I’m running a survey, before jumping to action simply by looking at the means, I plan to look up my standard deviations at the table above and figure out whether action is truly needed. I’d encourage you to do the same 🙂

Advertisements
Surveys: exploring statistical significance

Flexible Work [Werk]

When we talk about giving employees more flexibility around doing their work, we often time have different dimensions of flexibility in mind.

The team at Werk did a great job in offering a shared taxonomy for talking about flexibility:

Flexibility 101

The make a distinction between 6 types of flexibility:

  1. Work from home (Werk: DeskPlus™) employees are based out of a company office, but can work at a location of their choosing for some portion of their time. Utilizing location variety can enhance productivity, reduce the burden of a long commute, increase creativity, and/or meet other needs.
  2. Part Time employees work on a reduced hours schedule. Part Time does not mean an individual is no longer in an advancement track role — employees utilizing Part Time have the experience and skills to meet their objectives on a reduced hours schedule.
  3. Step way (Werk: MicroAgility™) employees have the autonomy to step away from their work to accommodate the unexpected in micro increments of 1–3 hours. Employees are responsible for communicating their plans and meeting their daily objectives. The ability to make micro-adjustments to the workday prevents an employee’s personal life from becoming a major work life disruption.
  4. Flexible workday (Werk: TimeShift™) employees reorder their working hours to create an unconventional schedule that optimizes productivity and performance. An employee could shift their workday an hour to avoid a long commute, to break their day into sprints, or in a formalized condensed work week program.
  5. Minimal Travel (Werk: TravelLite™) employees have minimal to no travel, with a maximum of 10% travel annually (2–4 days per month or its annual equivalent). Employees can reduce travel requirements by utilizing virtual meetings.
  6. Remote employees do not work at a company office — they can work from anywhere. While many Remote arrangements are fully location independent, some may have location considerations, such as the need to attend occasional in-person meetings or service a region.

I find these distinctions very valuable in giving us shared language to discuss this broad and amorphous topic.

My biggest qualm with the Werk framework is their decision to brand (and trademark…) the types of flexibility that did not have a broadly accepted definition around them. Looking at the field psychology as an interesting case study, the need to “brand” different psychological techniques resulted in a proliferation of brands with highly similar to completely identical underlying principles. Which in turn made it harder to see the forest for the trees and slowed down progress.

Flexible Work [Werk]

Book Review: BRIEF


I recently finished Brief: make a bigger impact by saying less by Joseph McCormack.

Brief makes a compelling case for shorter and clearer communication. Something that many of us struggle with, and is particularly critical in a professional context where good communication is essential for driving effective collaboration.

Brief offers two compelling, though somewhat overlapping, frameworks for organizing your content in an effective and short narrative. While the “Narrative Map” seems to be a more abstract structure for effective storytelling, the “Brief Map” seems to be a contextualized application of it to more common communication triggers in a professional setting.

The Narrative Map

  • Focal Point — What holds the story together? The main point or organizing principle, letting listeners know what the narrative is about and why they should care.
  • Setup (challenge) — Where every story begins, introducing the issue, problem or unmet need that forms the groundwork for the rest of the narrative.
  • Setup (opportunity) — The opportunity that is inherent in every story. The audience feels there’s a positive force to resolve the conflict for the remainder of the narrative.
  • Body (how, where, etc.) — Three to five key elements that enable the story to develop and move forward. These elements tell who, how, when and where the story gets resolved.
  • Payoff (conclusion) — The story’s payoff — where the challenge and opportunity are resolved, resulting in changing how the audience thinks, feels or acts.

The Brief Map

  • Background/Beginning — What is the current situation, issue, or problem?
  • Relevance/Reason — What does it really mean for the audience? What do we want them to do with the information?
  • Information for inclusion — What key pieces of information or ideas do we need to share to give the audience a clearer understanding of the situation
  • Ending or conclusion — What does success look like?
  • Follow up — What questions do we anticipate at the end?

It was interesting comparing and contrasting the Brief Map to the framework that I created a while back and integrating the best pieces of both:

I’m glad that I read “Brief” as I’m confident that I’ll come back to these two frameworks either when drafting my own communications or when coaching others. But overall, I’d “Brief” falls into the category of books that “should have been a 10-page whitepaper/HBR article” as I didn’t get much more out of it past these two tools. So I guess “Brief” should have been briefer 🙂

Book Review: BRIEF

The type of team diversity you’re probably not paying attention to

Source: First Round Review

A few months back, I partnered with the team at First Round Review on bringing some of the content in this publication to the First Round community.

If my Gmail account is correct, I’ve been a member of the First Round Community at least since August 2015 and over the years have found it to be one of the best sources of leadership and managerial advice, particularly in a growth stage company context.

We decided to an updated take on one of my favorite posts:

And I could not be happier with the result:

The type of team diversity you’re probably not paying attention to

It was a true honor being given the opportunity to give back to a community that was so helpful for me over the past 3 years, and I’m delighted to share the final product with this group as well.

The type of team diversity you’re probably not paying attention to

Thoughts on Diversity, Inclusion and Belonging — a first rough draft


June 2018 Pride Parade in Kyiv, Ukraine
June 2018 Pride Parade in Kyiv, Ukraine

A 9-hour flight with no entertainment system is the perfect opportunity to put on paper some of my thoughts on a very complex topic.

For the last 2–3 years I’ve been watching the conversations around the topic of diversity unfold. As someone who views “live at let live” as a core principle I’m trying to live by, and social justice as a core value I believe in, I was often surprised by how excluded I felt from engaging on this topic. I feared that the curious/questioning/contrarian approach by which I make sense out of the world around me will be misinterpreted as “just another white dude who doesn’t get it”. But I will learn very little out of fear, so instead, I’d like to offer two ideas on how we might be able to advance the dialogue going forward based on observing and participating in many conversations on this topic over the past few years.

If there’s one thing that I know about my current point-of-view is that it is wrong. In the sense that I’m sure I’ll be embarrassed by its crudeness in a few months/years once I gain a more refined understanding of this topic. But that’s what learning is all about, and creating this snapshot in time will help accelerate the pace of learning.

1. Start with Inclusion rather than Diversity

To explain what that means, I found the following distinction between the terms helpful:

  • Diversity is a state in which multiple perspectives are present
  • Inclusion is a set of behaviors that enable these perspectives to manifest themselves and influence the course of business
  • Belonging is a feeling of being to be your full self as part of a group and be valued for that

While the definitions themselves are far from perfect, the critical distinction is between a state, behavior and feeling.

Many organizations start with diversity, specifically trying to change the mix of individuals that the organization consists of, as measured by a set of agreed-upon categories (gender, ethnicity, etc.) with the aim of eliminating, or at least reducing under-representation. At first glance, it seems like a compelling strategy because measuring progress seems easy (we’ll discuss a big caveat here later) and the change that’s required seems rather mechanical: tweak the recruiting process to reduce bias. And yet, even
organizations that have been at it for years now, struggle to deliver meaningful results, and for some of the ones who do, results seem to be short-lived.

Inclusion is a harder goal to start with, but one that is more likely to yield long-lasting results. The distinction at the beginning of this section helps explain the benefits of this approach:

  1. Focusing on changing the composition prevents us from doing more with the diversity that we already have in the existing members of our organization. This is important because diversity does not automatically translate to inclusion and belonging. Which leads to the 2nd point:
  2. Part of the challenge with obtaining long-lasting diversity results stems from the lack of inclusion. We can bring diversity in. But if these members don’t end up feeling like they belong — they will leave. By focusing
    on inclusive behaviors first, we may get improved diversity, almost as a by-product.

2. Re-frame the conversation: from “us vs. them” to “all of us”

Much of the dialogue today is centered around specific identity attributes (gender, sexual orientation, ethnicity). In addition to being incredibly reductionist in describing what makes us us, it forces us to have this conversation using an “us vs. them” language: minorities vs. majority, whites vs. non-whites, men vs. women, straight vs. gay, etc. It creates a false pretense that progress for one must come at the expense of the other and it alienates many in the “them” camp rather than enlists them in committing to the difficult behavior change that’s needed.

We’re more likely to achieve the outcome that we seek by focusing on what we have in common rather than on what sets us apart. So instead, we can shift the conversation to focus on the need to belong as a universal need. Regardless of how each of us self-identifies.

A little over a year ago, CultureAmp and Paradigm designed a Diversity, Inclusion and Belonging survey that we can glean some interesting
insights from. Even though the survey is no longer available on the CultureAmp website, it can be accessed using the WayBack machine here (hardly anything on the internet truly disappears)

The survey is organized around 7 dimensions: belonging, decisions, diversity, fairness, purpose, resources and voice; with 2–4 statements pertaining to each:

Belonging:

  • I can be my authentic self at work
  • Even when something bad happens, I don’t question whether or not I belong at my company
  • I feel respected at my company
  • I feel like I belong at my company

Decisions: 

  • I am included in decisions that affect my work
  • Perspectives like mine are included in the decision making at my company
  • I am satisfied with how decisions are made at my company

Diversity:

  • My company values diversity
  • My company builds teams that are diverse

Fairness:

  • I believe that my total compensation is fair, relative to similar roles at my company
  • My job performance is evaluated fairly
  • People from all backgrounds have equal opportunity to succeed at my company
  • Administrative tasks that don’t have a specific owner are fairly divided at my company

Purpose:

  • I understand how my work contributes to my company’s mission
  • The work that we do at my company is important

Resources:

  • When there are career opportunities at my company I am aware of them
  • I know where to find information to do my job well
  • My company believes that people can always greatly change their talents and abilities
  • My company enables me to balance work and personal life

Voice:

  • When I speak up, my opinion is valued
  • I can voice a contrary opinion without fear of negative consequences
  • At my company, there is open and honest two-way communication

I have some minor qualms with the relevance and phrasing of some of the statements but I find the overall dimensions to be rather compelling and comprehensive.

A few observations on the universality of these statements looking at the survey design and results:

  1.  They use an “all of us” language that makes it a lot easier to rally behind working collectively to improve them in the organization
  2. Some of the lowest rated statements are issues for EVERYONE. Awareness of career opportunities is a good example
  3. The differences across statements are significantly larger than the differences within statements (at least when sliced by gender). There’s a 55% difference between the highest and lowest rated statement, and only a 15% difference between men and women on the most polarizing statement.

Would these two ideas solve the challenge of making progress on diversity, inclusion and belonging? I think that’s rather unlikely. But I believe they might allow us to advance the conversation that will lead to the progress we all seek.

Thoughts on Diversity, Inclusion and Belonging — a first rough draft

Tribalism and Intractable Conflicts [Ripley et al]

Like many, I’ve been closely watching what seems to be like a growing divide in US culture, which has been growing at an accelerated pace since the 2016 Presidential Election.

I’ve been meaning to write a post about it, because it’s a topic that permeates the membrane-like boundaries between organizations and “the rest of society”, and I believe that organizations, and specifically the dialogue that takes place inside organizations, can be a big force for good tackling these loaded topics.

The first insight in unpacking this topic was using the term “tribalism” to emphasize with and describe the behavior that all parties seem to exhibit. Simply put, it refers to engaging in the dialogue with a deeply entrenched “us vs. them” mentality. Listening to the other side’s arguments and experiences, looking for flaws in their logic and reasons for why your point of view is right. And letting your confirmation bias run wild, rather than evaluate new information based on fact and logic.

The first two essays that I read on this topic in late 2017, “The Dying Art of Disagreement” by Bret Stephens, and “Can our Democracy Survive Tribalism?” by Andrew Sullivan, painted a beautiful, balanced and depressing picture of the core role that tribalism plays in the current state of affairs.

I first considered writing this post after reading these but stopped short because while they paint a very nuanced picture of the current situation, they haven’t done much in terms of clarifying what the path forward should look like.

More recently, I read “Tribal World” by Amy Chua which provides a more historical perspective on major international affairs and US political events through the lens of tribalism. Chua argues that:

In seeking to explain global politics, U.S. analysts and policymakers usually focus on the role of ideology and economics and tend to see nation-states as the most important units of organization. In doing so, they underestimate the role that group identification plays in shaping human behavior. They also overlook the fact that, in many places, the identities that matter most — the ones people will lay down their lives for — are not national but ethnic, regional, religious, sectarian, or clan-based. A recurring failure to grasp this truth has contributed to some of the worst debacles of U.S. foreign policy in the past 50 years: most obviously in Afghanistan and Iraq, but also in Vietnam.

While I hoped that she’d continue this arc and offer some advice on how to better integrate the acknowledgment that tribalism exists into future policy-making, her parting thoughts focused instead ways to reduce tribalism through increasing economic stability and upward mobility. Hopeful, but not very actionable. The biggest insight for me was starting to view tribalism not as something to be eliminated but as something to be acknowledged and design for when considering interventions. The reason-based, see-the-other-side interventions fail to do that. And understanding that is actually progress 🙂

Luckily, my next discovery was a seminal piece called “Complicating the Narratives” by Amanda Ripley. Of all the pieces referenced here, this is the one I’d encourage you most to read. Ripley looks at the role that media and journalism have played in amplifying the tribal dynamics but she also boldly goes where no one had gone before: ascribing some key counter-intuitive guidelines for how journalists (and others) can help improve the dialogue going forward.

Ripley’s piece draws from many sources, but most heavily from the work of Peter Coleman (which is now on my reading queue…). Coleman’s research focused on “Intractable Conflicts” which he defines as:

[conflicts that] are intense, deadlocked, and resistant to de-escalation or resolution. They tend to persist over time, with alternating periods of greater and lesser intensity. Intractable conflicts come to focus on needs or values that are of fundamental importance to the parties. The conflict pervades all aspects of the parties’ lives, and they see no way to end it short of utterly destroying the other side. Each party’s dominant motive is to harm the other. Such conflicts resist common resolution techniques, such as negotiation, mediation, or diplomacy.

As someone who grew up in Israel until my late 20s, that definition deeply resonated. Not just for the obvious example of the Israeli-Palestinian conflict, but also for some of the less globally known conflicts within Israeli society such as the one around the relationship between state and religion.

Building on the work of Coleman and others, the gist of Ripley’s counter-intuitive advice is the following:

The lesson for journalists (or anyone) working amidst intractable conflict: complicate the narrative. First, complexity leads to a fuller, more accurate story. Secondly, it boosts the odds that your work will matter — particularly if it is about a polarizing issue. When people encounter complexity, they become more curious and less closed off to new information. They listen, in other words.

Ripley breaks down this advice into 6 distinct strategies, which she covers in detail in the piece:

  1. Amplify contradictions
  2. Widen the lens
  3. Ask questions that get to people’s motivations
  4. Listen more, and better
  5. Expose people to the other tribe
  6. Counter confirmation bias (carefully)

I am confident that it’s one of the pieces that I’ll find myself reading over and over again, discovering new insights every time I do so.

Tribalism and Intractable Conflicts [Ripley et al]

Live in Greatness protocols [McCarthy]

Human interaction is hardly ever easy, and often times just a bit of pre-defined structure can go along way in helping us get our needs met.

Almost a decade ago Jim and Michele McCarthy developed a set of such structures, which they branded as:

The Core Protocols

The core protocols are a set of conversations structures, or scaffolding, aimed at helping us get on the right foot in having some of the most important and frequent personal interactions in our teams:

  • Check in (begin meetings)
  • Check out (end meetings)
  • Pass (declining to participate in something)
  • Ask for help
  • Protocol check
  • Intention check
  • Decider
  • Resolution
  • Perfection game (iterating on a proposal/idea)
  • Personal alignment (self-reflection)
  • Investigate (understanding someone else’s behavior)

They are incredibly helpful starting points for some of our most common interactions.

Live in Greatness protocols [McCarthy]