Mind the (survey) gap

Doing employee research right

A 2013 study by the Engagement Institute found that over 80% of global organizations survey their employees regularly, with an increasing number administering regular pulse surveys throughout the year. 

While employee survey research is one of the most common methods for conducting employee research, it is also the ONLY one widely used, leaving many organizations with significant blind spots and missed opportunities in improving their organizational health. Those challenges stem from both flawed execution and structural limitations of the employee survey methodology. 

A recent paper by Lewis Gerrard and Patrick Hyland offers a critical review of both theory and practice of employee survey research:

Employee Survey Research: A Critical Review of Theory and Practice

The origins of employee survey research can be traced all the way back to the 1920s. Today, well-designed organizational survey projects are characterized by three main activities: 

  • Careful measurement with the clear aim of measuring personal perceptions objectively. 
  • Robust data analysis, seeking to generate insight from survey responses. 
  • Collective feedback & action planning, recognizing the importance of involving employees, managers, and leaders in the process of interpreting results, recognizing areas for change, and developing plans for improvement. 

While this simple bar is not always met by many employee survey projects, the paper focuses on deeper issues. 

At the theoretical level, employee survey research is based on several incomplete or outdated philosophies about the nature of organizations. For example, the design and analysis of surveys assume a clear and consistent cause-and-effect relationship between various organizational phenomena. However, complex adaptive systems theory, suggests that this view is overly simplistic and does account for chaotic, disruptive and emergent organizational dynamics that make predictable causal links unlikely. 

The paper offers a few additional examples of organizational theories/philosophies that underlie employee survey research and recent developments and discoveries have proven to be either incorrect or incomplete. 

At the practical level, the paper highlights three methodological challenges and three analytical mistakes that often get in the way of drawing accurate insights out of employee survey research. 

Methodological challenges 

  • Overlooking the limitations of cross-sectional design: survey data is collected anonymously and analyzed at the aggregate level (department/organization) ignoring changes in employee population such as new hires, departures, and cross-departmental transfers between survey administrations. 
  • Ignoring common method bias & the necessary conditions for claiming causality: causality analysis is conducted within a single survey resulting first, in an inability to measure and control for systematic measurement error — variance introduced by the measurement method itself. And second, in limiting the credibility of any causal relationships uncovered between survey items/dimensions since they were only explored at a single point in time. 
  • Surveying only the evaluating and remembering self: since surveys often ask employees to make global evaluations about work based on retrospective reflections, they only capture people’s after-the-fact, reflected evaluations of events, which was shown to be different than their moment-to-moment assessment of the same events as they were happening. 

These methodological challenges are not without solutions: 

  • Longitudinal survey design, tracking attitudes at the individual level and making comparisons based only on employees who completed all surveys over a given time period can address cross-sectional design limitations. 
  • Creating temporal separation between measuring predictor variables and criterion variables — conducting statistical analysis across surveys rather than within surveys can address the “single point in time” challenges. 
  • Experience sampling techniques like The Day Reconstruction Method can complement the standard reflective method ensuring that data from both the “experiencing self” and the “remembering self” are captured. 

Analytical mistakes 

1. Assuming more is better: consistently positive survey results have three potential drawbacks: 

  • Strongly favorable attitudes may prevent organizational leaders from engaging in the critical process of reflection and change, increases the challenge of breaking out of the status quo. 
  • Employees in some roles may actually benefit from a sense of dissatisfaction at work, where a sense of frustration is an important driver of action.
  • In some cases, a stronger endorsement of an item is not predictive of increased outcomes. Some attitudes and employee states may be more beneficial if they fluctuate rather than maintained at a consistently high level. 

2. Overemphasizing contextual factors: the focus on measuring climate factors and shared employee experiences has led to a lack of practical integration between the psychology of individual differences and perceptions of workplace experience. A 2018 meta-analysis established that around 50% of the variance in an employee’s work engagement is predicted by personality and character variables. Therefore, variables like engagement are heavily influenced by the nature of the people selected into the organization. 

3. Assuming survey results will lead to change: while employee survey research has been a staple organizational practice for several decades, over a recent 18 year period (2000–2018), on average 30% of employees have been engaged at work, with a maximum variation of 4% in each direction. A principal assumption of the employee survey process is that leaders have the capacity and willingness to use employee feedback to create change. Yet this assumption has not always held true. Furthermore, studies demonstrated that factors like CEO personality play a key role in culture development, suggesting that a significant change in culture may only be possible with a significant change in leadership. A change that many organizations are often reluctant to make. 

Best-in-class employee research programs 

In addition to performing the activities required for well-designed organizational projects, addressing the methodological challenges and avoiding the analytical mistakes of employee survey research, the authors offer the following recommendations for creating best-in-class employee research programs: 

  • Learn to use multiple methodologies — complement employee survey research with additional methodologies that make up for its blind spots and shortcomings. 
  • Assess organizational phenomena at multiple levels — team effectiveness, for example, should be explored by assessing individual team member personalities, group-level behavior, and organizational-level culture. 
  • Integrate multiple datasets — rather than analyzing attitudinal data in isolation, link it with effectiveness, performance and productivity datasets to generate more valid organizational insights.
  • Engage multiple stakeholders — employee research programs aim to create insight, energy and direction for organization members, from front-line employees to c-suite members. Ongoing dialogue with that diverse group of stakeholders will help ensure that this objective is indeed achieved. 
Mind the (survey) gap

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s