2023 Engagement Survey

Animal Charity Evaluators’ (ACE’s) annual charity evaluations aim to identify organizations that will benefit animals as much as possible with additional donations. One of the four charity evaluation criteria we consider when evaluating charities is Organizational Health.¹ We do this by assessing whether there are any aspects of an organization’s leadership or workplace culture that pose a risk to its effectiveness or stability, thereby reducing its potential to help animals and possibly negatively affecting the reputation of the broader animal advocacy movement. For example:

• Schyns & Schilling (2013) report that poor leadership practices result in counterproductive employee behavior, stress, negative attitudes toward the entire company, lower job satisfaction, and higher intention to quit.
• Waldman et al. (2012) report that effective leadership predicts lower turnover and reduced intention to quit.
• Wang (2021) reports that organizational commitment among nonprofit employees is positively related to engaged leadership, community engagement effort, the degree of formalization in daily operations, and perceived intangible support for employees.
• Gorski et al. (2018) report that all of the activists they interviewed attributed their burnout in part to negative organizational and movement cultures, including a culture of martyrdom, exhaustion/overwork, the taboo of discussing burnout, and financial strain.
• A meta-analysis by Harter et al. (2002) indicates that employee satisfaction and engagement are correlated with reduced employee turnover and accidents and increased customer satisfaction, productivity, and profit.

As part of our Organizational Health assessment, we distribute an anonymous engagement survey to all staff working at the charities under evaluation. We consider the results of this survey alongside other information provided by the charities’ leadership staff, such as the policies and processes they have in place to support their employees.

This blog post outlines our process for this year’s engagement survey, the potential limitations of our approach, and some of the ways in which we are seeking to address these limitations. We have also provided the full list of survey questions.

Process

Leadership staff at each charity distributed ACE’s engagement survey to all paid staff members and to volunteers working at least five hours per week. We used Qualtrics to administer the survey and translated it into other languages when requested. We did not ask respondents for any identifiable personal details, so as to protect their anonymity and encourage honest responses.

This year, we developed the survey in collaboration with organizational consultants Scarlet Spark. To help ensure that our questions for staff were reliable predictors of organizational health, we based them on recognized frameworks such as the Gallup Q12 Employee Engagement Survey, the Maslach Burnout Inventory, Google’s Project Oxygen, and cross-cultural research by Culture Amp. We asked volunteers an alternative set of questions specifically designed to assess volunteer engagement and satisfaction.

We required at least 65% of the charity’s paid staff to respond to the survey to ensure that we had a representative sample of responses. There was no participation threshold for volunteers, as most organizations do not have a fixed number of volunteers.

If a charity scored particularly low on any aspect of staff or volunteer engagement, we followed up on these factors with the charity’s leadership to hear their perspective and understand any relevant context. To protect respondents’ safety and anonymity, we only shared aggregated organizational-level data with leadership and did not share individual survey responses or other confidential information. On a case-by-case basis, ACE recommended that the charity address any outstanding concerns, for example by:

• Conducting a comprehensive staff survey to assess employee engagement, satisfaction, and areas for improvement.
• Establishing regular channels for communication and feedback, such as open-door policies, suggestion boxes, or anonymous reporting mechanisms.
• Developing professional development opportunities and career advancement pathways for staff.
• Seeking external expertise on how to improve staff morale.
• If low staff morale is being caused by a specific person, carrying out a performance review with that person and agreeing on the specific ways in which their behavior needs to change, ​​including a timeline by which changes must happen.

The engagement survey contained a link to an anonymous Whistleblower Form,² developed with support from legal experts at Animal Defense Partnership, for any employees or volunteers who wished to report issues of harassment and discrimination. In most cases where we decide to take action based on such reports, our response consists of sharing relevant non-confidential information with the leadership of the organization in question and hearing their perspective. This process exists to help us gain a better understanding of what happened, whether the leadership members were aware, and what measures they took, if any. Depending on the outcome of this discussion, we may then evaluate whether the leadership members took, or plan to take, appropriate action and factor this into our overall Organizational Health assessment.

Limitations

While we strive to continually improve our assessment of charities’ organizational health, we believe our engagement survey process has room for improvement.

First, we are unable to fully investigate any harassment and discrimination claims we receive due to a combination of time constraints, lack of expertise, and the often anonymous nature of such claims. We recognize that this may cause frustration among charities that we evaluate, especially when we are unable to share specific details about these claims for reasons of confidentiality.

This year, we sought to improve the channel for people to submit claims by linking to the more comprehensive Whistleblower Form co-developed with Animal Defense Partnership rather than asking directly about harassment and discrimination in the Engagement Survey. We hope this helps ensure that people understand the implications of providing such information, improve the comprehensiveness of any such information that we receive, better enable us to follow up with claimants, and better identify the level of detail we are able to share with the leadership of the charity in question. At the same time, we recognize that requiring people to fill out a separate, more comprehensive form may reduce the number of reports that we receive, and we did receive significantly fewer such reports this year than in previous years.

Second, our engagement survey only captures a limited window of a charity’s workplace culture and may not fully represent the broad range of experiences within the organization. In particular, we recognize that surveying staff and volunteers can lead to inaccuracies due to selection bias. Results also may not reflect employees’ true opinions, as respondents are aware that their answers could influence ACE’s evaluation of their employer. We also recognize that our assessment represents a snapshot at a point in time and may not fully capture ongoing cultural shifts within an organization.

This year, we included a broader range of questions in the survey and collaborated with organizational consultants Scarlet Spark to help ensure these questions are likely to be effective predictors of organizational stability and effectiveness. As in previous years, we do not rely solely on the results of the engagement survey to make our assessment. Rather, we assess organizational health from multiple perspectives to arrive at the most appropriate decision within the time available based on all the information we have, including our follow-up conversations with the charity’s leadership.

Third, there is no universally agreed-upon “best practice” for organizational leadership and culture. With a wide range of frameworks, models, and approaches available, it can be challenging to establish a singular standard for evaluation, which may lead to a variety of interpretations and expectations among charities. As mentioned above, this year we developed our organizational health assessment in collaboration with Scarlet Spark to help ensure we are using the most relevant research. Where possible, we used recognized frameworks such as the Gallup Q12 Employee Engagement Survey. As in previous years, we also seek to gather input both from the charity’s leadership and non-leadership staff so that we can understand any issues from multiple perspectives.

Lastly, our assessment may be biased towards certain Western workplace practices. As a U.S.-based organization with staff based predominantly in the U.S. and Western Europe, our understanding of best practices for organizational health is inevitably skewed toward the cultures with which we are most familiar. We seek to recognize this bias at all stages of the assessment and to continually learn from the charities that we evaluate, including through follow-up discussions with the charity’s leadership team, rather than imposing a one-size-fits-all approach to each charity’s unique situation. This year, we modified our engagement survey questions to reduce their focus on Western cultures and piloted the questions with charities from different global regions to help ensure this was successful. We will continue to explore how best to improve the applicability of our assessment across all national contexts, using evidence from the countries where our evaluated charities are based.

Full list of questions in 2023

This is the complete set of questions that we used in our 2023 engagement survey. The first question directed respondents to either the “Questions for paid staff” or “Questions for volunteers” section.

---

1. This criterion was called Leadership and Culture from 2020 to 2022. We found that “leadership” was often misunderstood as referring solely to the qualities of individual leaders and that “culture” was understood in very different ways across countries and demographics. With the new name Organizational Health, we intend to highlight the broad focus of this criterion and to clarify that its goal is to identify any significant risks to the organization’s effectiveness and stability.  

2. The publicly accessible version of this form can be found via ACE’s Third-Party Whistleblower Policy on our website.