How We Evaluate Workplace Culture

Table of Contents

• Our Approach to Evaluating Workplace Culture
• Our 2019 Culture Survey
  • Purpose
  • Survey Questions
  • Analysis
  • Limitations
• Frequently Asked Questions

Our Approach to Evaluating Workplace Culture

We believe that a healthy workplace culture is an essential component of any effective organization. That’s why having “a healthy culture and a sustainable structure” is one of our seven evaluation criteria. We have observed over the years that—even if an organization is seemingly doing everything else right—they will not fulfill their potential if they have a culture in which their team does not feel empowered to do their best work.

Evaluating a charity’s culture from an external perspective is challenging. Though having a healthy culture has long been one of our evaluation criteria, we acknowledge that we have not always accurately assessed the cultures of the charities we review. (See our “mistakes” page.) For the past three years, we have been steadily working to improve our evaluations of workplace culture. In 2017, we began interviewing two non-leadership staff members of each organization under review. In 2018, we created a culture survey that we sent to each charity under review, though we allowed charities to send us their internal culture surveys if they preferred. We found that our culture survey gave us a much fuller picture of each charity’s culture and that distributing the same culture survey to every organization was essential, since charities’ internal surveys vary widely in content, relevance, and quality. Therefore, in 2019, participation in our culture survey became an eligibility requirement for receiving a recommendation from ACE.

We still ask each charity’s leadership about their perception of their organization’s culture, of course. It’s also increasingly common for us to receive individual testimonials about an organization’s culture directly from current or former employees. We consider all three sources of information (conversations with leadership, survey responses from current staff, and individual testimonials from current and former staff) to be crucial to our evaluations of workplace culture.

Our 2019 Culture Survey

Purpose

The main purpose of our culture survey is to catch any serious cultural problems that might prevent a charity from operating effectively. Since we find that cultural tensions have a tendency to escalate, we also hope to detect more minor problems before they might become a liability for the organization. Our goal is not to uncover every one-off incident of tension or disagreement, and it’s certainly not to broadcast every incident in our reviews. We simply want to do thorough due diligence and avoid recommending charities that might encounter major, foreseeable problems before our next evaluation round. We have a responsibility to our donors to only recommend giving opportunities that have our full confidence.

Survey Questions

Below is a copy of our 2019 culture survey. The first 12 questions are adapted from the Gallup Q12 employee engagement survey. The rest were developed by our team in 2018 and refined prior to our 2019 evaluation round.

Note that we have versions of this survey in English, French, German, Italian, Polish, Portuguese, and Spanish, which were translated by our staff and volunteers. Below is the English version, and others are available upon request.

Culture and Engagement

The following statements are adapted from the Gallup Q12 employee engagement survey. Please respond to each statement with a numerical rating. In all cases, your numerical ratings correspond to the following scale:

1 = No, I strongly disagree
2 = No, I moderately disagree
3 = No, I slightly disagree
4 = I neither agree nor disagree
5 = Yes, I slightly agree
6 = Yes, I moderately agree
7 = Yes, I strongly agree

1. Do you know what is expected of you at work?
2. Do you have the materials, equipment, and training to do your work well?
3. At work, do you have the opportunity to do what you do best every day?
4. During your last seven work days, have you received recognition or praise for doing good work?
5. Does your supervisor, or someone at work, seem to care about you as a person?
6. Is there someone at work who encourages your development?
7. At work, is it easy for you to give your opinion in conversations, suggest new ideas, and have your contributions valued by leadership?
8. Does the mission or purpose of your organization make you feel your job is important?
9. Are your colleagues (fellow employees) committed to doing quality work?
10. Do you have a best friend at work?
11. In the last six months, has someone at work talked to you about your progress?
12. In the last year, have you had opportunities to learn and grow?

Compensation, Benefits, and HR

Please respond to each question with a numerical rating. You may optionally elaborate on your ratings in the text boxes. In all cases, your numerical ratings correspond to the following scale:

1 = No, I strongly disagree
2 = No, I moderately disagree
3 = No, I slightly disagree
4 = I neither agree nor disagree
5 = Yes, I slightly agree
6 = Yes, I moderately agree
7 = Yes, I strongly agree

1. Do you feel adequately compensated for the work that you do (e.g., salary, hourly pay, etc.)?
   1. 1–7 rating choices
   2. (Optional): Please elaborate:
2. As part of your compensation package, are you provided with sufficient benefits (e.g., paid time off, flexible working hours, health insurance, vacation time)?
   1. 1–7 rating choices
   2. (Optional): Please elaborate:
3. Is there someone you could go to if you had a problem at work (e.g., problems with sexual harassment, performance reviews, and so on)?
   1. 1–7 rating choices
   2. (Optional): Please elaborate:

Leadership, Strategic Vision, and Transparency

Please respond to each question with a numerical rating indicating your level of agreement. You may optionally elaborate on your ratings in the text boxes. In all cases, your numerical ratings correspond to the following scale:

1 = No, I strongly disagree
2 = No, I moderately disagree
3 = No, I slightly disagree
4 = I neither agree nor disagree
5 = Yes, I slightly agree
6 = Yes, I moderately agree
7 = Yes, I strongly agree

1. What are three adjectives that you would use to describe your organization’s communication style?
   1. (text box)
2. Does the system for evaluating staff performance need to be changed or improved upon?
   1. 1–7 rating choices
   2. Please elaborate (optional):
3. Are the leaders of your organization attentive to the organization’s strategy (i.e., is their strategic planning the result of a conscious, reflective process)?
   1. 1–7 rating choices
   2. Please elaborate (optional):
4. Does your organization’s board support the organization in achieving its strategic vision?
   1. 1–7 rating choices
   2. Please elaborate (optional):
5. Does your organization promote internal transparency (i.e., are its leaders open about their decision-making and accountable for any mistakes)?
   1. 1–7 rating choices
   2. Please elaborate (optional):
6. Does your organization promote external transparency (i.e., is it honest with donors and other funders, and does it publicly and constructively respond to criticism)?
   1. 1–7 rating choices
   2. Please elaborate (optional):

Diversity, Equity, and Inclusion (DEI)

Please respond to each question with a numerical rating indicating your level of agreement. You may optionally elaborate on your ratings in the text boxes. In all cases, your numerical ratings correspond to the following scale:

1 = No, I strongly disagree
2 = No, I moderately disagree
3 = No, I slightly disagree
4 = I neither agree nor disagree
5 = Yes, I slightly agree
6 = Yes, I moderately agree
7 = Yes, I strongly agree

1. During the past year, have you experienced or witnessed any instances of harassment or discrimination in your workplace?
   1. 1–7 rating choices
   2. Please elaborate (optional):
2. Does your organization have staff members from diverse backgrounds (sexual orientation, gender identity, race) and/or does it prioritize DEI in its plans for growth?
   1. 1–7 rating choices
   2. Please elaborate (optional):
3. Have you and your colleagues been sufficiently trained in issues of discrimination, inclusivity, sexual misconduct, and/or diversity by your organization, or by an outside group working with your organization?
   1. 1–7 rating choices
   2. Please elaborate (optional):
4. Does your organization protect staff, interns, and volunteers from harassment and discrimination in the workplace?
   1. 1–7 rating choices
   2. Please elaborate (optional):
5. Do you get along well with your colleagues?
   1. 1–7 rating choices
   2. Please elaborate (optional):
6. What is the most important action that leadership could take to be more inclusive or to better support staff who are members of marginalized groups, that they are not already doing?
   1. (text box)
7. Is there anything else—either positive or negative—under the umbrella of your organization’s diversity, equity, and inclusion practices that you’d like to comment on?
   1. (text box)

Final Thoughts

1. Is there anything else you would like to share with ACE about your organization’s culture?
   1. (text box)

Analysis

We use Google Forms to administer our survey, and responses are sent directly to a folder in our Google Drive. Only one ACE staff member (this year, our Director of Research) has access to the folders with responses.¹ That staff member reviews all survey responses and prepares summaries for the rest of the team. The summaries include:

• Average response scores for each question
• Whether the distribution of responses is approximately normal or whether it has any skewness or kurtosis (only when relevant)
• An average of the first 12 scores, which is our proxy for employee engagement
• Any themes from the open responses
  • We understand a response to represent a “theme” if it is mentioned by at least three respondents throughout the survey

The summaries of results do not include anything else. More specifically, they do not include:

• Exact quotes from open responses
• Any identifying information about respondents

These summaries are shared with our evaluation team members and with the corresponding charities, upon request.

Limitations

We are aware of several limitations to our culture survey, which we note in our reviews. First, they may be subject to selection bias since we distribute the survey to a charity’s current—but not former—staff. In general, we expect this bias to positively skew the results of the survey, since a charity’s current employees are likely happier at the charity, on average, than its former employees were. We become especially concerned about selection bias when we review charities that have recently had a high number of staff departures, and we note in our reviews when we are aware of such situations. We do not proactively reach out to a charity’s former staff in order to counteract the effects of selection bias because there is no way for us to do so systematically. However, when former employees of a charity reach out to us, we take their reports seriously.

A second limitation of our survey is response bias. That is, of the current staff members who receive the survey, some may be more likely to participate than others. In general, we expect employees to be more likely to participate in our survey if they have especially strong feelings—either positive or negative—about their workplace culture. This bias might skew the responses positively in some cases and negatively in others. We become especially concerned about response bias for charities that have relatively low response rates. Luckily, the response rates for our survey tend to be fairly high in general. The average response rate for the charities we reviewed in 2019 was 74%.

A third limitation of our culture survey is that we are in the complicated position of evaluating charities when we may influence their work. In particular, we may be able to direct significant funding towards charities that we evaluate positively. We realize that this may substantially restrict the information that a charity’s employees are willing to share with us, and (all else equal) we expect responses to our survey to be positively skewed as a result. However, at least some respondents do share critical feedback about their organizations with us, and therefore we still find the surveys useful for helping charities identify areas for improvement.

The last limitation we will mention here is that we do not precisely track who participates in our surveys. Protecting respondents’ privacy is our priority, and we therefore do not require them to give their names or log in with an organizational email address. Every charity has a unique survey link that is sent to all of that charity’s staff (and only that charity’s staff). Technically, however, a staff member could share the survey link with an external party and/or take the survey more than once. We have no reason to believe that this has ever happened. We have never noticed multiple sets of identical responses, and our surveys have never had response rates of more than 100% (though this year, three charities had response rates of exactly 100%). While we feel that the possibility of survey fraud is important to mention here, we are much less concerned about it than we are about the three concerns listed above. We will therefore continue to prioritize respondents’ anonymity over preventing the possibility of fraud.

Frequently Asked Questions

How is the survey distributed?

We ask charities for a full list of team members with corresponding email addresses. We send a link to the survey to everyone on the list, with a brief email explaining our goals. We typically give teams about one week to take the survey. In some cases, charities strongly object to sharing their full lists of employees with us. We allow these charities to distribute the survey themselves, provided that they CC us and sign a statement confirming that they sent it to everyone on their team.

How can international employees participate, particularly if they don’t speak English?

Thanks to our several bilingual team members and generous volunteers, we have translations of our survey in English, French, German, Italian, Polish, Portuguese, and Spanish! We will seek additional translations in the future should the need arise.

Can ACE survey former employees of each organization?

Some members of our community have reached out to us asking why we don’t survey a charity’s former employees. They are concerned about selection bias, which we mentioned in the “limitations” section of this blog post.

Unfortunately, we have no way of systematically surveying a charity’s former employees. While charities might have a list of former employees, it would likely not include current contact information in all cases. We are also aware that a charity’s culture can change quickly, and former staff may not be as informed as current staff on an organization’s work environment.

Is ACE allowed to ask employees about harassment?

Yes, ACE is legally allowed to ask a charity’s employees if they’ve ever witnessed or experienced harassment in their workplace. We understand, of course, that this is a sensitive topic and we try to avoid unnecessarily alarming or potentially triggering language. Importantly, no one is required to answer this question or to participate in our survey at all.

Why does ACE ask whether employees have a “best friend” at work?

The “best friend” question is part of the employee engagement measure that we adapted from Gallup’s Q12 survey. It is a validated measure, and we changed it as little as possible when we adapted it for our purposes. We have already received some questions about the relevance of having a best friend at work, and apparently Gallup has as well—they’ve written about it extensively. The short answer is: Gallup’s research suggests that having a best friend at work is positively correlated with engagement.

Can respondents take the survey more than once? / Can people submit fake survey responses?

Protecting respondents’ privacy is our priority, and we therefore do not require them to give their names or log in with an organizational email address. As a result, someone could technically take our survey more than once and/or submit fake responses. We have no reason to believe that this has ever happened. We have never noticed multiple sets of identical responses, and our surveys have never had response rates of more than 100% (though this year, three charities had response rates of exactly 100%).

How does ACE interpret the results of surveys with very small sample sizes?

First, we try to understand why the sample size was small. Does the charity have a very small staff or did the survey have a low response rate? We work to interpret the results of the survey in the appropriate context and consider whether response bias might be a significant issue.

We do not conduct any complicated statistical analyses on our survey, or make claims of statistically significant results. We never make direct statistical comparisons between questions or between charities.

Since ACE influences funding to charities, will employees be disincentivized from reporting problems on ACE’s survey?

We hope that respondents will be honest with us on our surveys, but we’re well aware that they have reason not to be, since our recommendation of their employer is on the line. Importantly, many respondents do share critical feedback about their employers with us, so they are not entirely disincentivized from doing so.

We are grateful to everyone who provides honest input on our work. We hope that respondents are aware of the reasons in favor of being honest with us. For instance, if they help us identify a problem, we might be able to bring it to their leadership’s attention and suggest some solutions. We also hope that our culture survey data can help the movement as a whole. When we observe themes in responses across different charities, it helps us determine our research priorities and provide resources for the movement (see, for example, our recent roundtable posts on remote culture).

Who can access the responses to the surveys? Are they stored securely?

Responses are stored on Google Drive and are only accessible to one ACE staff member (this year, our Director of Research). We will delete all survey responses from our Drive in December, after our evaluations are published.

Will ACE publish the survey responses?

We may publish information from the survey responses that is included in our summaries of results. This information includes:

• Average response scores for each question
• Whether the distribution of responses is approximately normal or whether it has any skewness or kurtosis (only when relevant)
• An average of the first 12 scores, which is our proxy for employee engagement
• Any themes from the open responses
  • We understand a response to represent a “theme” if it is mentioned by at least three respondents throughout the survey

Our summaries of results do not include anything else. More specifically, they do not include exact quotes from open responses or any identifying information about respondents. Since these types of information are excluded from our summaries, we will never publish them in our reviews.

We do not necessarily publish all of the information that is included in our summaries. Additionally, we send our reviews to each charity for approval prior to publication. If a charity strongly objects to us sharing information from the survey, they have the option to withdraw from our evaluation process at any time.

Will ACE share the survey responses with charities’ leadership?

Upon request, we will share our summary of survey responses with the corresponding charity’s leadership. Again, the information in our summaries is limited and does not include identifying information about respondents.

Does ACE have a responsibility to share reports of illegal activity with the relevant charity’s leadership?

ACE has no legal obligation to share reports of illegal activity (e.g., harassment or discrimination) with the relevant charity’s leadership.

If we receive a report of illegal activity that is corroborated by at least three sources, we will flag that concern in our draft of the charity’s review and offer the charity’s leadership an opportunity to respond prior to publication. However, we will never share a report with a charity verbatim unless we have explicit permission from our source to do so (and we have never asked). Our top priority is to protect our sources’ confidentiality.

Can ACE require organizations to take a survey?

We technically do not require or force any organization to participate in our culture survey. Our entire evaluation process is completely optional. We invite charities to participate and we proceed only with their agreement. We do not even require all charities that we evaluate to take our survey. However, we are not comfortable recommending charities that choose not to participate in our culture evaluations. In other words, participation in our survey is an eligibility requirement for receiving a recommendation, but not for participating in our process.

Should ACE require organizations to take a survey?

We believe that participating in our culture survey should absolutely be a requirement for charities receiving an ACE recommendation. Distributing our survey helps ensure fairness and consistency in our review process. Without it, we would rely on interviews with randomly selected staff members (which might not be representative of all staff opinions) and ad hoc reports from our community. By using our survey, we gather a similar amount of information about each charity we review. Distributing the survey is—by far—the best way we have found to gain a full picture of a charity’s culture, and we are not comfortable recommending a charity without this due diligence.

Thank you for taking the time to read about our approach to evaluating workplace culture. As always, we welcome your feedback in the comments below, or you can email our research team.

---

1. This year, our Director of Research had a conflict of interest with one charity under evaluation, so a different member of our research team had exclusive access to that charity’s culture survey results.