Some Updates to Our Charity Evaluation Process: 2017

As we begin our 2017 evaluation process, we’re releasing a few updates to our criteria and evaluation methods. These updates are slightly smaller than those we implemented in 2016.

This year we have:

• Reworded and reordered our criteria
• Continued tweaking how we evaluate organizational factors that influence effectiveness through our last two criteria
• Added conversations with non-leadership employees to our plan for comprehensive evaluations
• Started offering participation grants
• Updated the structure of the criterion which addresses the qualitative aspects of a charity’s programmatic impact

New criteria wording and order

As part of our continuing effort to standardize our use of language and make our writing as clear as possible, we revisited the statements of each of our criteria for charity evaluation to ensure that each was brief, clearly conveyed the major ideas we were looking to assess, and avoided any unnecessary jargon.

We also revised the order of the criteria to improve the flow of our comprehensive reviews. In particular, we’re now assessing the general case for impact of the charity’s programs before calculating their cost-effectiveness in detail. We’re also discussing the charity’s past successes or failures before noting whether they have learned appropriately from those experiences.

Here are our revised criteria:

If these sound like the same criteria you thought we used last year, that’s intentional. If you’d like to learn more about our criteria, you can read our detailed criteria page, or see the old wording on the archived version.

Updated approach to criterion two

In previous years, we evaluated each charity’s “mission effectiveness” by qualitatively assessing the effectiveness of the primary types of interventions the charity employed. We worried that providing qualitative assessments of each type of intervention was redundant, given that for most charities we also provide quantitative estimates of the charity’s implementation of each intervention in a different section of our reviews. In fact, the qualitative estimates often seemed less informative than the quantitative ones. We also found that we were highly uncertain about the effectiveness of some types of interventions (particularly those which we have not yet evaluated).

This year, when we evaluate each charity according to Criterion Two, we plan to shift our focus from the types of interventions each charity uses to the strategies the charity employs in their attempt to create change. We’ve developed a list of some common mechanisms underlying the impact of animal advocacy interventions:

• Strengthening public support
• Strengthening alliances
• Improving policies and laws
• Capacity building
• Changing the food industry

We’ve developed a list of interventions that are intended to create change by each of the mechanisms listed above. Prior to writing our charity reviews, our research team will evaluate the effectiveness of each intervention relative to others that work by the same mechanism. We feel more confident evaluating the effectiveness of interventions relative to other, similar interventions than we did evaluating the effectiveness of interventions independently of one another.

Updates to how we evaluate organizational factors

We found that we weren’t fully satisfied last year with the ways that we assessed charities on criteria 6 and 7, both of which deal with organizational factors that we expect to contribute to charity effectiveness, but that aren’t easy to directly connect to effectiveness by observation alone. This year we chose to spend time investigating the literature about nonprofit effectiveness—and organizational effectiveness in general—in order to identify specific practices that we can evaluate and that are connected to organizational performance by relatively good evidence.

To assess whether a charity has strong leadership and a well-developed strategic vision, we will focus on determining the extent to which:

• The charity’s mission emphasizes effectively reducing suffering/helping animals
• The strategy of the charity supports the growth of the animal advocacy movement as a whole
• The charity’s board is composed of individuals from a diverse set of occupational backgrounds who have had diverse experiences
• The board of the charity participates regularly in formal strategic planning on behalf of the charity, and involves other stakeholders (e.g., staff, donors, and beneficiaries) in that process

To assess whether a charity has a healthy culture and a sustainable structure, we will focus on determining the extent to which:

• The charity receives support from multiple and varied funding sources
• The charity provides staff and volunteers with opportunities for training and skill development, helping them grow as advocates
• The charity has staff from diverse backgrounds (e.g., occupation, education) and with diverse personal characteristics (e.g., race, gender, age), and views diversity as a resource that can improve its performance
• The charity provides employees with a workplace free of harassment and discrimination

The listed points don’t address every aspect of either criterion, or every aspect that we might want to consider in our evaluation of a particular charity. We’ll also include in our reviews strong impressions we form about the concepts covered by the broader criteria but that don’t fit into the lists above.

For more information about the reasons we’ve selected to focus on the dimensions above, please see this blog post.

Conversations with more staff for comprehensive evaluations

Many aspects of organizational culture can best be evaluated by consulting multiple stakeholders, particularly including staff at various levels of an organization. While we’ve sometimes been able to get these multiple viewpoints through casual conversations or through staff, volunteers, and donors reaching out to let us know about their experiences, we know that this has left big gaps in our understanding, due in part to the unsystematic nature of these encounters. To help us get a more well-rounded view of all our comprehensively evaluated charities, we’re adding two brief, confidential conversations with non-leadership staff to our comprehensive evaluation process.

These conversations will differ from our current conversations, and those from our 2015 deep review of The Humane League, in several ways:

• We will reach out directly to individual staff members we want to contact, and we will not share their names with the charity leadership we are working with on other aspects of the review.
• We will focus the conversations specifically on organizational culture, avoiding topics such as job responsibilities and the charity’s work that we can cover in communication with leadership staff.
• We will not publish notes from our conversations, or details about the participants, though we will use the conversations to guide content included in our reviews.

These differences are intended to make it safer for staff to tell us about any concerns they have about their work environment or the leadership of the charity, by reducing the chance that anything they say to us could have negative consequences for their jobs or careers. Because organizational culture and leadership can be a sensitive topic, particularly for employees of the charity in question, we feel that in this case it’s appropriate to lose some transparency in order to improve our understanding.

Participation grants

As we mentioned in an earlier blog post about ACE’s prospective goals, this year we will offer participation grants to evaluated charities for the first time. These will be small grants intended primarily to help compensate charities for the time they spend interacting with us during the evaluation process. The size of the grant will depend on whether a charity has participated in an exploratory or a comprehensive review process, with the grants made to comprehensively evaluated charities being larger to reflect the additional time we ask charities to put into this process. While we’ve found many charities to be very generous with their time as we evaluated them, not all charities can afford to spend time on evaluations knowing they may not be chosen for a recommendation. Now that we are in a position to do so, we hope that making participation grants will help us convey our respect for charities’ time, and will help charities find the time to work with us. We expect these grants to make the biggest difference to charities that are especially pressed for time and resources, such as those with few or no paid staff.

We hope that the grants will serve to increase participation in our evaluation process, either by allowing charities to participate that might not have been able to, or by helping to strengthen ACE’s relationships with charities by acknowledging that each charity we choose to review is in our view doing important work. We also hope that increased participation in our evaluation process will help charities to become more reflective and identify opportunities for growth and positive change.