Top

2018 Evaluation Process

Table of Contents

Overview

Once a year, we conduct evaluations of charities at various levels of detail and update our recommendations as a result of those evaluations. Prior to November 2018, our most recent recommendation update took place in November 2017.

During the first few months of 2018, we made some updates to our charity evaluation process and conducted some of the foundational and intervention research that informs our reviews. Our formal evaluation process took place from June through November.

New to our evaluation process this year was our engagement with specialist consultants for each of our seven criteria. Prior to writing the reviews, brief explanations of each criterion—along with drafts of the standardized language used in our reviews—were sent to the respective consultant for feedback and were subsequently revised in light of that feedback. After writing initial drafts of the reviews, we sent drafts of each criterion section from all reviews to the consultant of that criterion for additional feedback.

Other additions to this year’s process include the charity evaluation handbook and organizational culture surveys we sent out to charities. The handbook is intended to help the charities through the process by providing the schedule and all of the policies in one place. The organizational culture surveys were intended to help us gain a broader picture of each charity’s culture than we could ascertain from the interviews alone.

The general timeline of our evaluation process was as follows:

  • We selected charities to review and sent out invitations to participate along with the charity evaluation handbook in early July.
  • We began gathering information and conducting research for comprehensive reviews in July.
  • We completed full drafts of our comprehensive reviews in September.
  • In September, we solicited feedback from our board, our Executive Director, our former Director of Research, and the criterion consultants, and worked to incorporate it into the initial drafts.
  • We finalized our recommendation decisions in early October and communicated them to the charities under review between October 22 and October 24, 2018.
  • We published our recommendations on November 26, 2018.

The members of our research team who are not primarily involved with the animal advocacy research fund or the experimental research division did most of the work on the reviews. A team of volunteers (with help from our Director of operations) helped summarize conversations with charities that had been transcribed using Otter Voice Notes. Our Executive Director, former Director of Research, Managing Director, a Researcher from the experimental research division, and two Board Members provided feedback on complete drafts of the reviews. Our Managing Director, Managing Editor, and copyeditor worked to copyedit the reviews and all associated content. Finally, our communications team, with the help of our Web Developer, published and announced the results.

Fig. 1: Flowchart depicting the 2018 Evaluation Process

Our Selection Process

We began our 2018 evaluation process by compiling an internal list of charities to consider evaluating. The list included:

  • Charities that requested to be evaluated
  • Charities that members of the ACE staff and board suggested evaluating
  • Charities that third parties asked us to evaluate
  • Charities that we had considered or evaluated in the past that:
    • Were selected as Top Charities in 2017
    • Were selected as Standout Charities in 2016 (as we wanted to ensure that we had an up-to-date understanding of their work)
    • Had been close to the threshold for further investigation in the past but had been excluded for some reason
    • We wanted to reconsider due to changes in their programming or our understanding of their activities
    • We had last reviewed in 2015 (as those reviews would expire in 2018)
  • International charities that we had not previously evaluated and that we understood to have significant influence in their home countries
  • Charities that we had identified as potentially promising during a systematic search for charities in Brazil and neighboring countries in South America.1

Following this process, we generated a list of 112 charities to consider evaluating in 2018.

Six members of our research team worked together to select which of the 112 charities under consideration would be reviewed. The first step was for each team member to use the charities’ websites, social media pages, and any other relevant sources of information (such as news stories or past reviews) to provide initial ratings of each charity.2 These initial ratings were used to rank the charities from least to most likely to be selected for review. The team then met and discussed reasons for or against selecting specific charities until consensus was reached and 15 charities were selected. These charities were selected based on factors such as how likely we thought each of the charities was to be recommended and how useful we thought the knowledge we would acquire and potentially publish from the comprehensive evaluation would be.

We contacted each selected charity to invite them to participate, and at this point, five of the 15 charities declined to be reviewed and one failed to respond.3 We held an additional meeting and selected three more charities to invite to participate. Two of these three accepted, and we ended up with a total of 11 charities to evaluate.

Comprehensive Evaluations

In total, we conducted comprehensive reviews for 11 charities this year. This included all three of our Top Charities from 2017, four of our Standout Charities from 2016, and four charities that we had not previously reviewed.

Early on, we had a discussion about whether we wanted to stop short of comprehensive evaluations and instead write up shorter, exploratory evaluations for any charities this year. We decided that all charities under consideration were worthy of comprehensive evaluations.

We conducted evaluations according to our general process for comprehensive reviews. Charities that agreed to participate were asked to provide documentation of their finances, accomplishments, and strategy, and we scheduled information gathering interviews with their leadership. To assess workplace culture, we asked each charity to provide us with contact information of all staff so that we could conduct confidential calls, and we provided a culture survey that we asked each charity to distribute to their staff and volunteers. After conducting interviews and gathering the documentation, we sent some initial follow-up questions and volunteers drafted call summaries for each of the interviews. In preparing and drafting the reviews, each member of the evaluation team researched and wrote the same one or two criteria sections across all reviews.

With help from our Director of operations, a team of volunteers helped summarize conversations with charities that had been transcribed using Otter Voice Notes. Our consultants for each criterion provided feedback on initial drafts of their respective sections of the reviews. Our Executive Director, former Director of Research, Managing Director, Managing Editor, a Researcher from the experimental research division, and two Board Members provided feedback on complete drafts of the reviews. Our Managing Director, Managing Editor, and copyeditor worked to copyedit the supplemental documents and reviews. Finally, our communications team, with the help of our Web Developer, published and announced the results.

After editing our comprehensive reviews and making our recommendation decisions (described below), we sent the reviews to the corresponding charities for approval, along with our conversation summaries and other supporting documents we intended to publish. Charities had the opportunity to request edits, including requests to remove confidential information or to correct factual errors. Nonetheless, all reviews represent our own understanding and opinions, which are not necessarily those of the charity reviewed. This year, all 11 of the charities for which we drafted comprehensive reviews agreed to have their reviews published.

Recommendation Decisions

After the majority of each comprehensive review was drafted, but before the reviews were entirely finished or sent to charities for approval, six members of the research team, our former Director of Research, our Executive Director, our Managing Director, and two Board Members met to discuss the selection of Top and Standout Charities. In preparation for this meeting, all participants indicated in a chart their individually prepared suggestions for Top Charities and Standout Charities. There was substantial initial agreement on the status of some charities, but not all. We discussed strengths and weaknesses of each charity and updated our votes in the chart when they changed following discussion. While the Board Members and former Director of Research provided points for discussion and their opinions on recommendation decisions, all decisions were made by votes from the research team, Managing Director, and Executive Director. At the conclusion of this first meeting, we had reached unanimous agreement on the recommendation status of three charities.

The same six research team members, Executive Director, and Managing Director met three more times over the next week and participated in several email threads to finalize our selection of 2018 Top and Standout Charities. We reached unanimous agreement around most of our recommendation decisions, and for the remainder, decisions were made by a majority vote. We will not provide more details about our decision process here, though most of it involved the specific aspects of individual organizations that we describe in our reviews.

Additional Information

  1. Brazil and other BRIC countries with rising levels of meat consumption and relatively less developed animal advocacy movements seem like possibly impactful places to work and relatively likely places to find some effective charities. We are in the process of systematically searching other BRIC countries for promising charities. For more information on animal advocacy in BRIC countries, see Faunalytics’ recent report, Attitudes Toward Farmed Animals in the BRIC Countries. 

  2. Staff members rated the charities on a scale from -2 to 2, where +/- 2 indicated that a charity definitely should / should not be reviewed, +/- 1 indicated a charity probably should / should not be reviewed, and 0 indicated uncertainty. Five of the six participating staff members rated all 112 charities and one staff member rated 18 of them. Scores were summed for all charities to generate an initial ranking. A significantly negative sum indicated that most team members thought that charity probably or definitely should not be reviewed, whereas a significantly positive sum indicated that most team members thought that charity probably or definitely should be reviewed. 

  3. Of the charities that declined the opportunity to participate, most responded that they were too busy at the time and or that they preferred to wait until 2019 because they expected their situation to be changing in the near future.