Shallow Evaluations Process Update

We are currently in the process of conducting our first round of organization evaluations as Animal Charity Evaluators. While the details of our evaluations of particular organizations will remain confidential until we are ready to publish our recommendations, we’re going to share reports of our process as we progress through the different evaluation stages. Right now, we’re putting the final touches on the shallow review process. Here’s what we’ve done so far.

Develop Question and Organization Lists

We had two lists that were integral to the entire shallow review process. The first was the list of organizations to evaluate. We chose to evaluate primarily farm animal organizations and large animal advocacy organizations doing significant amounts of work on behalf of farmed animals, because farmed animals are such a large percentage of the animals directly harmed by humans. We generated an initial list based on the farm animal charities currently listed on our research pages, a legacy of the work of past volunteers. To this we added some other organizations that we knew were working on behalf of farmed animals. We also reached out to several leaders in the farm animal advocacy community to ask for suggestions of organizations that they thought were particularly effective. To simplify our first attempt at this process, we restricted the list to only those groups based in or working primarily in the United States. Our final list has 29 organizations.

The second list was a list of questions based on our criteria for organizations. We expected to evaluate a relatively large number of organizations and wanted to do so as efficiently as possible. This meant relying on public data so that we wouldn’t have to wait for responses and so that organizations would not have to give us significant amounts of their time without a good chance of a recommendation in return. We created questions that focused on aspects of an organization’s fit with our criteria that would be publicly visible. We compiled these questions into a reference document that could be used when reviewing each organization.

Find and Summarize Relevant Information

The bulk of our work on the shallow investigations was devoted to finding answers to as many questions as possible for each organization. Three people worked on this stage of the project: ACE’s Executive Director and Director of Research, and one volunteer. We divided the work by question, so that there would be no bias in the evaluations caused by one of us judging a certain question more harshly than another did.

The amount of information we were able to find varied widely between organizations, but there were strong patterns to what information was generally available. We were able to find several years’ Forms 990 for almost every organization, although some were too new, not actually registered as non-profits, or had filed an abbreviated form that is not as frequently archived online. These gave us a good sense of whether an organization is growing, at a steady size, or shrinking. To get a sense of which organizations might have most or least need for increased funding, we also considered the ratio of assets to expenditures and compared it to similar organizations or to the same organization at different points in time.

For some organizations, we were also able to use information from Form 990 to develop a rough understanding of how their funding was divided among the different programs they pursued. However, most organizations did not use program breakdowns that corresponded to the categories we would want to use in creating a cost effectiveness estimate. In most cases we did not create formal cost-effectiveness estimates because we believed they would rely too much on guesswork not only about the effectiveness of the interventions undertaken, but also about the amount of resources the organizations devoted to each program. We expect to be able to include cost effectiveness estimates in our reviews for those organizations that we are still considering, since we will be able to get more detailed information directly from them.

We sought other information primarily from organizations’ websites, but also from searches for mentions of them in the news and elsewhere. Sometimes we used previous versions of their websites to shed light on how their structure and activities had changed over time. Every organization provided enough information online that we could compose a general outline of their activities and philosophy. However, the amount of detail about advocacy accomplishments varied greatly. Some organizations listed every event they conducted, organized primarily by timing. Others aggregated their accomplishments, detailing how many people they reached or animals they helped yearly. Still others simply indicated that they engaged in a certain activity, such as leafleting or running a sanctuary, but did not provide specific information about the extent of their efforts.

Most organizations also provided some information about their organizational structure and staff in key leadership roles, and if they used volunteers, some information about what volunteer roles were available. Some provided information about their future plans and about past activities, often by making available end of year reviews or newsletters. Very few provided information about what they would be able to do with increased levels of funding or about the methods they used to evaluate their programs.

Select Organizations to Review More Deeply

After compiling the information about each organization, we considered which should be reviewed more deeply to possibly receive our top recommendation. Our primary consideration was to review each organization that, based on what we had learned in the shallow investigation, we might find to be uniquely effective through further investigation over the next two months. We had varying amounts of knowledge from outside the evaluation process, which we tried to discount for now, because placing too much weight on it might give an unwarranted advantage or disadvantage to certain organizations. (For instance, we tried to avoid giving organizations too much credit for information they had been invited to share with us in the past but do not make publicly available, since we had not previously made contact with all the other organizations we were reviewing to request similar information).

In general, organizations which we chose to continue reviewing:

• Attempt to help farmed animals through public outreach and/or legislative advocacy. While some other approaches, including the development of new technology, have potentially high returns, we did not find groups engaging in other efforts that both seemed to have potentially high returns and seemed possible for us to understand reasonably well within the time frame we have set for this round of recommendations.
• Focus attention on the rights and welfare of farmed animals. While we do not know whether this is the most effective message to convey to affect people’s actions in the short term, we think spreading concern for farmed animals is a more robust long term strategy than spreading animal-friendly behaviors for other reasons.
• Have at least some paid staff members. While we reviewed some all-volunteer organizations that do very interesting work, these organizations appeared to be less active than those with paid staff, and it was less clear that increased levels of funding would have allowed them to increase their activities in the near future, especially because we do not yet know how much money our recommendation will move.
• Maintain informative and up-to-date websites. Since our shallow investigations were conducted without contacting any organizations directly, we didn’t know what an organization is doing unless they wrote about it publicly.

Write Reviews

We wrote a shallow review for each organization that we do not intend to contact and review more deeply. Each such review includes a paragraph briefly describing the activities of the organization in question and several paragraphs summarizing our evaluation of its impact. The impact evaluations cover the most salient points we learned in reviewing an organization, including unique strengths and weaknesses of its programs and any specific concerns that were relevant to our decision not to pursue further review.

Seek Permission to Publish Reviews

Our next step is to seek permission from reviewed organizations to actually publish the reviews that we have written. We may want to investigate some of these organizations more closely in the future or to work with them in other ways. As such, we value our relationships with them and only want to publish our reviews if the organizations involved believe they are fair and accurate. Therefore, we plan to contact each organization and ask them to read our review and suggest corrections for any inaccuracies they find in the way we have described their organization. We will also offer them the option of withdrawing from our review process, in which case we would not publish any part of our review, but only list their organization as having declined to be reviewed.

We believe that the organizations we have reviewed will ultimately benefit from our publishing the reviews, as they will provide information to visitors using our site as a starting point for learning about the animal advocacy landscape beyond our top recommendations. Some organizations which we did not choose to continue reviewing provide good opportunities for potential volunteers or good resources for animal advocates, and being listed on our site with a review will help them get exposure in the community of advocates and volunteers that they seek to reach. Additionally, not all visitors to our site will share all our values and judgements, and by providing the reasoning behind our decisions, our reviews will offer a way for such visitors to make their own choices about where to donate.