In order to maximize cooperation from the organizations we evaluate, we give them the option to decline to be reviewed or to deny the publication of our review (and supporting materials) at any time. Out of respect for the confidentiality of these groups’ private communications, we have not previously discussed the reasons that individual charities declined to be reviewed/published.1 We recently decided that collecting this information would be of value to ACE and our readers, with the added benefit of further increasing our transparency. To that end, we asked 33 charities who had declined to be reviewed/published in the 2015 charity evaluation process or later and for whom we didn’t publish a review in the 2015, 2016, or 2017 charity evaluation process to select 1–3 reasons why they declined to be reviewed/published. They chose these reasons from a list of seven options we provided, one of which allowed the charity to write in further details.2 The remainder of this post outlines and discusses the frequency of different responses (or lack thereof) to our inquiry regarding why these charities had declined to be reviewed/published. It also provides further information about these charities, and briefly discusses some general points relevant to a consideration of why these charities might have made this decision.
Information About Charities that Declined to be Reviewed/Published
The frequency for each of the responses (or lack thereof) are shown in Table 1. Further information about the reason(s) selected by individual charities can be found by following the relevant link on their “View Page” button on our list of animal charities.
|Response (or lack thereof)||Frequency|
|They did not respond to our contact||14|
|They asked not to be listed on our website3||2|
|They responded but didn’t select any reason4||1|
|They chose to wait until the following year to be evaluated||10|
|They were too busy during Animal Charity Evaluators’ evaluation season to participate in the review process||4|
|They disagree with Animal Charity Evaluators’ evaluation criteria, methodology, and/or philosophy||2|
|They do not consider themselves an animal charity||2|
|They do not support Animal Charity Evaluators’ decision to evaluate charities relative to one another||1|
|Animal Charity Evaluators drafted a review that they felt misrepresented or misvalued their charity||0|
|They do not currently have a significant need for more funding||0|
Of the 33 groups we contacted, 12 had declined to be reviewed/published in the 2015 charity evaluation process rather than in the 2016 or 2017 evaluation process.5 Nine of those groups did not respond to our recent email.6 The length of time between those groups declining to be reviewed/published and our asking why they had done so probably plays an important role in explaining why 75% of those groups did not respond. The most frequently reported reason for declining to be reviewed/published was that the charity would like to wait until a later year to be evaluated. The relatively large number of charities selecting this reason may have been due in part to the fact that there were a number of young charities being selected for review. Younger charities may choose to wait to be reviewed in order to better establish their programs and staff and in order to develop a longer track record.
It is worth noting that Mercy For Animals (MFA), a former ACE Top Charity, declined to be reviewed/published in 2017. MFA is one of two groups that declined to be reviewed/published in this year’s evaluation process but that are listed as “Comprehensive” on our published list of animal charities.7 We decided to list those two groups as “Comprehensive” rather than as “Declined to be reviewed/published” because we believed their previous comprehensive review had not yet expired.8 By “not yet expired” we mean that those comprehensive reviews provide information that is not sufficiently outdated to warrant listing the charities of interest as “Declined to be reviewed/published.” As a result of that reasoning, we did not list those two organizations in that way, and therefore of course did not ask them to select explanatory reasons.9
General Discussion of the Reasons Charities Might Decline to be Reviewed/Published
Some charities decline to be published relatively early in the process, following an email in which we seek feedback from an individual within the organization who has the authority to approve or reject a review. In other cases, charities decline to have their review published quite late in the evaluation process, once we’ve discussed specific issues in the review with representatives from the charity and we seem to have a clear idea of what they disagree with. While we think that the reasons do vary from group to group, many of these reasons often seem to stem from some disagreement about how to view effectiveness. Our reviews typically include our concerns about a group’s effectiveness, and groups are sometimes reluctant to publish a review which includes such concerns—even if our review conveys a belief that they are doing quite good work overall.10
It is worth noting that charities’ reported reasons for declining could be influenced by bias—perhaps significantly so. Much animal advocacy research (and social science research in general, for that matter) is susceptible to the influence of various forms of response bias. One key type of response bias that seems likely to be in play in this context is social desirability bias. This bias influences respondents to answer in a way that will be viewed favorably by others, as opposed to answering truthfully. Charities’ responses regarding why they declined to be review/published may be particularly susceptible to social desirability bias given that they might be motivated to report answers that prospective donors would find favorable.11 It is also worth noting that of course ACE may be similarly susceptible to social desirability bias and/or other forms of response bias in our discussion of this or any other topic related to our work.
Further details about the reasons a specific group declined to be reviewed/published may be available by following the relevant links from our list of animal charities. We encourage interested parties to reach out to specific groups for further information, if needed.
Questions for Further Consideration
Further consideration of the following questions could significantly inform and/or update our views in areas related to the topic of this blog post.
- Should we change our policy to be more explicit about the point in the process that charities decline, making it clear that they either declined to be reviewed or declined to be published?
- Should we allow charities to publish responses to their evaluations on our website, similarly to how GiveWell does this?12
- Are there other general points pertaining to why charities might decline to be reviewed/published that are worth mentioning in our future content on this topic?
By “declined to be reviewed” we mean declining to participate in the review process before we have drafted a review. We note that this includes instances in which the charity does not reply to our contact after repeated attempts. By “declined to be published” we mean declining to continue the review process after a review has been drafted, including instances in which the charity stops communicating with ACE without having agreed to the review being published. “Declined to be reviewed/published” refers to both of the two previous scenarios.
Charities were asked to please select 1–3 option(s) from the below list that best explained their decision to decline being reviewed/published:
“[Charity] declined to be reviewed/published because…
- They were too busy during Animal Charity Evaluators’ evaluation season to participate in the review process
- They chose to wait until the following year to be evaluated
- (Optionally, you may choose to include more details, e.g., “because they were developing a new program,” “because they were bringing on new staff,” “because they were in search of a new executive director,” etc.)
- They disagree with Animal Charity Evaluators’ evaluation criteria, methodology, and/or philosophy
- They do not support Animal Charity Evaluators’ decision to evaluate charities relative to one another
- Animal Charity Evaluators drafted a review that they felt misrepresented or misvalued their charity
- They do not consider themselves an animal charity
- They do not currently have a significant need for more funding”
Rather than responding by selecting 1–3 reasons for why they declined to be reviewed/published, these charities instead responded by requesting that they not be listed on our website. It is our general policy not to list charities on our website if they tell us they prefer not to be listed, but we note that we did not list that response as one of the options to select from in our email.
Rather than responding by selecting 1–3 reasons for why they declined to be reviewed/published, this charity instead responded without selecting a reason. They haven’t yet responded to our follow-up email that again asks them to select 1–3 reasons for why they declined to be reviewed/published.
In the 2016 charity evaluation process, both of the groups contacted who declined to be reviewed/published responded to our email. In the 2017 charity evaluation process, 14 of the 19 groups contacted who declined to be reviewed/published responded to our email.
The other group is Animals Australia.
We currently think that reviews expire three years after they are published. I.e., reviews published at the end of the 2015 evaluation process will expire at the end of the 2018 evaluation process. As a result of this view, we now consider several reviews from 2014 to have expired. Expired reviews are archived on our site.
For those interested, some information regarding the possible reasons MFA declined to be reviewed/published can be found in our Reddit ‘Ask Us Anything’ from November, 2017.
Some further discussion of the reasons charities might decline to be reviewed/published can currently be found on our Frequently Asked Questions (FAQ) page.
For this reason it is perhaps unsurprising that no charities reported that ACE drafted a review that they felt misvalued their charity (arguably the most dissuading response for a prospective donor) and that the two most frequently reported responses may be the ones that prospective donors would find most desirable.