ACE periodically surveys donors to both our organization and our Top Charities who report that their donations were influenced by our recommendation. This post discusses some key results from our 2016 donor survey, sent in October and November 2016 to donors who had made donations since our 2015 ACE and Top Charity donor surveys were conducted. Among other results, we found that:
- Respondents reported giving more to animal charities because of ACE.
- Many respondents were making their first donations to one or more of our Top Charities.
- The most frequently mentioned alternative use for donations was donating according to the principles of effective altruism, though other responses were also popular.
- Donors to ACE wanted us to improve our reach and the quality and communication of our research.
Using our mailing list of known individuals who donate to ACE and our Top Charities (which at the time of the survey were Animal Equality, Mercy For Animals, and The Humane League), ACE sent out a link via email asking these individuals to anonymously complete a 12-question survey. Top Charities were also asked to send the survey to all of their donors from the past year who had indicated that they were influenced by ACE. Unlike previous years, this survey included questions intended for ACE’s donors as well as donors to our Top Charities. As a consequence, the 2016 survey responses are not meaningfully comparable to previous years without additional filtering.
As with previous surveys however, ACE did not provide a specific incentive to complete it. Consequently, responses to this questionnaire are likely skewed by response bias. Specifically: rather than representing a full range of ACE’s mailing list respondents, survey respondents likely represent those members of ACE’s mailing list who feel most motivated to respond to ACE and our Top Charities. Further, mailing list members do not necessarily represent an accurate cross-section of either ACE or our Top Charity donors. So, while we can learn something about our donors by analyzing this survey data, we should be skeptical about drawing any firm conclusions. In particular, we would expect respondents to be more enthusiastic about ACE and/or our Top Charities than the typical donor who did not respond.
This survey had 183 responses: 170 full responses and 13 partial responses. This survey also used branching logic: not all respondents were shown all questions. Thus, for example, the questionnaire only asked those who had donated to a specific charity how much of a role ACE had played in their decision to donate to that charity.
The survey likely had a response rate of between 10% and 24%. We aren’t sure of the exact response rate because we don’t know exactly how many people Top Charities sent the survey to, or how much those lists overlapped with each other and with ACE’s list. (Since respondents were instructed to complete the survey only once, a high rate of duplication between lists could result in a lower apparent rate of participation in the survey.) The pool of possible respondents had at most 1846 members, which means that if there was little or no overlap between lists used by different charities, the response rate was around 10%. ACE sent the survey directly to 478 donors to ACE and Top Charities, resulting in 116 completed responses, a rate of 24%. Since many responses that could be traced to this email campaign came in before other charities sent out the survey, we think this represents roughly the highest possible response rate for the survey, even accounting for the possibility that some people were sent the survey multiple times. If the sample were representative, the margin of error for multiple choice questions answered by all respondents would be around ±7%.1
How ACE influenced donor behavior
Since we track our own impact largely through the donations we influence, we were especially interested in understanding more about how ACE’s recommendations had (or had not) changed respondents’ behavior. Respondents to the survey indicated that ACE’s recommendations had influenced their behavior in multiple ways:
- Many respondents donated to Top Charities for the first time because of ACE’s recommendations. The survey asked donors which charities they had given to, and for those who indicated having given to a Top Charity in the past year, whether this was their first donation to that charity. The percentage responding that this was their first donation ranged from 49% of donors to MFA to 79% of donors to Animal Equality.
- Most respondents who donated to Top Charities reported that their donations were either “strongly” or “very strongly” influenced by ACE’s recommendations. On a five-point scale ranging from “very weakly” to “very strongly,” the percentage indicating that ACE’s recommendation influenced their behavior either “strongly” or “very strongly” ranged from 78% of donors to MFA to 91% of donors to Animal Equality.
- Most respondents indicated that ACE’s presence increased the amount of their donations to animal charities in 2016. We asked respondents to report whether and how much they had increased the amount of their donations to animal charities because of ACE. 83% of respondents said that they had increased their donations to animal charities because of ACE, and about a quarter of these respondents said that without ACE, they wouldn’t have donated to animal charities at all.
- The most frequently mentioned alternative use for donations was other donations made according to the principles of effective altruism. Around 40% of respondents listed donations made according to the principles of effective altruism as one of the ways they would have used their money if they didn’t donate to ACE or our Top Charities. Approximately half of this group listed these donations as the only alternative use for the money they had donated. Other popular responses were (in order): donations to other animal charities, donations to other farmed animal advocacy charities, and savings.
Overall, we’re happy with the impacts we’re having on donors. We think it’s especially encouraging that respondents reported increasing their donations to animal charities (and to charity in general) based on our work.
What respondents think about ACE
We also use donor surveys to help inform our own outreach and operations. This year we asked donors to ACE to indicate the reasons they supported ACE. The most popular reason was to support research on which interventions are most effective (62% of respondents who reported donating to ACE), followed by support for research on which charities are most effective (47%). To that end, we hope to publish at least three new or revised intervention reports in 2017, and of course we will continue to publish updated charity recommendations annually.
We also asked all respondents to tell us what area of our work was most important for ACE to improve. Since this question accepted free-form text responses, categorization was harder than for other questions on the survey (but we got a more detailed picture of each respondent’s ideas, especially if they chose to answer at length). The most common themes mentioned were:
- Improvements to ACE’s reach were mentioned in 21 responses, including making ourselves better known, making farmed animal advocacy better known as a cause area, and making our recommendations more useful for donors outside the U.S.
- Improvements to ACE’s research quality or communications were mentioned in 17 responses, including improving communication about our cost-effectiveness models, displaying information in formats that are easier to understand, and improving our intervention research.
We greatly appreciate the time put in by everyone who took the survey, and are working to address the suggestions made there.
The margin of error tells us how close we can expect the results of our survey to be to the results of a hypothetical survey that was answered by all members of the population we’re trying to sample. If the survey margin of error is ±7%, then for a question to which 50% of our sample answered “yes” we’d expect with 95% confidence that between 43% and 57% of the total population would answer “yes.” The item margin of error is lower for responses selected by closer to 0% or 100% of respondents, higher for questions answered by fewer respondents, and higher if the sample is not representative in ways that are relevant to the question being asked.