Evaluating the Effectiveness of Charitable Programs
Program evaluation is a valuable guide to program development, but it can also be intimidating to start. Maybe you’re imagining a randomized controlled trial of your programs, and you know you don’t have the resources for that. Or maybe you don’t even know what you would evaluate. But just about all programs can be evaluated in some way, and often it’s easier than you think. And with self-evaluation, you set the goals and priorities, so you can make sure they’re meaningful to you. Here’s how to make it work for you:
1. Set measurable goals.
Animal advocates often have sweeping, qualitative goals, like ending factory farming or speciesism. These are worthy and inspiring ends, but it’s hard to tell how much progress you’re making if you frame all your goals in very broad terms. To do meaningful self-evaluation, you’ll need some goals towards which you can make clear, measurable progress.
If you already have measurable goals that mark the path towards accomplishing your broader vision, that’s great. If not, take some time to work out what short- or medium-term goals you’ll need to reach on your path to helping animals. What could you see happening in the next week, month, or year that would help you know you’re on the right track—or that you need to make a change?
Phrase your goals as specifically as possible. For example, if you want to get more media coverage of your organization’s activities (whether they’re protests, investigations, rescues, or social events), don’t leave things there. Set a goal number of media stories or people exposed to your message that you hope to reach each month.
Once you’ve come up with some goals, look over them one more time. Are they really describing things that are connected to your overall mission, or just things that are convenient to measure? If you’ve set some goals that aren’t connected to your mission, tweak them so they are connected, or get rid of them altogether. If you’re evaluating something you don’t think truly matters, you’ll just feel like you’re wasting your time.
2. Find a baseline.
For each one of your specific, measurable goals, figure out how you’re doing right now, or what you’ll be comparing to when you do your program evaluation.
If your goals are tracking your own actions, this is easy. If the goals deal with something you were already tracking or can look up, dig through your recent work to find out how you’ve performed on them recently. Need to know how many people saw last month’s Facebook posts? Facebook has been tracking them all along; just check with the person who runs your page. If they deal with something you can easily observe but weren’t tracking, use next month as a baseline. Have a goal of getting 10 more people to an in-person demonstration every week? Start counting the next time you hold it, so you know what’s going on now.
If your goals call for changes in the state of the world or in other people’s behavior, things might be trickier. For large scale changes, do your research. If your goal is to reduce the number of chickens in battery cages by 5%, you’ll need to know how many chickens are in battery cages now. If you can’t find out, your goal isn’t really measurable. But if you can, the process might also overlap with the research you needed to do anyway in order to figure out who you’ll need to persuade to help you. If you’re looking at small scale changes, like convincing individuals to change their diets, you may need to use surveys to check whether you’re meeting your goals, so consider whether you want to also survey people when you first encounter them as a baseline, or to use general information about the population you’re working with. Your own survey will be more reliable if you do it well, but using data from another source will be cheaper and easier.
3. Record information about changes.
Once you’ve found a baseline, the next step is to continue collecting information that will help you see what progress you’re making. Often, you’ll be tracking the exact same data you were for the baseline, just for different weeks or months. Set up a single spreadsheet, folder, or email label to help you keep track as you go along, so that you don’t need to search for information later. Use a calendar reminder to make sure you’re collecting data regularly, so that you don’t forget, especially if there are things you need to write down while they’re happening, such as the number of people who stop by your table at an event.
If you need to follow up with others in order to see whether you’re meeting your goals, what you do now might be different from what you did while you were setting a baseline. Set a schedule for following up with people if there are only a small number you need to check in with, for instance, if you need to make sure that companies are following through on their commitments to you. If you need to check up on individual pledges to change diet or become more involved in advocacy, you’ll probably need to contact a representative sample of people with a survey. This is one of the most involved types of self-evaluation, so we recommend getting help if you need it. Start with our survey guidelines and question bank, and don’t hesitate to get more help, from ACE or from another group like Faunalytics or Statistics Without Borders.
4. Interpret the data.
After you’ve been tracking your progress for a while, or after you’ve worked with a group of people and then followed up to see whether they’ve made the desired changes, it’s time to check whether you’re reaching your goals. Make sure you have all the relevant data collected in one place, then start working with it to make it understandable. If you’re counting something like the number of people reached or animals placed in homes, find weekly and monthly totals. Graph your performance over time so you can easily see long-term trends or times when you did especially well or poorly.
If you’ve contacted individuals for follow-up, sort their responses into relevant categories. Find the percentage of people who behaved as you hoped. Don’t forget to compare answers to different questions if relevant: if you have group of people who were vegetarian when you did a follow-up survey, how many of them also said they were vegetarian before they learned about your program? If you’ve sent your survey to people who went through more than one of your programs, separate out the results so you can tell if one program is doing better than another. Again, use graphs and tables to help you see patterns in the data.
Don’t forget that, whatever you’re tracking, some of the variation in your results will be due to chance. If you’re seeing only small changes, be cautious about assuming they’re connected to something you’re doing. Big changes and changes that cover a long period of time or a lot of data points are more likely to be persistent and reflect real changes in your effectiveness. If you’re not sure whether the changes you’re seeing are dependable, try asking someone for help with the statistics, whether a paid professional or a volunteer who does statistical work in their professional life.
5. Draw actionable conclusions.
If your goals were measurable, you should now know whether you met them!
If you’re meeting your goals in every area, congratulations! Take some time to reflect. Things seem to be going well for you. Do they also feel like they’re going well on a larger scale? Are your expectations too small? Maybe you need more ambitious goals for the future, or maybe you’ve just had a really good few months. If there are new things you started doing in order to meet your goals, it sounds like you should keep doing them. Or maybe you should modify them to see if you can do even better.
More likely, there are some areas where you did really well, and others where you had setbacks. Your goal may have been to get 3 new volunteers per month, and your average was okay, but there was one month when you didn’t get any new volunteers at all. It may be disappointing not to have done perfectly, but this is actually the situation you want to be in. You’ve set yourself challenging goals which allow you now to see where you excelled and where you might be able to do a little better. Consider why you did better at some goals than others. Is one of your programs going better than another? Maybe that’s what you’re really good at, and you should shift focus so you can expand it and have an even bigger impact. Or maybe the other program is promising, but there are some things that didn’t go as planned and you know how to fix them for next time. Did you miss your goal number of new Facebook followers one month but do much better the next month? Look back at your activity to see whether that had to do with what or how much you were posting, or compare your progress on that goal to progress on other goals that might be related. Make a plan that can help you do better in the coming months, based on what you have learned.
Maybe you didn’t quite reach any of your goals. That’s okay, too: you were probably just a little overambitious in setting them, especially if this was your first try. Look back and try to see what was different about times when you were achieving the most. Compare your progress in different areas to see where you’re spending your time and how that relates to what you’re accomplishing. And think about the big picture: do you have so much going on that you could reach any one of your goals, but not all ten at once? Or is the number of goals right, but all your actual targets were too high? You’re not living up to your own expectations, so there must be something you want to change, whether it’s narrowing your focus to do one thing really well, or making small tweaks to all your programs. Figure out what that thing is – and then, if it’s a long list, remember that you tend to aim big, make sure the top items on the list are the highest priority, and focus on those.
6. Reset and start again.
Now you’ve set goals, measured your progress, and planned a course of action based on whether you reached them. Program evaluation complete!
For a day or so, anyway. The best way to continue making progress is to do it all again. Not only will you then get the benefits of measuring how well your new plan works, but you’ll also get better at setting goals and evaluating your success each time you do it. Did you have a goal that turned out not to be measurable when you went to set a baseline? Now you get to reformulate it or replace it. Is there a question you asked on your survey that you couldn’t actually use the answers from? Use that space for something else, or make the survey shorter and get more of the people you give it to to respond to all the questions. For goals and tracking methods that worked this time, tracking the next period will already be a habit, and analyzing the results will be easy: just do the same thing again.
Make a habit of reviewing your progress yearly, or seasonally if your programs naturally fit into the school year or another schedule. You’ll get to see all that you’re accomplishing, and all the ways you help animals that might not be noticeable on a daily basis. This will give you the tools to do even better and increase your impact in helping animals.