How Should We Evaluate Organizational Factors That Affect Charity Performance?

ACE evaluates charities using multiple criteria, in an effort to identify the most cost-effective animal advocacy charities and programs for donors to support. Some of our criteria—such as cost-effectiveness calculations and room for additional funding—address factors that have a clear, direct influence on how effective we would expect additional donations to be. Because of the significant uncertainty involved in our evaluations of these criteria, we also use criteria that are less direct indicators of cost-effectiveness. These give us the ability to evaluate the ways that organizational factors such as staff training and the capacity to learn from experience affect the organization’s success. Two of our criteria that address organizational factors are:

Criterion 6. The charity has strong leadership and a well-developed strategic vision

Criterion 7. The charity has a healthy culture and a sustainable structure

Since organizational factors are less directly tied to cost-effectiveness than some of our other criteria, we’ve sometimes struggled to identify how best to use these factors to identify the strongest donation opportunities. In 2016 we restructured our criteria to include aspects of an organization’s culture beyond the transparency needed for us to conduct our reviews. We found that we weren’t confident which aspects of culture were most important for us to address however, and we weren’t satisfied that our evaluations of organizations’ leadership and strategy were as strongly grounded in evidence as our evaluations of other criteria. We believed our evaluations of criteria 6 and 7 could be much better.

This year, we conducted a review of relevant literature to help us identify the organizational factors that are thus far evidenced to be most strongly correlated with performance.¹ Many common recommendations for nonprofit management do not seem to be evidence-based. Even for those which are supported in the literature, the support often comes from studies with small sample sizes, questionable outcome measures, or many degrees of Researcher freedom. We would expect that some of these studies would be hard to replicate, as is common in other fields with similar issues. We therefore tried to identify factors supported by multiple studies or strong meta-analyses, and we are most confident in our conclusions when they are supported by multiple studies with similar findings. However, we think that even using conclusions for which there is some support from expert recommendations and/or the academic literature is preferable to conducting an entirely ad hoc analysis of organizational factors.

We plan to structure our evaluation of criteria 6 and 7 to focus on the factors identified in our review, as well as the factors that help us to determine the charity’s alignment with ACE’s goals and focus. The rest of this blog post details the dimensions we’ll be using to evaluate charities on these criteria, as well as the reasons we’ve decided to focus on them.

6. The charity has strong leadership and a well-developed strategic vision.

We’ll use this criterion to evaluate charities on their leadership, mission, and strategy. Specifically, we’ll focus on understanding to what extent:

• The charity’s mission emphasizes effectively reducing suffering/helping animals
• The strategy of the charity supports the growth of the effective animal advocacy movement as a whole
• The charity’s board is composed of individuals from a diverse set of occupational backgrounds who have had diverse experiences
• The board of the charity participates regularly in formal strategic planning on behalf of the charity, and involves other stakeholders (e.g., staff, donors, and beneficiaries) in that process

In addition to these four dimensions, we’ll also include any other strong impressions we have about the organization’s leadership and strategy, such as reflections on current leadership staff² and whether the organization seems to generally follow recommended leadership and board practices. While we’ve selected the above focus areas because they seem most important given the available evidence (discussed below) and our goals for the evaluation, we don’t want to miss the opportunity to comment on other factors which may be especially relevant or noticeable for a particular charity.

The charity’s mission emphasizes reducing suffering/helping animals effectively.

This dimension measures alignment of the charity’s goals with the goals ACE seeks to promote through our recommendations. This reduces the risk that the charity’s programs will shift over time to become much less effective in our view. For example, while we might review or even recommend a charity that promotes plant-based diets primarily out of concern for human health, we would need to be aware that their program focus could shift in ways that we wouldn’t expect from a charity running a similar program motivated by concern for nonhuman animals.

The strategy of the charity supports the growth of the effective animal advocacy movement as a whole.

This dimension measures alignment of the charity’s goals and actions with the goals ACE seeks to promote through our recommendations. All else equal, a charity that effectively supports the entire movement in addition to helping animals will ultimately be more impactful than a charity that helps animals but does not support the movement as a whole. We might expect to see a strategy supporting the movement as a whole through activities that benefit the whole movement (including creating research and training advocates), through activities that fill an otherwise unoccupied or neglected niche in the movement, or through evidence that other activists find the charity’s work useful (including direct reports that they do and collaborations with a wide variety of charities and advocates).

The charity’s board is composed of individuals from a diverse set of occupational backgrounds who have had diverse experiences.

The board of a charity has an important role in setting the charity’s strategy and helping it perform effectively. Studies have reported that organizational performance is associated with board performance (as measured by stakeholder perceptions). While many factors may contribute to board effectiveness (and thus improved charity performance), diversity of the board in terms of occupation and experience seems to be one of the best-supported of these factors. Diversity of nonprofit boards by occupation has been found to be associated with better fundraising and social performance.³ Diversity of nonprofit boards by a measure composed of multiple areas—including education, profession, and personal characteristics (e.g., race, gender, age)—has been associated with the use of inclusive governance practices,⁴ which serve to help the board incorporate community perspectives into strategic decision making. These two studies alone might not be persuasive, but there is also a large body of evidence investigating the effect of team diversity on performance in other contexts. While that literature includes some mixed effects, ultimately we are fairly confident that taken as a body it supports the idea that teams composed of individuals with different roles, tasks, or occupations are likely to be more successful than those which are more homogeneous.

The board of the charity participates regularly in formal strategic planning on behalf of the charity, and involves other stakeholders (e.g., staff, donors, and beneficiaries) in that process.

As noted above, the board of a charity has an important role in leading the charity’s strategy, and board performance affects the performance of the organization. It is generally recommended for non-profit strategic planning to be carried out by the Board of Directors with input from other stakeholders, such as staff, donors, and beneficiaries of programs. Formal strategic planning processes, such as those typically recommended, include setting time aside specifically for strategic planning, engaging in deliberate strategic analysis, and drawing up plans to aid in implementing the decisions that have been reached. Organizations with formal strategic planning processes have been found to have better financial and social performance than those with less formal practices and to have more success in implementing strategic decisions.

However, this and other generally recommended board and management practices have not been studied as extensively as team diversity. While there is a large body of literature supporting a correlation between the number of recommended practices an organization employs and its performance as perceived by various stakeholders, the specific practices correlated with performance vary. Additionally, a follow-up study looking at whether increased use of recommended practices predicted increased effectiveness found no correlation after controlling for initial effectiveness. This suggests that the correlation between recommended practices and effectiveness may arise from effective organizations using recommended practices to indicate their strength to potential funders and other stakeholders, rather than from the practices themselves promoting effectiveness.

Nevertheless, we may be able to use this correlation to gain information about which organizations are likely to be most effective, regardless of the reason the correlation exists. It is important to be aware, however, that recommended practices vary across contexts. If an organization is operating in a context where a specific practice is not generally recommended, a model in which effective organizations adopt recommended practices will not allow us to use that practice to test for organizational effectiveness. For us, this means that practices which are often recommended in the U.S. but are not standard practice elsewhere (e.g., nonprofit boards should be relatively large and have little overlap with organization staff or other related parties) can only be used to inform our judgement of organizations based within the U.S. Additionally, we should be aware that if we place heavy weight on any given list of practices, we may incentivize organizations to spend time and effort implementing them, which is ideal if implementing the practices leads to effectiveness, but not necessarily otherwise.

The evidence for the usefulness of strategic planning is a little better than that for many recommended practices, as it has been studied and found to be correlated with effectiveness on its own—not only as part of a larger investigation into the usefulness of generally recommended practices. We’re cautiously including strategic planning on our list of practices to focus on, as well as remaining open to noticing and learning from unusual levels of adherence or nonadherence to recommendations more generally. However, we remain aware of the risk of emphasizing recommended practices to a level that is disproportional to their contributions to charities’ performance.

7. The charity has a healthy culture and a sustainable structure.

We’ll use this criterion to evaluate charities on their stability, their support for staff and volunteers, and their day-to-day management practices. Specifically, we’ll focus on understanding to what extent:

• The charity receives support from multiple and varied funding sources
• The charity provides staff and volunteers with opportunities for training and skill development, helping them grow as advocates
• The charity has staff from diverse backgrounds (e.g., occupation, education) and with diverse personal characteristics (e.g., race, gender, age), and views diversity as a resource that can improve its performance
• The charity provides employees with a workplace free of harassment and discrimination

In addition to these four dimensions, we’ll also include any other strong impressions we have about the charity’s culture and structure, such as whether the organization has survived past leadership transitions, and whether they are receptive to criticism and willing to engage with other perspectives. While we’ve selected the dimensions above to focus on because they seem most important given the available evidence (discussed below) and our goals for the evaluation, we don’t want to miss the opportunity to comment on other factors which may be especially relevant or noticeable for a particular charity.

The charity receives support from multiple and varied funding sources.

Studies have found funding diversity to be correlated with financial stability and with strategic decision making. While these studies are correlational and subject to the problems discussed above for evidence regarding charitable best practices in general, funding diversity is also a common-sense contributor to organizational stability, since it lessens the risk experienced by charities that are heavily dependent on voluntary funding from a few sources. We can also easily look for it both in terms of reliance on multiple specific funders, and in terms of reliance on multiple types of funders (such as program revenue or government grants in addition to private donations).

The charity provides staff and volunteers with opportunities for training and skill development, helping them grow as advocates.

There is a significant body of evidence suggesting that staff training and skill development are positively correlated with organizational outcomes. Much of this research has been done in the context of for-profit organizations, but increasing staff capabilities seems likely to be at least as useful in the context of the charities we review. While for-profit organizations may not capture all of the gains associated with staff training if staff leave for other jobs in the same field, trained animal advocates who leave a charity to continue working in animal advocacy continue to help animals with their new skills.

The charity has staff from diverse backgrounds (e.g., occupation, education) and with diverse personal characteristics (e.g., race, gender, age), and views diversity as a resource that can improve its performance.

As noted above, there is a significant body of evidence suggesting that teams composed of individuals with different roles, tasks, or occupations are likely to be more successful than those which are more homogeneous. Increased diversity by demographic factors such as race and gender has more mixed effects in the literature, but gains through having a diverse team seem to be possible for organizations which view diversity as a resource (using different personal backgrounds and experiences to improve decision making) rather than solely a neutral or justice-oriented practice. We think this is especially applicable within the animal advocacy movement because of the need to include and work with communities not currently well-represented within mainstream animal advocacy groups—with whom we may then be better equipped to communicate.

The charity provides employees with a workplace free of harassment and discrimination.

This is one of many practices which are common-sense contributors to a healthy organizational culture. Providing a workplace free of harassment and discrimination is also key to supporting the workplace diversity we recommend, and to insulating the organization from potential costs of allowing inappropriate behavior in the workplace, such as staff turnover or legal consequences.

Further questions

While we hope that our current analysis will help us write more useful evaluations in 2017, we still have some unresolved questions and expect to continue to learn from research and experience. For example:

1. A correlation between two factors occurs when the presence or absence of one factor implies something about the likelihood of the other factor being present. In this piece, we mostly discuss positive correlations, where two factors often occur together. However, correlation does not imply causation. In the case of a positive correlation between two factors, it could be that the first factor causes the second, or vice versa, or perhaps the two factors are both driven by a third, unobserved, factor. Correlational research alone cannot distinguish these situations. 

2. We searched the literature for evidence of correlations between characteristics of leadership staff and organizational performance, but were unable to find any. We aren’t sure whether this means that (i) there are no characteristics of leadership staff that correlate with organizational performance, (ii) such correlations have not been well-studied, or (iii) we weren’t searching the right literature. 

3. Social performance is the aspect of organizational performance related to how well the organization performs its social mission; in the case of a non-profit, usually this is its primary mission. In the linked study, consultants rated how well organizations (which were all YMCAs) fulfilled their missions in their communities. 

4. Inclusive governance practices help an organization maintain awareness of stakeholders in the organization’s programs, seek information from multiple sources, and involve stakeholders in organizational processes. For example, the board might collect statistics related to those who use the organization’s programs, invite experts to board meetings, or form advisory committees to let the organization receive strategic input from both Board Members and other stakeholders.