This blog post details our new research review process. Among other changes to our process, we are planning to seek external review for all major research projects. If you’re interested in helping, please fill out the external reviewer sign-up form.
Throughout the past few years, ACE has worked to make our research rigorous and reliable. As our team has grown, we’ve tried many ways of ensuring that our audience can have confidence in the materials we produce. We’ve discussed pieces at great length internally of course, but we’ve also shared draft versions with subject experts outside of our staff—including our Board Members, members of Statistics Without Borders, staff members at charities we write about, and other members of the effective animal advocacy community. Early on, we even shared draft versions of some of our content with entire Facebook groups. We also receive feedback from the community on pieces that we’ve published, and that can be (and often is) incorporated into revisions or other later work. All of this review can take time, but it has led to much stronger content in many cases. If you’ve offered feedback on our work in the past, we’re grateful for your help.
In the past we’ve made decisions about who should review content and at what stages on a case-by-case basis. That system had some advantages: we were able to try out many options, we could be flexible about our review process based on the needs of each project, and we could make it work with varying levels of resources. It also had disadvantages: we were constantly rethinking what was needed in the way of review, projects were not all subjected to the same level of scrutiny, and often we weren’t really sure when a project was ready to publish or who should make that decision.
Now that we have more experience and a larger team, we’re planning to try something new for our research content. We’re implementing a formal plan for both internal and external review of research projects. This plan incorporates some practices that we’ve found have given us especially good results when we used them in the past, and gives us a clearer path to follow to ensure that each piece of research we publish has received adequate review prior to being published. As our team continues to grow, having a formal plan will help us maintain consistent practices and quality of research.
We are still testing and adjusting our new review process, so we’ve decided not to publish the full details for the time being. However, here is a brief overview of some key features:
- Each project will be assigned a primary critic whose role is to consult with the project leader throughout the life of the project, identify possible critiques of the project or objections to its claims, and ensure that appropriate steps are taken to respond to critiques offered. The primary critic will also help decide when and from whom to seek additional review for the project.
- ACE’s newly appointed Research Editor will review each project early in the writing process—to provide help with structure and content—and then again later in the process for copyediting. This frees the primary critic to continue focusing on substantive critiques.
- Other members of ACE’s research team will review each project at least once, at a time determined by the primary critic.
- ACE will seek external review at least once for larger projects—such as those expected to result in new or substantially updated pages on our site—and for smaller projects as time permits. External reviewers will be provided with some specific questions, but also invited to give feedback on any aspect of the project they feel needs improvement.
- We will modify our review process for our charity evaluation process, which already uses all research staff in substantive roles. We plan to assign project leaders and primary critics to individual sections of our reviews.
The largest change from our past practices is the introduction of the role of primary critic. The role of the primary critic is to strengthen our research output by identifying flaws in research and ensuring they are adequately addressed. They are to work cooperatively with the project leader as much as possible; for example, if they identify a flaw in the research, they should also do their best to describe what steps they expect would resolve it, if applicable. We think that assigning this role to one staff member may be more effective than sharing the responsibilities of editing among multiple staff members, because it more clearly indicates who should speak up if important faults are not adequately addressed.
Right now, we’re phasing in this process by choosing primary critics for existing projects, and we’ll be applying the full process to any new projects we begin. We’re also currently looking for people who are interested in serving as external reviewers for some of our projects. If you’d like to contribute your time and knowledge, please fill out the external reviewer sign-up form to let us know!
Jennifer Liepin says
Thank you for your reviews.
As someone who just had a challenging experience working at Farm Animal Rights Movement, it is exciting to know that people can read reviews on your site before deciding where to volunteer or attempt to work at.
I am glad that you are constantly re-evaluating how you review animal charities, and I certainly will be submitting an app to help out.
For those considering donating or working at FARM, I recommend reading the following about my experience working as a Tour Operator on their 10 Billion Lives Tour:
http://nynomads.com/2017/03/why-you-may-not-want-to-work-for-the-farm-animal-rights-movement/
I hope the best for this org, but want this feedback to be out there for others to consider.
Allison Smith says
Thanks, Jennifer!
For you and any others interested in helping us know more about specific charities, I want to clarify that the best thing you can do to help is to reach out and share with us your experiences. Because individual charity reviews are so sensitive, there’s a very limited group of people we can share them with before they’re published. But we can take in lots of different perspectives as we prepare them in the first place.