This blog post details our new research review process. Among other changes to our process, we are planning to seek external review for all major research projects. If you’re interested in helping, please fill out the external reviewer sign-up form.
Throughout the past few years, ACE has worked to make our research rigorous and reliable. As our team has grown, we’ve tried many ways of ensuring that our audience can have confidence in the materials we produce. We’ve discussed pieces at great length internally of course, but we’ve also shared draft versions with subject experts outside of our staff—including our board members, members of Statistics Without Borders, staff members at charities we write about, and other members of the effective animal advocacy community. Early on, we even shared draft versions of some of our content with entire Facebook groups. We also receive feedback from the community on pieces that we’ve published, and that can be (and often is) incorporated into revisions or other later work. All of this review can take time, but it has led to much stronger content in many cases. If you’ve offered feedback on our work in the past, we’re grateful for your help.
In the past we’ve made decisions about who should review content and at what stages on a case-by-case basis. That system had some advantages: we were able to try out many options, we could be flexible about our review process based on the needs of each project, and we could make it work with varying levels of resources. It also had disadvantages: we were constantly rethinking what was needed in the way of review, projects were not all subjected to the same level of scrutiny, and often we weren’t really sure when a project was ready to publish or who should make that decision.
Now that we have more experience and a larger team, we’re planning to try something new for our research content. We’re implementing a formal plan for both internal and external review of research projects. This plan incorporates some practices that we’ve found have given us especially good results when we used them in the past, and gives us a clearer path to follow to ensure that each piece of research we publish has received adequate review prior to being published. As our team continues to grow, having a formal plan will help us maintain consistent practices and quality of research.
We are still testing and adjusting our new review process, so we’ve decided not to publish the full details for the time being. However, here is a brief overview of some key features:
- Each project will be assigned a primary critic whose role is to consult with the project leader throughout the life of the project, identify possible critiques of the project or objections to its claims, and ensure that appropriate steps are taken to respond to critiques offered. The primary critic will also help decide when and from whom to seek additional review for the project.
- ACE’s newly appointed Research Editor will review each project early in the writing process—to provide help with structure and content—and then again later in the process for copyediting. This frees the primary critic to continue focusing on substantive critiques.
- Other members of ACE’s research team will review each project at least once, at a time determined by the primary critic.
- ACE will seek external review at least once for larger projects—such as those expected to result in new or substantially updated pages on our site—and for smaller projects as time permits. External reviewers will be provided with some specific questions, but also invited to give feedback on any aspect of the project they feel needs improvement.
- We will modify our review process for our charity evaluation process, which already uses all research staff in substantive roles. We plan to assign project leaders and primary critics to individual sections of our reviews.
The largest change from our past practices is the introduction of the role of primary critic. The role of the primary critic is to strengthen our research output by identifying flaws in research and ensuring they are adequately addressed. They are to work cooperatively with the project leader as much as possible; for example, if they identify a flaw in the research, they should also do their best to describe what steps they expect would resolve it, if applicable. We think that assigning this role to one staff member may be more effective than sharing the responsibilities of editing among multiple staff members, because it more clearly indicates who should speak up if important faults are not adequately addressed.
Right now, we’re phasing in this process by choosing primary critics for existing projects, and we’ll be applying the full process to any new projects we begin. We’re also currently looking for people who are interested in serving as external reviewers for some of our projects. If you’d like to contribute your time and knowledge, please fill out the external reviewer sign-up form to let us know!