Understanding the Peer Review Process: An Interview With Michael Willis (Part 1)
In this interview, Enago’s Kuntan Dhanoya (Vice President, Business Development) had the opportunity to speak with Michael Willis, Senior Manager, Peer Review at a leading publisher. Michael is also currently the President of the International Society of Managing and Technical Editors (ISMTE) and a member of the Management Committee at PEERE.
In the first part of this interview series, we will take you through Michael’s journey in the publishing industry and the various challenges of the current system as well as the developments that are taking place to improve the peer review system. Michael also shares some key issues faced by peer reviewers and how it is becoming increasingly important to have a defined training program for peer reviewers. In addition, Michael discusses his role at PEERE and the primary objective behind this initiative.
Kuntan: Michael, you have had an incredible journey working in the scholarly publishing industry. Could you tell us more about this journey, and did you always want to specialize in peer review management?
Michael: My first job was in a faculty library at a university. From there, I moved very quickly to a small startup company in the early days of online journals, so that was really my first exposure to academic publishing and dealing with content, authors, and publishers. I then moved in 2005 to the company that I work for now, in a peer review management role. This was my first introduction to peer review, and since then I have been working pretty much in peer review the whole time. I wouldn’t say I started off with a burning ambition to work in peer review but I fell into it by accident.
Kuntan: There has been a lot of discussion about alternatives to the traditional peer review process, and we have seen publishers come up with new models such as collaborative peer review and post-publication peer review. We are also seeing new initiatives like PRE (Peer Review Evaluation) and Publons. In your opinion, what is an ideal peer review system, and what are the challenges for journals or societies seeking to get closer to an ideal peer review system?
Michael: I wouldn’t say that the traditional form of peer review is dead or dying. Instead, I think that it could use some reform in order to redress some of the unbalanced views of peer review. However, sometimes one or two bad apples make the whole thing look bad, even though there are a lot of good apples in traditional peer review. The vast majority of peer reviews work well and there is still a lot to be said for the traditional form of pre-publication peer review. If there are new models of peer review now, it is probably because we have opportunities for exploring new avenues, particularly with the Web, that we didn’t have formerly. Along with these new technologies come new ways of doing things. For example, it wasn’t practical to have post-publication open peer review until very recently. However, I think that traditional pre-publication closed peer review still has a lot of mileage left.
Some of the other interesting things emerging in peer review at the moment are about how we recognize peer review as an activity. There is no question that peer review is really demanding on researchers’ time. Researchers spend a lot of time doing peer review that always largely goes unrecognized, which can make people angry or frustrated. One key for the future for peer review is how we recognize reviewers’ efforts. This could be in all sorts of forms and may involve giving reviewers tangible rewards or make employers, institutions, and funders recognize peer review as a valid research activity. There is a whole spectrum of possibilities.
Kuntan: Should reviewers have some common training material available or common resources, and would that help in eliminating some issues or challenges that we see in peer review today?
Michael: In this regard, I certainly think there is a lot to be done not only in terms of training existing reviewers but also in training a new generation of reviewers in how to review. I don’t know whether a shared common resource is necessarily the best way forward, but training is definitely needed. Surveys of peer reviewers have shown that even the best ones and those with the longest experience could benefit from and would appreciate training. Often, training has taken place rather informally, for example, a junior researcher may have been mentored by his or her PI in their lab, and given the opportunity to do some peer review alongside them. When thinking about how we can train the next generation of reviewers in a more formal way we need to not only think about the position of publishers but also that of managing editors of journals. This training could include not just giving them guidance on how to do things, which I think most journals do anyway such as generic guidance explaining the editor’s expectations on how to review a paper but also giving them slightly more concrete and formal guidance about the sorts of things that need to happen in peer review. For example, what the reviewer should be looking for and what the reviewer’s role is in the publication process. This could also include dealing with anything from ethics to the quality of the methodology in a manuscript, whether the science is reproducible, whether there is enough information to make it reproducible, and looking at data.
Kuntan: What, according to you, is the first step in this direction?
Michael: I think that a cultural change needs to happen, at the research institution level and also at the funder level as well, because unless reviewers realize that they are going to get some sort of appreciation or recognition for their work, they are not going to have the incentive to undertake any sort of training. It will still be carried on as a fairly informal sort of activity. So I think that research institutions or organizations that employ reviewers need to allow their employees to spend time on peer review as an activity.
The whole question of peer review recognition was actually the theme of the Peer Review Week in September 2016, with 16 organizations participating which included the ISMTE. However, there are other organizations involved in the whole publishing ecosystem, and I think it would be really interesting to see whether any consensus emerges about how reviewer recognition could be taken forward in the industry as a whole.
Kuntan: You have been part of PEERE for a year. Could you tell us about this new initiative?
Michael: PEERE is a European Union–funded project and is a formal part of the EU’s program in science and technology running for four years, from 2014 to 2018. Over 30 member countries are represented, and the theme of the initiative is about new frontiers in peer review. You can find more information on the website: http://www.peere.org/.
There are various stakeholders, but it consists primarily of a collection of researchers, including editors-in-chief and people who are senior researchers in their institutions. They share a collective interest in peer review as an activity and a desire to find a fair, evidence-based approach to peer review. The committee chairman once described it as opening up a black box on peer review; however, it is about accurate in this case. My role as the representative of a publisher and one of the project stakeholders is to feed data from our journals into the project on which the researchers can then conduct analysis. At the moment, the trouble with a lot of research on peer review is a shortage of data. We have a few journals doing research with a few bits of information from their own journals, or a group of journals, but nothing on a large scale. PEERE’s ambition is to make that happen. Getting data from large publishers to feed into research projects can help us reach conclusions about peer review that can then be implemented or used to reform the process. The effort is primarily on journal peer review, but PEERE is interested in looking at grant peer review as well.
Kuntan: Any examples of the research you or the group is doing on peer review, such as the topics of those papers?
Michael: For me, the most interesting topic is what motivates peer reviewers to do reviews. A couple of groups are working on this, and there are some interesting models, mostly sociological, of the behavior of peer reviewers and how that informs our decisions in selecting reviewers or encouraging people to do reviews. If a peer reviewer is driven primarily by financial gain, he or she won’t get much out of the experience, because there isn’t much financial gain in peer review. The question, then, is whether to increase the financial payoff to try to attract reviewers. This approach is fairly theoretical at this point, but the question is important. I think that as of now we have more information on why people decline to do reviews than on what makes them reviewers—at least that is how it strikes me—and we need both sides of the equation.
One of the big issues about peer review is the assumption that peer review is in crisis because we don’t have enough reviewers, which leads people to ask why we don’t have more. The flip side of that is why people do agree to be reviewers and what incentives are there to persuade them to do reviews. That goes back to our earlier discussion about recognition, for example.
Kuntan: What has been your biggest takeaway or learning experience from working on this initiative for the past year?
Michael: For me personally, one of the most exciting things is working with a group of researchers who come from all sorts of different backgrounds and contexts. There is a good representation in terms of discipline—a lot from economics and sociology, but also biomedicine and mathematics. They come from all over Europe, and all are interested in this question of peer review and whether there are valid ways to improve it. Often in my day-to-day work, I can get pretty focused on one or two particular journals, but this is a global phenomenon of interest to people everywhere. It has been really interesting for me to get the perspectives of researchers and editors-in-chief across Europe and outside the areas that I normally work in.
(This interview is a part of our interview series of connecting scholarly publishing experts and researchers.)
