Making Peer Review More Open

“Marginalia” by Open Library is licensed under CC BY 2.0.

Traditional peer review relies on anonymous reviewers to thoughtfully assess and critique an author’s work. The idea is that blind review makes the evaluation process more fair and impartial–but many scholars have questioned whether this is always the case.

Open review has the potential to make scholarship more transparent and more collaborative. It also makes it easier for researchers to get credit for the work they do reviewing the scholarship of their peers. Publishers in both the sciences and the humanities and social sciences have been experimenting with open review for almost two decades now, but it is only recently that open review seems to have reached a tipping point. So what exactly is open review, and what does it entail?

There are two main types of open review: named review and crowd-sourced review. With named review, the names of the peer reviewers as well as their reports are published online alongside the scholarship in question, making them available for anyone to read. A number of open access journal publishers use named review, including BioMed Central, Frontiers, F1000, and eLife. The open access journal Nature Communications allows authors to choose whether or not they want open review. About 60% of them do so. Elsevier recently started allowing peer-review reports to be published alongside articles for a few of its journals. However, peer reviewers do not have to attach their names to reports (and a little less than half do so). Giving reviewers the choice to remain anonymous if they wish, while making the reports open, can be one way to address one of the main critiques of open review: that it will cause scholars to be less candid. Early career faculty might be particularly sensitive to this issue.

With crowd-sourced review, a draft of the article or book is made available online for the public to comment on before it is officially published. This allows for the authors to get feedback from a greater variety of individuals, including people who might never have been approached to be a peer reviewer. One of the first monographs to go through this type of review was Kathleen Fitzpatrick’s Planned Obsolescence (NYU Press, 2010). MediaCommons Press, which hosted this draft of Fitzpatrick’s book, used CommentPress, a WordPress plugin, to facilitate feedback from interested readers. Most recently, Matthew J. Salganik used crowd-sourced review for his book, Bit By Bit: Social Research in the Digital Age (Princeton University Press, 2017). Salganik created the Open Review Toolkit (which uses the annotation tool hypothes.is) in order to help other scholars do the same thing. Crowd-sourced review can occur before traditional peer review, or concurrently. One of the challenges associated with this type of review is letting the world know that the manuscript is online and ready to be commented on. It may require that the author take a more active role in the review process than they are used to. Another challenge is directing readers to leave the type of substantial comments that would be most useful to the author.

In addition to these more formal types of open review, which are usually facilitated by publishers, researchers are also taking it upon themselves to use the web in different ways to solicit feedback on their work. More and more scholars, particularly in the sciences, are posting preprints. Some humanities scholars are even sharing drafts of their work in public Google docs and enabling commenting.

What do you think of open review? Would you be willing to make your peer-review reports public?

A Few Cues About Peer Review

The following is a guest post from Ryan Mulligan, Editor at Temple University Press.

One way a scholar might come into contact with a university press is through being asked by an editor to serve as a peer reviewer for a book manuscript the press is considering publishing. Peer review is essential to checking the scholarly credentials of the books university presses publish and helps ensure that they will find an appreciative audience and have the bona fides to be assets to the academic community. Here are some considerations you might wonder about if you are asked to serve as a peer reviewer.

Good peer reviewers know that they are writing for two audiences: the press (including its editorial board) and the author. They balance their suggestions for the author with evaluation, and even some summary, for the press’s benefit. It is sometimes useful to clarify, for the editor and the editorial board’s benefit, the background into which the manuscript enters and what is at stake in whatever intervention it seeks to make. Good peer reviews note what is new and impactful about the project and point out any weaknesses in the scholarship. They might consider the manuscript’s suitability for course use, the level of the writing, or what set of scholars and fields constitute the manuscript’s likely audience. Beyond who could theoretically find this book relevant, realistically, if the press published this book, is there an appreciable set of scholars who will consider it their job to read it? Reviewers’ comments should also be useful to the author as they revise the book. Have they considered the relevant counter arguments? Are there relevant sources or alternative explanations they haven’t considered? Have they defined their terms sufficiently? Are they inventing terms that aren’t useful? And then the reviewer might need to play editor. Does the argument flow? Does the organization of the manuscript make sense? As a reader, do you feel confident at a given moment in the text, where the author is going and how the section you’re reading contributes to the overall argument?

You don’t need to be a copyeditor – most university presses will take care of the spelling and grammar in a later stage of the publishing process. But especially in fields that are heavy on technical language and jargon, the press may be dependent on you to make some editorial interventions. I see many reports that offer a main section of broad discussion of the manuscript and then a separate section of more detailed comments that tackles individual errors or points of weakness one by one. Some reviewers like to open up the “Comments” section of their word processor and mark up the manuscript as they go. There’s no way for me as an editor to present those comments to my board, but many authors find this generative, so go for it if you like, but be cautioned that it’s much harder for me to keep your identity as a reviewer anonymous from the author when I pass on these comments.

I’ve seen some effective concise reviews of only a page, but most good reviews tend to be two to five pages, more if they catalogue errors or offer detailed commentary. Some reviews are framed as letters to the editor, press, or editorial board, but most are just reports not addressed to anyone in particular. Some presses, including Temple, have a list of questions they will include when they send along the manuscript for review. Reviewers can consider these questions one at a time in turn or just keep them all in mind as they write a more free-form review. Presses, editorial assistants especially, appreciate when the reviewer remembers to turn in any supporting forms completed with their review. They also appreciate reviewers sticking to their deadlines, but being forthcoming and communicative when they can’t make a deadline.

Let’s talk for just a moment about bad peer reviews! The worst reviews I’ve seen are a couple paragraphs thrown together before deadline that just say “you should publish this book.” That’s not going to help the book get past the editorial board; the press is just going to have to turn right around and ask someone else to write a whole new review. But that doesn’t happen very often. Most commonly, a peer review will disappoint me in that its criticisms boil down to, “this isn’t the book I would have written if I’d tackled this topic myself.” I understand the instinct to write that report. But it’s not useful to the press. You will naturally have certain interests that you would have brought to the study to nudge it in a different direction that would be more exciting to you. Your focus and emphasis might have been different. And your writerly voice would of course have been different. But try to put yourself in the press’s shoes and evaluate what the book does for the discipline, what it brings to the table, and how it could be framed to better accomplish its own goals, which may be different from yours.

I’ve seen peer reviews from scholars of all levels of experience; seniority is no predictor of the quality of a review. The quality of the review comes from the reviewer’s ability to speak to both author and press, consider the book’s realistic impact and audience, evaluate its success in reaching its goals, and offer helpful suggestions for improvement. I hope my own suggestions here are helpful for anyone asked to review a book project for a scholarly press.

For more perspectives on peer review for academic presses, including some of the ways that procedures might vary for different presses, take a look at the Association of American University Presses handbook of Best Practices for Peer Review.

Paying for Peer Review?

peerreview

“Peer Review” by AJCann is licensed under CC BY-SA 2.0.

A recent article in Times Higher Education reports that a new British publisher, Veruscript, plans to pay authors for their peer review work. Although paying for peer review is not a new idea, it is only recently that scholarly journals have begun to experiment with the practice.

Perhaps the most prominent mega-journal that pays for peer review is Collabra, which is published open access by the University of California Press. Launched in 2015, Collabra publishes scholarship on life and biomedical sciences, ecology and environmental sciences, and social and behavioral sciences. To cover the costs of making articles freely available, Collabra charges authors an article processing charge (or APC) of $875. Collabra then takes a portion of that money ($250) and places it in a fund to pay editors and reviewers. Reviewers are offered money no matter whether they accept or reject a manuscript. Reviewers can choose to take the money outright, donate it to the Collabra APC waiver fund, or donate it to their own intitution’s open access publishing fund. The amount is low enough that scholars are not motivated to review just because of the money, yet it’s a small way to reward the academic labor that goes into reviewing articles.

Still, the practice of paying for peer review remains controversial within the academy. Many academics feel that peer review is a community service that should not be monetized. Serving as a peer reviewer, they argue, is simply part of one’s job as a scholar. Others point out that there are ways to reward peer reviewers without actually paying them. The Molecular Ecologist, for example, publishes a yearly list of its best reviewers. The open access mega-journal PeerJ offers another kind of incentive for peer reviewers. Under their economic model, individuals pay a flat fee for membership. Membership allows authors to submit papers and preprints. However, in order for authors to keep their membership from lapsing, they must submit one “review” a year. This can be an informal review, such as a comment on an article, or a formally requested peer review (which is by invitation only). Finally, the for-profit company Publons showcases the work of peer reviewers by making it possible for scholars to create Publons-verified profiles in which they list all the journals they have reviewed for. Publons claims that their model will help scholars get credit for what is usually invisible work, as well as give them another way to demonstrate their subject expertise.

What do you think? Should reviewers be compensated for their work?