If open reviewing hasn’t come to a journal asking you to review…post your review on the internet

[Another post in the Changing world of scientific publishing series.]

Peer-review activist doesn’t wait for journals to adopt open peer reviewing.

Is this a solution to the trollism and sandbagging that happens in  anonymous peer review but that is undetectable to readers of the subsequently published paper?

Web technology gives us all the tools we need to ditch the lame game of secret peer review and define new rules, a better game, for scienceNiko Kriegeskorte

So often when I write a blog post about a bad article, I’m frustrated by not be able to peek in on the review process. Recently I wrote  about a paper in the British Journal of Psychiatry in which the abstract put a glowing spin on the null findings that were only reported in a table in the paper. I wanted to know how this possibly got unnoticed through review. It would have been great to see the reviewers comments.

Some, but not all journals are now adopting open peer review.

Eva Amsen has explained the https://blog.f1000.com/2014/05/21/what-is-open-peer-review/ :

Benefits for authors and readers

  • Author can see who reviewed their work
  • Reviewer comments put paper in context which is useful additional information for readers
  • Reduces bias among reviewers
  • More constructive reviews
  • Published reports can serve as peer review examples for young researchers.

Benefits for reviewers

  • Shows the reviewer’s informed opinion of the work
  • Demonstrates experience as a reviewer
  • Can take credit for the work involved in conducting the review

When I published Replication initiatives will not salvage the trustworthiness of psychology in BMC Psychology, the final paper was accompanied by earlier drafts and reviewers’ comments. One of the reviewers, Brian Earp made his comments also available on the Internet. I was taken aback, but if I had more time, it could have made for an interesting exchange between Brian and myself with me contesting some of his points.

Sometimes I read journal articles that I strongly suspect got toned down by the editor and reviewers. Recently I saw a preprint, Clinical Science and the Replicability Crisis which might have gotten muted in its criticism of the trustworthiness of the clinical psychology science literature. The circumstantial evidence I had was that one of the authors had previously been more critical in other writings, but he was now incoming editor of an APS journal, Clinical Psychology Science. The APS publications committee is very divided on just how critical comments can be and whether we should name names when pointing to someone who has consistently produces bad science. The article in question praised a Facebook group page as having a suitably civil tone where pointed criticism is as rare as live dodo birds. Did reviewers influence the published paper or did the authors originally so favor niceness over incisiveness and usefulness of criticism?

But now an article has appeared in Wired that discusses how an open publication activist is now posting his reviews on the Internet before even submitting them.

The Rogue Neuroscientist on a Mission to Hack Peer Review

So much for secret, anonymous peer review. The tweet linked to the blog of a neuroscientist named Niko Kriegeskorte, a cognitive neuroscientist at the Medical Research Council in the UK who, since December 2015, has performed all of his peer review openly. That means he publishes his reviews as he finishes them on his personal blog—sharing on Twitter and Facebook, too—before a paper is even accepted.

Scientists traditionally keep reviews of their papers to themselves. The reviewers are anonymous, and publishers protect their reviewers’ identities fastidiously, all in the name of honest, uncensored appraisal of scientific work. But for many, the negatives of this system have started to outweigh the positives. So scientists like Kriegeskorte, and even the journals themselves, are starting to experiment.

Kriegeskorte’s posting policy has made a lot of people uncomfortable. He’s faced resistance from journal staff, scientific editors, and even one scientist who anonymously reviewed a paper that he reviewed openly. “People in the publishing business, my feeling is that they feel that this is deeply illicit,” Kriegeskorte says, “but they don’t know exactly which rule it breaks.” Still, after more than a year of this experiment with exclusively writing reviews on his blog—he’s done 12 now—Kriegeskorte says he’ll never write a secret review again.

Be careful! Don’t try this at home unless you are familiar with the ethics of reviewing manuscripts.

In the requests for reviews of manuscripts, a lot of journals emphasize that the manuscript is to be treated as a privileged communication which should not be divulged without permission. Even when I invite PhD students and postdocs to provide a review that I mentor, I first asked the the journal editor permission to share the manuscript with them.

Fortunately, Kriegeskorte respects ethics, but has developed a procedure by which he can still publicly share his review:

If Kriegeskorte is invited by a journal to write a review, first he decides whether he’s interested enough to review it. If so, he checks whether there’s a preprint available—basically a final draft of the manuscript posted publicly online on one of several preprint servers like arxiv and biorxiv. This is crucial. Writing about a manuscript that he’s received in confidence from a journal editor would break confidentiality—talking about a paper before the authors are ready. If there’s a preprint, great. He reviews the paper, posts to his blog, and also sends the review to the journal editor.

If there isn’t a preprint, things get a little unorthodox. Kriegeskorte emails the authors of the paper and tells them that he’s been invited to review it. This act, of identifying himself to the authors before he’s even reviewed the paper, is normally a no-no. In at least one case, that by itself got him thrown out as a reviewer. He contacts the authors to ask them if they would post their paper on biorxiv or another preprint server, and tells them that he’ll only review the paper if they post it.

The Wired article explains why Kriegeskorte publicly shares his reviews:

All of these reforms are in the name of escaping the bad habits and troll-isms that scientists experience in the peer review process. At highly selective journals like Cell, Science, and Nature, professional editors are often looking for reasons to reject a paper rather than accept it—so unscrupulous, biased reviewers can weight the scales. “It is very easy to kill any paper without ever having to bend the facts,” says Kriegeskorte. “All you have to do is look at the strengths and then look at the weaknesses and then focus attention on the weaknesses.” Review reforms may also rein in the requests for additional experiments that can add years to a PhD student’s life, Marder says.

Beyond that, open review might ultimately be better for the reviewers, who under the old system get no credit for the hours of work they put into each review. And being involved in a group decision process, whether behind closed doors like at eLife or in a public forum, will reduce the chances that reviewers make errors. “I think it’s a really complicated story about where people feel the power dynamic is. And what the danger is,” says Marder—the danger of how an author will take a reviewer’s criticisms, and of how critical a reviewer will be.

ebook_mindfulness_345x550I will soon be offering e-books providing skeptical looks at mindfulness and positive psychology, as well as scientific writing courses on the web as I have been doing face-to-face for almost a decade.

Sign up at my new website to get advance notice of the forthcoming e-books and web courses, as well as upcoming blog posts at this and other blog sites.  Get advance notice of forthcoming e-books and web courses. Lots to see at CoyneoftheRealm.com.