Steps associate editors could take to reduce journal’s emission of faulty science.
I would no sooner become an associate editor for a journal of ill repute anymore than I would become the night manager for the Mustang Ranch. In both cases I might be able to affect positive change, but my aims would be so counter to the prevailing business models that the effort would be just too frustrating for everybody.
Okay, it’s unfair to classify Psychological Science as a journal of ill repute, even if the journal has taken some recent knocks to its reputation. Some problems were related to inconsistent thoroughness of peer review. Others have been compounded by carelessly letting poor science get through peer review and then making a big fuss about it in press releases. Both sets of problems are consistent with a business model favoring newsworthiness and claims of scientific breakthrough over what is robust and trustworthy.
Brent Roberts and Ayse Uskul are new PS Associate Editors. I don’t believe I have met either of them, but I hear nothing but good about them. I congratulate them on their new positions. I hope they are up to the task of restoring the credibility of PS.
Despite the discontent with PS that I share with many others, I would see it as ungrateful and uncool for Roberts and Uskel to publicly denounce the past editors or editorial policies of the journal
However, I do have ten suggestions for changes that they should immediately, even if discreetly, introduce. Impatient critics need to appreciate that change will take time to become apparent in published articles. Another “sadness impairs color perception” article could well be in the pipeline. If so, the journal will have to take another drubbing on PubPeer. Yet to be fair, we need to assume that the influence of Roberts and Uskel is taking time to appear and that such articles are not their doing.
Here are my ten suggestions, which I hope that we will soon see evidence of being implemented.
Abandon Pink Floyd rejections
An outgoing editorial by a recent PS
first editor gloated about how he built the reputation of the journal. Notoriously, he heavily relied on desk rejections, i.e, not sending manuscripts out for review and depriving authors of any useful feedback. Among the reasons:
The Pink Floyd Rejection: Most triaged papers were of this type; they reported work that was well done and useful, but not sufficiently groundbreaking. So the findings represented just another brick in the wall of science.
We all know what can happen when the representation of psychological science is deprived of too many well-crafted bricks in the wall. More importantly, we also need to appreciate the considerable inducement to questionable research practices by junior and senior psychologists when they are compelled to demonstrate that their solid work is not simply another brick in the wall.
Adopt the Pottery Barn rule.
The American chain of high-end mall stores may not actually have a Pottery Barn rule , but journals should advertise and adhere to a Pottery Barn Rule.
Once a journal has published a study, it becomes responsible for publishing direct replications of that study. Publication is subject to editorial review of technical merit but is not dependent on outcome. Replications shall be published as brief reports in an online supplement, linked from the electronic version of the original.
Currently, attempted replications of articles that appear in PS are herded into groups and led off to ghettos. This strategy serves to punish those so foolish as to attempt to improve the trustworthiness of the journal. The visibility of such large assembled groupings also serves to tarnish the reputation of the journal. Note that attempted replications of articles originally appearing in Psychological Science contributed heavily to the proportion of failed replications in the report of the Open Science Collaboration: Psychology (OSC) article in Science.
Adopt CONSORT, the Consolidated Standards for Reports of Clinical Trials, and require completion of a checklist modifying to fit the kinds of studies being submitted to the journal and the details needed for the independent interpretation interpretation.
Journals of the Association for Psychological Science (APS) are the only credible journals publishing psychological studies that have not formally adopted CONSORT.
The first requirement of CONSORT is that titles and abstracts of studies reporting interventions be prominently labeled as randomized trials. My colleagues and I critiqued an article in PS that did not even mention that it was a clinical trial until the supplementary materials. The article was basically a poorly reported randomized clinical trial (RCT) with null findings, although dressed up to look like something very different. Its publication in PS was exploited in promotion of expensive workshops and other positive psychology merchandise. If reviewers had been initially alerted that this was a RCT, they would have been more likely to notice obvious flaws in the conduct, reporting, and interpretation of this trial.
CONSORT is a useful guide to what to include in reporting clinical and laboratory studies where research participants are randomized to conditions that differ in manipulations intended to affect adaptational outcomes. Beyond that, CONSORT represents good reporting of experimental psychology, something often lacking in PS – accounting for all subjects who were randomized, allotment concealment, blinding of experimenters, and primary and secondary outcomes.
Studies appearing in PS do not typically involve interventions with clinical populations. Nonetheless, more frequently than noticed, PS articles involve manipulations intended to demonstrate how human well-being can be improved. The authors then go on to generalize to what can be accomplished in everyday environments. Such articles typically consist of reports of single or a few underpowered studies that are poorly reported in the PS article, but nonetheless are subsequently used to make broad generalizations in press releases and subsequent articles in the popular and business media. Adherence to CONSORT would facilitate independent evaluation of the claims being made because of greater transparency about the methods and results.
Adopt CONSORT for abstracts
Readers form strong, even immutable opinions about articles after only scanning abstracts. There is a funneling process from finding a title in an electronic bibliographic search or citation in another article or press release to retrieving and reading the actual article. Much of the progression stops with the abstract.
PS abstracts are about as uninformative as a Taco Bell advertisement when it comes to allowing independent evaluation and comparison. CONSORT has an extension checklist for abstracts. Even if the PS editors are not keen on adopting CONSORT for abstracts, they could at least encourage more informative abstract by adopting the traditional IMRAD format.
Something needs to be done about woefully uninformative abstracts in PS that foster misunderstanding of what the articles actually contain. There is evidence from other sources that misrepresentations of scientific articles in the media are often tied directly to exaggerated claims by the authors in abstracts. Reviewers should therefore be encouraged to check the correspondence to what is said in the abstract and was contained in the article.
Require disclosures of conflict of interest.
Some authors seek a greased path from an early-release PS article to an op ed piece in the New York Times to TED talks and then on to lucrative corporate speaking engagements or promotion of self-help books. Such a path of course brings temptations to report results serving to fan the hype and suppress the rest. Readers deserve routine disclosures of conflicts of interest so they can decide for themselves whether an APS article deserves particular scrutiny because of what is at stake professionally more financially for the authors.
Prioritize what articles will receive press releases and vet what is said about them.
There have been some self-inflicted embarrassments to PS from press releases attracting scrutiny to articles, the credibility of which cannot sustain such attention.
Don’t depend on being able to create an amnesia in readers scrubbing the Internet of all press releases. Some readers are on to you and preserve links that can be put back on the web.
Searching for old PS press releases can be an entertaining game. There are more out there, but one can get a good chuckle from the 2011 press release entitled “Life is one big priming experiment…” which declared:
One of the most robust ideas to come out of cognitive psychology in recent years is priming. Scientists have shown again and again that they can very subtly cue people’s unconscious minds to think and act certain ways.
Ah, yes, 2011 was when PS was in its prime.
Lower the threshold for accepting letters to the editor critical of articles that you have published.
There is a risk that the de facto place for finding critical commentary on PS articles will become PubPeer. Critics can find the advantages of anonymity, peer support, and a wonderful process in which the next comment builds on previous ones. However, PS ceding post publication peer review commentary to this outlet leaves a lot at risk to both the authors of the paper being criticized in the reputation of journal itself. PS is also ceding a long of control: editors can no longer monitoring comments will give the authors of the precise paper the traditional last word.
Maybe as an expansion of the Pottery Barn rule, PS could accept responsibility for publishing reasonable criticisms if they publish an article.
More generally, PS should accept the post publication peer review is not going to go away and should find ways of better harnessing it and improving the quality of the science PS presents the world.
Exercise the same peer review and tone policing of authors’ responses to critical letters to the editor as you do for the letters themselves.
I’ve talked to other people who have attempted to submit letters to the editor. We find we share a common complaint. We have had excessive fussing editing of our letters for tone, only to receive gratuitous nastiness from authors who flagrantly ignore the points that we have made. This becomes a further incentive to move post publication peer review elsewhere. It also invites counterattack in the social media.
Comply with the Committee on Publication Ethics (COPE) and establish explicit procedures for appeals of editorial decisions and stick to those procedures.
The journal endorsed COPE procedures, so now live up to them, starting with publishing procedures. Authors disputing what they construe to be unfair and arbitrary editorial decisions from PS editors currently have no recourse in terms of a published set of procedures by which they can make an appeal.
Again, talking to a number of people who have submitted letters to the editor, I have found the associate editors become very defensive about decisions that they have made, getting overly protective of their friends and cronies among the authors being criticized, and arbitrarily and summarily reject appeal. Unfair opportunities are offered to authors who would been criticized to censor and even veto publication of criticism.
I know, it’s tough being an associate editor, but in many situations it’s best to recluse oneself from what should be orderly, laid out, fair procedure of appeal.
Replace the badges program with warning labels for articles that do not (1) preregister,(2) place key materials in an open access repository, or (3) make their data available without interested parties going through a freedom of information act request or proving that their interest in the data is not vexatious.
Really, at the APS headquarters in or in your own universities are badges awarded for flushing after using the toilet? Adopting these practices should become so commonplace that badges become silly. Articles don’t have these features should be less frequent, but shameworthy.
Readers may have noted some embarrassment to the current badge system in all the uproar over the sadness impairs color perception article. When criticized in post-publication peer-review, the authors took down the data for which they had a badge for having uploaded and re-uploaded it with curious evidence of manual changes.