Two years have passed since I requested release of the PLOS One PACE data, eight months since the Expression of Concern was posted. What can we expect?
If we spot some usually unrecognized connections, we can see the PLOS One editors are biased towards the PACE investigators, favoring them over other stakeholders in whether the data are released as promised..
Spoiler: The PLOS One Senior Editors completed the pre-specified process of deciding what to do about the data not being shared. They took no action. Months later the Senior Editors reopened the process and invited one of PACE investigators Trudy Chalder’s outspoken co-authors to help them reconsider.
A lot of us weren’t cynical enough to notice.
International trends will continue toward making uploading data into publicly accessible repositories a requirement for publication. PLOS One has slowed down by buying into discredited arguments about patient consent forms not allowing sharing of anonymized data.
PLOS One is no longer at the vanguard of open science through routine data sharing.
The editors’ section of the Expression of Concern ends with:
In spite of requests to the authors and Queen Mary University of London, we have not yet received confirmation that an institutional process compatible with the existing PLOS data policy at the time has been developed or implemented for the independent evaluation of requests for data from this study. We conclude that the lack of resolution towards release of the dataset is not in line with the journal’s editorial policy and we are thus issuing this Expression of Concern to alert readers about the concerns raised about this article.
This is followed by the PACE investigators’ response:
Statement from the authors
We disagree with the Expression of Concern about our health economic paper that PLOS ONE has issued and do not accept that it is justified. We believe that data should be made available and have shared data from the PACE trial with other researchers previously, in line with our data sharing policy. This is consistent with the data sharing policies of Queen Mary University of London, and the Medical Research Council, which funded the trial. The policy allows for the sharing of data with other researchers, so long as safeguards are agreed regarding confidentiality of the data and consent as specified by the Research Ethics Committee (REC). We have also pointed out to PLOS ONE that our policy includes an independent appeal process, if a request is declined, so this policy is consistent with the journal’s policy when the paper was published.
During negotiations with the journal over these matters, we have sought further guidance from the PACE trial REC. They have advised that public release, even of anonymised data, is not appropriate. As a consequence, we are unable to publish the individual patient data requested by the journal. However, we have offered to provide key summarised data, sufficient to provide an independent re-analysis of our main findings, so long as it is consistent with the REC decision, on the PLOS ONE website. As such we are surprised by and question the decision by the journal to issue this Expression of Concern.
Check out my critique of their claim to have shared data from the PACE trial with other researchers-
Conflict of interest: Nothing to declare?
The PACE authors were thus given an extraordinary opportunity to undermine the editors’ Expression of Concern.
It is just as extraordinary that there is no disclosure of conflict of interest. After all, it is their paper is receiving expression of concern because of the authors’ failure to provide data as promised.
In contrast, when the PLOS One editors placed a discreet Editors Note in 2015 in the comment section of the article about the data not being shared when requested, it carried a COI declaration:
Competing interests declared: PLOS ONE Staff
That COI aroused the curiosity of Retraction Watch who asked PLOS One:
We weren’t sure what the last line was referring to, so contacted Executive Editor Veronique Kiermer. She told us that staff sometimes include their byline under “competing interests,” so the authorship is immediately clear to readers who may be scanning a series of comments.
Commentary from Retraction Watch
A spokesperson for PLOS told us this is the first time the journal has included a statement from the authors in an EOC:
This has been a complex case involving many stakeholders and we wanted to document the different aspects of the case in a fair manner.
We asked if the journal plans to retract the paper if the authors fail to provide what it’s asked for; the spokesperson explained:
At this time, PLOS stands by its Expression of Concern. For now, we have exhausted the options to make the data available in accordance with our policy at the time, but PLOS still seeks a positive outcome to this case for all parties. It is our intention to update this notice when a mechanism is established that allows concerns about the article’s analyses to be addressed while protecting patient privacy. PLOS has not given the authors a deadline.
Note: “PLOS did not given the authors a deadline.”
One of the readers who has requested the data is James Coyne, a psychologist at the University Medical Center, Groningen, who submitted his request 18 months ago (and wrote about it on the PLOS blog site). Although some of the data have been released (to one person under the Freedom of Information Act), it’s not nearly enough to conduct an analysis, Coyne told us:
This small data set does not allow recalculation of original primary outcomes but did allow recalculation of recovery data. Release of the PLOS data is crucial for a better understanding of what went on in that trial. That’s why the investigators are fighting so hard.
Eventually, Coyne began suggesting to PLOS that he would organize public protests and scientific meetings attended by journal representatives.
I think it is the most significant issue in psychotherapy today, in terms of data sharing. It’s a flagrant violation of international standards.
The Retraction Watch article cited a 2015 STAT article that was written by Retraction Watch co-founders Ivan Oransky and Adam Marcus. That article was sympathetic to my request:
If the information Coyne is seeking is harmful and distressing to the staff of the university — and that’s the university’s claim, not ours — that’s only because the information is in fact harmful and distressing. In other words, revealing that you have nothing to hide is much less embarrassing than revealing that you’re hiding something.
The STAT article also said:
To be clear, Coyne’s not asking for sex tapes or pictures of lab workers taking bong hits. He’s asking for raw data so that he can evaluate whether what a group of scientists reported in print is in fact what those data show. It’s called replication, and as Richard Smith, former editor of The BMJ (and a member of our board of directors), put it last week, the refusal goes “against basic scientific principles.” But, unfortunately, stubborn researchers and institutions have used legal roadblocks before to prevent scrutiny of science.
The PLOS One Editors’ blog post.
The Expression of Concern was accompanied by a blog post from PLOS Iratxe Puebla, Managing Editor for PLOS ONE and Joerg Heber, Editor-in-Chief on May 2, 2017
Since we feel we have exhausted the options to make the data available responsibly, and considering the questions that were raised about the validity of the article’s conclusions, we have decided to post an Expression of Concern  to alert readers that the data are not available in line with the journal’s editorial policy. It is our intention to update this notice when a mechanism is established that allows concerns about the article’s analyses to be addressed while protecting patient privacy.
This statement seems to suggest that the ball is in the PACE investigators’ court and that PLOS One editors are prepared to wait. But reading the rest of the blog post, it becomes apparent that PLOS One is wavering on the data sharing policy
Current challenges and opportunities ahead
During our follow up it became clear that there is little consensus of opinion on the sharing of this particular dataset. Experts from the Data Advisory Board whom we consulted expressed different views on the stringency of the journal reaction. Overall they agreed on the need to consider the risk to confidentiality of the trial participants and on the relevance of developing mechanisms for consideration of data requests by an independent body or committee. Interestingly, the ruling of the FOI Tribunal also indicated that the vote did not reflect a consensus among all committee members.
Fact checking the PLOS One’s Editors’ blog and a rebuttal
John Peter fact checked the PLOS One editors’ blog. It came up short on a number of points.
“Interestingly, the ruling of the FOI Tribunal also indicated that the vote did not reflect a consensus among all committee members.”
This line is misleading and reveals either ignorance or misunderstanding of the decision in Matthees.
The Information Tribunal (IT) is not a committee. It is part of the courts system of England and Wales.
…the IT’s decisions may be appealed to a higher court. As QMUL chose not to exercise this right but to opt instead to accept the decision, then clearly it considered there were no grounds for appeal. The decision stands in its entirety and applies without condition or caveat.
The court had two decisions to make:
First, could and should trial data be released and if so what test should apply to determine whether particular data should be made public? Second, when that test is applied to this particular set of data, do they meet that test?
The unanimous decision on the first question was very clear: there is no legal or ethical consideration which prevents release; release is permitted by the consent forms; there is a strong public interest in the release; making data available advances legitimate scientific debate; and the data should be released.
The test set by this unanimous decision was simple: whether data can be anonymized. Furthermore, again unanimously, the Tribunal stated that the test for anonymization is not absolute. It is whether the risk of identification is reasonably likely, not whether it is remote, and whether patients can be identified without prior knowledge, specialist knowledge or equipment, or resort to criminality.
It was on applying this test to the data requested, on whether they could be properly anonymized, that the IT reached a majority decision.
On the principles, on how these decisions should be made, on the test which should be applied and on the nature of that test, the court was unanimous.
It should also be noted that to share data which have not been anonymized would be in breach of the Data Protection Act. QMUL has shared these data with other researchers. QMUL should either report itself to the Information Commissioner’s Office or accept that the data can be anonymized. In which case, the unanimous decision of the IT is very clear: the data should be shared.
PLOS ONE should apply the IT decision and its own regulations and demand the data be shared or the paper retracted.
Data Advisory Board
The Editors’ blog referred to “Experts from the Data Advisory Board.. express[ing] different views on the stringency of the journal reaction.”
That was a source of puzzlement for me. Established procedures make no provision for an advisory board as part of the process or any appeal.
A Google Search clarified. I had been to this page a number of times before and did not remember seeing this statement. There is no date or any indication it was added after the rest of the statement.
PLOS has formed an external board of advisors across many fields of research published in PLOS journals. This board will work with us to develop community standards for data sharing across various fields, provide input and advice on especially complex data-sharing situations submitted to the journals, define data-sharing compliance, and proactively work to refine our policy. If you have any questions or feedback, we welcome you to write to us at firstname.lastname@example.org.
The availability of data from reanalysis and independent probing has lots of stakeholders. Independent investigators, policymakers, and patients all have a stake. I don’t recognize the names on this list and see no indication that consumers affected by what is reported in clinical and health services papers have role in making decisions about the release of data. But one name stands out.
Who is Malcolm Macleod and what is he doing in this decision-making process?
Malcolm Macleod is quoted in the Science Media Centre reaction to the PACEgate special issue:
Prof. Malcolm Macleod, Professor of Neurology and Translational Neuroscience, University of Edinburgh, said:
“The PACE trial, while not perfect, provides far and away the best evidence for the effectiveness of any intervention for chronic fatigue; and certainly is more robust than any of the other research cited. Reading the criticisms, I was struck by how little actual meat there is in them; and wondered where some of the authors came from. In fact, one of them lists as an institution a research centre (Soerabaja Research Center) which only seems to exist as an affiliation on papers he wrote criticising the PACE trial.
“Their main criticisms seem to revolve around the primary outcome was changed halfway through the trial: there are lots of reasons this can happen, some justifiable and others not; the main think is whether it was done without knowledge of the outcomes already accumulated in the trial and before data lock – which is what was done here.
“So I don’t think there is really a story here, apart from a group of authors, some of doubtful provenance, kicking up dust about a study which has a few minor wrinkles (as all do) but still provides information reliable enough to shape practice. If you substitute ‘CFS’ for ‘autism’ and ‘PACE trial’ for ‘vaccination’ you see a familiar pattern…”
The declaration of interest is revealing in what it says and what it does not say.
Prof. MacLeod: “Prof Sharpe used to have an office next to my wife’s; and I sit on the PLoS Data board that considered what to do about one of their other studies.
The declaration fails to reveal a recent publication co-authored by Macleod and Trudy Chalder.
Wu S, Mead G, Macleod M, Chalder T. Model of understanding fatigue after stroke. Stroke. 2015 Mar 1;46(3):893-8.
This press release comes from an organization strongly committed to the protection of the PACE trial from independent scrutiny. The SMC even organized a letter writing campaign headed by Peter White to petition Parliament to exclude universities for Freedom of Information Act requests. Of course, that will effectively block request for data.
Why would the PLOS One editors involved such a person to reconsider what been a decision in favor of releasing the data?
Connect the dots.
Trends will continue toward making uploading data into publicly accessible repositories a requirement for publication. PLOS One has bought into discredited arguments about patient consent forms not allowing sharing of anonymized data. PLOS One is no longer at the vanguard of open science through routine data sharing.