Spokesperson clarifies that American Psychological Association does not endorse routine data sharing

I provide an update of the recent blog post about a researcher requesting a small set of variables to verify claims appearing in an article published in an APA journal. My update involves a further exchange of emails between this researcher and the APA publications office.

In the new exchange:

  1.  APA verifies that it has a weak policy concerning authors having to share data and it does not really endorse routine sharing of data.
  2.  APA indicates that any re-analyses of data requested from an article published in APA Journal have to stick strictly to reproducing analyses in the published article, and not go further without the express permission of the original authors.
  3.  Basically, analysis of data provided in response to a request can only be used to check specific statistical calculations and not to conduct further analyses that might shed light on the appropriateness of the original author’s conclusions.

That is usually not where any clarification of controversies will be found.

I present further evidence of contradiction and even hypocrisy in what APA says about data sharing.

I make three nominations to a Wall of Shame of those who resist correction of potential untrustworthiness in the psychological literature by penalizing and threatening those who request data for checking by reanalysis.

reusing-data-makes-me-research-parasite In an editorial entitled Data sharing,  the editors of New England Journal of Medicine have condemned as “research parasites” researchers who seek to test alternatives to authors’ hypotheses or new exploratory hypotheses with data shared from published articles. The APA seems to have much the same attitude.

Improving the trustworthiness of psychology depends crucially on the work of such research parasites.

Why? For many areas of psychology the scope and expense of research projects make replication initiatives impractical. Efforts to improve the trustworthiness of psychology involves insisting on completeness and transparency in what is published, but also on routine data sharing from published papers.  Well aware of the limitations of review of manuscripts before they are published, we need independent, post-publication review with access to the data. APA’s position is making any scrutiny more difficult.

This  incident again demonstrates the institutional resistance to data sharing and the institutional support available to authors who want to protect the claims from independent scrutiny.

I wonder what these authors were hiding by presenting obstacles to accessing their article. We won’t get to see.

After all, Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results.

 What was reported so far

  1.  In the recent guest blog post, Professor Christopher Ferguson of Stetson University described the extraordinary resistance he experienced when he attempted to obtain a few variables from an author to decide for himself if her claims held up about the effects of exposure to media violence on youth violence.
  2.  To summarize:
  3.  Chris first asked for data sets from two articles which were needed to check the claims in one of the articles.
  4. The spokesperson for the Flourishing Families Project, from where the data came, objected. She considered what Chris was proposing to be new analyses rather than merely fact-checking the original paper.
  5.  Chris revised his request to a more modest set of variables across 3 time periods, plus some covariate/control variables.  Maybe  a total of14-15 variables.
  6.  In response, he received an invoice for $450 and a contract that he was required to sign to obtain the data. The contract stipulated that Chris could only reproduce exact analyses presented in the article and would have to obtain permission of the Flourishing Families Project to publish any results. The contract carried the penalty of ethical charges if these terms were not met.
  7.  Chris launched a GoFundMe drive and raised the $450, but also contacted APA Ethics Committee.
  8.  While Chris was waiting for a response from APA, he received a new contract from the spokesperson for the Flourishing Families Project. The revised contract seemed to relax requirements that Chris get permission from the project to publish results. But the contract retained the restriction that he could only reproduce analyses contained in the article. It introduced a threat of a letter to Chris’s Dean if he did not accept this restriction.
  9.  Chris then received a letter from the APA Ethics Committee explaining that it is not appropriate for authors to restrict independent re-analyses needed to confirm their conclusions.
  10. A victory? No, not really. The spokesperson for the Flourishing Families Project then wrote to Chris, stating that her previous letter had been written in consultation with the APA Journals Office and General Counsel. Wow, brought in the lawyers. That should scare Chris.

The big picture issues

An earlier post at Stat How researchers lock up their study data with sharing fees had nailed the key issues in the author’s response to Chris:

The story highlights a potentially uncomfortable aspect of data-sharing: Although science is unquestionably a public good, data often are proprietary. Drug companies, for example, spend millions upon millions of dollars on clinical trials for their experimental products. While the public certainly has a right to know the results of those trials — favorable or not — researchers who want access to the data to conduct their own studies can’t reasonably expect the original investigators not to recoup the costs of sharing it.

Okay, but:

 Trying to recoup costs is fine in the abstract, but if it’s used as just another way to avoid sharing data, then it’s deeply objectionable.

And:

One thing about the Ferguson-BYU example is clear: We need explicit policies. Winging it will work about as well in this arena as it does in, say, presidential debates. Existing rules about data sharing, if they even exist, are vague and institution-specific, and permit researchers to erect obstacles, financial or otherwise, that give them effective veto power over the use of their data.

Now, a further exchange between Chris and APA

The next chapter in this story started with an email from Rosemarie Sokol-Chang, who identifies herself as Publisher, APA Journals (Acting). The full email is reproduced below, but I want to highlight:

“Certainly there is a group of scholars and organizations that advocate for open sharing of data – free and without restriction. While these movements exist, the extent to which to share data is up to the author, and in this case, the author chose not to freely share it.”

I guess we can count APA out of a “group of scholars and organizations…”

 The actual emails

 Subject: APA Journals – data share request

Dear Chris,

Jesse and I met to discuss your request to reuse data, in light of the letter you received from the Ethics Committee. From our read of the letter, we (Journals and General Counsel) are not interpreting the code differently from the Ethics Committee – that is, data should be released to those who “seek to verify the substantive claims through reanalysis and who intend to use such data only for that purpose…”

We understand the claim you are making regarding data sharing, but that extends beyond the ethical code APA set and follows. Certainly there is a group of scholars and organizations that advocate for open sharing of data – free and without restriction. While these movements exist, the extent to which to share data is up to the author, and in this case, the author chose not to freely share it. Per Standard 8.14:

“Psychologists who request data from other psychologists to verify the substantive claims through reanalysis may use shared data only for the declared purpose. Requesting psychologists obtain prior written agreement for all other uses of the data.”

I understand this is not the outcome that you want, but the author is complying with the current APA Ethics Code and the APA Journals policy of sharing data for verification.

All best,

Rose

Rosemarie Sokol-Chang, PhD
Publisher, APA Journals (Acting)
American Psychological Association
750 First Street NE
Washington, DC 20002

Decide for yourself, but I think this email indicates that APA is contradicting earlier communication, but denying it is doing so.

Chris replied:

From: Chris Ferguson (Psychology Professor) [mailto:cjfergus@stetson.edu]
Sent: Wednesday, November 30, 2016 6:01 PM
To: Raben, Jesse <jraben@apa.org>; Sokol-Chang, Rose <RSokol-Chang@apa.org>
Subject: Re: APA Journals – data share request

hi Jesse (and Rose):

Thanx for being willing to continue to dialogue with me on this.  Unfortunately I don’t think we’re at “apples to apples” yet.  To be clear (I thought I had been, but if not then I apologize) my intent is, consistent with 8.14, to verify the substantive claims of the original paper not to do “anything I want”…I don’t think I *could* do anything else with this dataset as it has so few variables in it.

However, my interpretation of both 8.14 as well as the letter I received from the Ethics Committee is that, while I must test the same hypotheses as the original paper, I am not restricted to the original analyses.  The Ethics Committee letter appears rather clear on this in fact as they say “Thus, the Committee feels that Standard 8.14(a) promotes the sharing of data for reanalysis, whether this be a straight replication of the published analysis or not.”  This makes a great deal of sense because, of course, the initial analyses may be wrong…and it makes no sense for a verification effort not to check this.

Let me ask you a few pointed questions.

1.) If I were to discover that a variable had been miscalculated in the original dataset, would I be able to recalculate it and rerun the analyses with the corrected variable?

2.) If I discovered that the original analyses were misspecified…would I be able to rerun a corrected analyses with proper specifications?

3.) If I were to learn that the analytic approach itself were inappropriate for the data, would I be able to test the substantive claims using alternate, more appropriate analyses.

4.) If I discovered other, unforeseen, errors in the data or analyses, would I be able to report these?

As I read the current contract, the answer to each of these questions would be “no.”  If I am incorrect in my interpretation (and I may be) I think it would be important for the contract to make clear what I *can* do, particularly in light of the unpleasant language in it threatening ethical complaints and calls to my dean.

Thank you for your consideration.

Chris

And then Rosemarie Sokol-Chang responded:

Hi Chris,

The intent of the 8.14 as we apply it to authors is to offer a check of the validity of what was reported. If you were to receive the data, and run the same analyses, and get different results – we would want the scientific literature cleaned up in light, so that the article didn’t persist with inaccurate results. If you were to find that the data looked “fishy” – which happens rarely but there are some big-name cases of numerous retractions by the same author – this is something APA Journals would also want to know to be able to take measures to clean up the record. This is the “verification” step.

Replication is duplicating an entire experiment – you’d be collecting new data following the same method. Reanalysis is using the same data set – and whether or not a requestor can use a data set to run any particular analyses not reported in the manuscript is ultimately up to the author.

All best,

Rose

Rosemarie Sokol-Chang, PhD

Publisher, APA Journals (Acting)

American Psychological Association

750 First Street NE

Washington, DC 20002

rchang@apa.org202-336-5667

www.apa.org/pubs/journals

Déjà vu all over again

 APA has been here before. See:

The APA and Open Data: one step forward, two steps back?

And what APA really meant in:

Access to Archives of Scientific Psychology Data

One of my earliest blog posts ever was about a study from this department. Did a Study Really Show that Abstinence Before Marriage Makes for Better Sex Afterwards? Requesting data was not an option  in 2011. I don’t think  getting a look at the data was needed to establish the patent absurdity of this study’s methodology and conclusions.

 Three Nominations for the Routine Data Sharing Wall of Shame

use the wall of shame.jpg
sarah-coyne

Sarah Coyne blocked efforts to independently verify her claims about effects of media violence on children.

laura

Laura Padilla-Walker got APA General Counsel involved and raise threat of going to Chris’s Dean.

rosemaire-soklol-chang

Rosemarie Sokol-Chang withdrew what had been apparent APA support for Chris’s request and clarified that APA does not support routine data sharing.

Postscript

Is it really wrong to shame researchers who are not willing to share data or do so with unnecessary obligations?

If you can earn badges for uploading new data, why shouldn’t we also give badges for sharing old data? – Uli Schimmack on FB. November 29, 2016