A guest blog post from Christopher Ferguson, Stetson University with a commentary from Coyne of the Realm
Evaluating the trustworthiness of psychological research cannot depend on independent replication. Many studies that claim clinical and public policy relevance are too large, required too many resources, or have too long a follow-up for replication to be feasible. Improving the trustworthiness of published psychological research depends heavily on the routine availability of data sets for independent scrutiny. Many journals, including those of the American Psychological Association require that authors make data available upon request. But as this invited blog post demonstrates, enforcement of data sharing is spotty. Those unwise enough to request data invite a lot of frustration.
The situation will only improve if researchers running into difficulty obtaining data publicize their frustrations, so that opinion can be mobilized to correct a pervasive and unfortunate situation.
- The story starts with a request to the Flourishing Families group for a small amount of data that would allow re-examination of their analyses and conclusions concerning effects of media violence.
- The request made use of a little-known requirement for publication in APA journals that authors make their data available to other competent investigators to verify their substantive conclusions.
- The response from Flourishing Families was an invoice for $450 and a contract that would restrict any analyses of the data to those narrowly required to verify the exact same analyses, but not to be able to explore the robustness of the paper’s claims through alternateanalyses.
- The APA ethics committee acknowledged that restricting analyses was probably not permitted, but failed to provide clarity or guidance on other issues, such as the invoice or requiring Flourishing Families’ permission to publish any reanalysis.
- Under the terms of the contract, publication of any analyses beyond those that were stipulated, even correcting computational errors in the original article, could be subject to ethical sanctions.
- The $450 was raised in a GoFundMe campaign but the restrictive contract remained.
- Ultimately, current APA ethics policy appears to fail to prevent substantial roadblocks to data sharing.
Media violence research was once the darling of psychological science. Back in 2000, the American Academy of Pediatrics claimed there were 3500 studies of media violence with only 18 not finding effects on youth aggression. Since then, confidence in media violence research has crumbled, with numerous null studies, failed replications and controversies about the poor methods and possible questionable researcher practices used in many studies. The AAP claim about 3500 studies proved to be apocryphal with the number pulled, not from a scientific source, but from a pop psychology book. Yet the debate goes on, with new studies adding fuel to the fire.
In early 2016, one longitudinal study, authored by Dr. Sarah Coyne at Brigham Young University, argued that watching violence or relational aggression on TV would lead to the same behaviors in youth (henceforth, S.Coyne, 2016).
Coyne SM. Effects of viewing relational aggression on television on aggressive behavior in adolescents: A three-year longitudinal study. Developmental Psychology. 2016 Feb;52(2):284-95.
The evidence for such claims from the study appeared weak. The authors used a path analysis/SEM approach. Effect sizes were near zero (standardized coefficients between .02 and.06) and, in the case of physical aggression, dependent on a relaxed (p < .10) criterion for statistical significance. The data came from a larger Flourishing Families dataset, which included many variables not used in S.Coyne (2016).
Eventually I got thinking…maybe these results aren’t robust. SEM based approaches to data can be squishy. And the authors hadn’t included many potential control variables (indeed, relationships tested were basically bivariate.) So I wrote to the Flourishing Families group in July, 2016 to request the S.Coyne (2016) data. Authors publishing in APA journals sign documents acknowledging they’ll provide data on request for verification of substantive claims. Thus, a data request seemed straightforward. But it would begin a strange journey into the land of APA ethics policy that is heavy on loopholes and low on clear guidance.
Since it’s at the crux of the matter, I’ll post the relevant APA ethics policy here (8.14):
After research results are published, psychologists do not withhold the data on which their conclusions are based from other competent professionals who seek to verify the substantive claims through reanalysis and who intend to use such data only for that purpose, provided that the confidentiality of the participants can be protected and unless legal rights concerning proprietary data preclude their release. This does not preclude psychologists from requiring that such individuals or groups be responsible for costs associated with the provision of such information. Psychologists who request data from other psychologists to verify the substantive claims through reanalysis may use shared data only for the declared purpose. Requesting psychologists obtain prior written agreement for all other uses of the data.
My request eventually made its way to Dr. Laura Padilla-Walker who, though not an author on S.Coyne (2016), served as contact person for Flourishing Families. Since other data, including some potential control variables on family environment from Flourishing Families had been published in another APA journal, I originally requested both datasets to see if the substantive claims of the S.Coyne (2016) would be robust once control variables were included. Dr. Padilla-Walker objected to this request, noting that this would constitute new analyses rather than merely fact-checking the original paper. Fair enough, I figured, what about just the data from S.Coyne (2016)? That dataset seemed pretty small…just four variables (TV relational and physical aggression, real-life relational and physical aggression) across 3 time periods, plus presumably age and gender. Maybe 14-15 variables.
In response to this request I was sent an invoice for $450 dollars and a contract.
The wording of the contract appeared to restrict me to only conducting the exact analyses (even if they were wrong) that S.Coyne (2016) had done, and requiring permission of the Flourishing Families group before publishing any findings. The contract explicitly threatened the filing of ethics charges were I to deviate from this.
Regarding the $450 charge, APA ethics policy allows for “costs associated with the provision of such information.” But what a reasonable cost is isn’t clarified. Generally, I’d assume this would be anything material…such as if I requested the data on a USB drive. Or if I made some unusual and time-consuming requests of the data (combining multiple unrelated datasets into one file, say.) But I’d imagine having a data file and code book ready is simply expected of scholars, not a recoupable cost. And I’d guess transferring roughly 15 variables from one file to another should take a graduate student ten minutes. But I was charged $300/hour for 1.5 hours work. Clearly, I’m in the wrong business (my salary as a full professor is an embarrassing fraction of $300/hour.) I called the APA Ethics office about this and the investigator I spoke to confirmed the APA had no particular guidelines on what constituted a reasonable charge. Brigham Young University’s Office of Research and Creative Activities likewise confirmed they had no policy against such a financial proviso.
Stipulation of fees can represent a non-trivial barrier to data requests. Being at a small, liberal arts school and without grant funding, I imagine many scholars like myself would find (arguably spurious) financial invoices a burdensome block to data requests. I came up with a creative solution: I launched a GoFundMe campaign! Within a short time, generous donations poured in and I was able to fund the invoice. But the problem of the contract remained.
Regarding the contract itself, the stipulation about only being able to reproduce the *exact* analyses of the original S.Coyne (2016) article seemed to prevent any substantive reanalysis that could verify or question the claims of the original article. What if different SEM models produced different results, or using regression raised questions for the conclusion of the article? Might Bayesian analyses reach a different conclusion? Or what if a variable had simply been miscalculated? As I read the language of the contract, I understood it to mean I couldn’t even fix a miscalculated variable if there was one (I was not corrected by Flourishing Families when I pointed this out to them). Is this level of specificity allowed under APA policy?
I reached out to the APA ethics committee in writing. While I waited for a reply, I sent Flourishing Families, a revised contract, suggesting some loosening of the restrictions on what I could do to verify their substantive claims. Ultimately after having sought input from the APA Journals Office and General Council, Flourishing Families wrote back with a new contract of their own. Dr. Padilla Walker informed me that the APA General Counsel had helped her with some of the specific wording. Some of the language regarding requiring permission from Flourishing Families to publish any results appeared to have been softened, but the stipulation regarding exact analyses had not been. Additionally, the new wording added a new level of negativity with a threat to call my Dean if I did not comply!
After a month or so, the APA Ethics Committee wrote back to confirm that it is not, in fact, permissible to include analysis restrictions on data requests for verification:
In their discussion the Committee focused on the key terms bolded in the above citation. Namely that data from published research should be made available for purposes of ‘verify[ing] substantive claims through reanalysis…” The Committee noted that there is a difference between a reanalysis and a replication study. Thus, the Committee feels that Standard 8.14(a) promotes the sharing of data for reanalysis, whether this be a straight replication of the published analysis or not. However this does not necessarily require the release of variables/data that were not included in the original published study.
However, as to the issue of whether the contract could include a demand that I obtain Flourishing Family’s permission to publish the results of a reanalysis (as in the original contract), the APA Ethics Committee whiffed, suggesting I contact the Journals Office or open science groups like the Open Science Framework.
When I contacted Dr. Walker at Flourishing Families with the information from the ethics committee she reminded me she’d gotten endorsement for her revised contract from the APA Journals Office and General Council. She forwarded an email to her from Dr. Rosemarie Sokol-Chang at the Journals office endorsing Flourishing Family’s revised contract and even suggesting some of the wording. I reached out to Dr. Sokol-Chang about this and provided the letter from the APA Ethics Office. As of this writing, she promised to confer with APA General Council and get back to me.
In the meantime, as the APA Ethics Committee suggested, I reached out to Dr. Brian Nosek at the Center for Open Science. Dr. Nosek stated to me that the stipulations of the Flourishing Families contract were not consistent with transparency and openness in science.
And there things grind, nearly five months after my original request. I am being charged $450 for a tiny dataset I can’t do anything with. This would be a bad deal, even ignoring the ethics issues. And, aside from possibly the proviso regarding whether I can be restricted only to exact analyses from the original study, there appears to be nothing in APA ethics policy to prevent original study authors from throwing up multiple financial and contractual roadblocks to a data request vetting their work.
As the charming conclusion of the Flourishing Families contract notes, were I to find evidence of a mistake in S.Coyne (2016) and decided the scientific community needed to know about it, it could be me who faced an ethics complaint.
Most eye-opening to me is the inherent vagueness of APA ethics policy which does little to prevent shenanigans from authors seeking to protect their policy from independent and open verification. APA policy remains vague, apparently purposefully, on key issues. Perhaps this is due to an inherent conflict of interest…arguably it’s not to the advantage of APA journals to have their articles fact-checked by independent scholars…fact-checking that could reveal errors and perhaps lead to retractions.
The funny thing about S.Coyne (2016) is in considering, really, what’s the worst that could come from my independent verification? Let’s say I disagreed with the Flourishing Families analyses and came to a conclusion of null effects of viewing violence on TV on adolescent behavior. Assuming I ever got my reanalysis published, they’d write a rebuttal…we all get pubs and, given how intractable the media violence debate is, the opinions of scholars in the field would likely change barely at all. I’ve yet to see a career shattered by such an exchange.
Instead we’ve exposed the limits of APA ethics policy. Policy which, at present, appears – whether intentionally or not – to discourage data sharing with independent scholars who are neither associated with the original authors or the APA itself. Until the APA adopts policies that truly promote open science, articles such as S.Coyne 2016 should be considered apocryphal, the evidentiary value they provide for science limited.
Chris Ferguson is a professor of psychology at Stetson University. He studies various media influences, including video game violence, thin-ideal media and body dissatisfaction and “sexy” media. He is a fellow of the American Psychological Association, and received an early career scientist award through APA’s Division 46 (Media Psychology and Technology). Aside from this professional work, he also writes speculative fiction which can be found at his website ChristopherJFerguson.com. He lives in Orlando with his wife and son.