Spinning a negative mindfulness therapy study into a published positive study

positive spin 2Most clinical psychology journals seldom publish honestly reported null findings from randomized trials of psychotherapy. The more these journals aspire to be seen as prestigious, the greater the reluctance to publish such reports. Reading only these journals, you would be hard-pressed to estimate how frequent negative trials actually are in clinical psychology.

A notorious example is Journal of Consulting and Clinical Psychology (JCCP) where the editorial board seems to abhor honestly reported null trials. A  colleague of mine once attempted to publish a honestly reported report of results in a well-designed, well resourced, clinical trial of imagery rehearsal for posttraumatic nightmares among Vietnam War veterans. Following an analysis plan that was publicly available before the first patient was recruited, this study did find not a positive effects on nightmares. When she submitted the manuscript to JCCP, it was rejected with one reviewer commenting “there is nothing to be learned from a negative trial.” That of course is nonsense. She appealed, but to no avail. Fortunately the article was eventually published in Journal of Traumatic Stress.

JCCP will publish results from null randomized trials, but not honestly reported null findings. Examples of reports of RCTs with null findings that were published in JCCP are nonetheless easy to find. Authors succeed in getting them published variously by

  • Emphasizing subgroup analyses rather than the main analyses of the originally identified primary outcomes .
  • Including multiple outcome measures and selecting the ones that achieved the strongest effects and ignoring or suppressing measures with null findings.
  • Constructing new outcome variables when primary and secondary outcome measures do not yield positive findings.

An example of an article in JCCP that emphasize post-hoc subgroup analyses null findings that went on to become cited hundreds of times is

 Dimidjian, S., Hollon, S. D., Dobson, K. S., Schmaling, K. B., Kohlenberg, R. J., Addis, M. E., … & Jacobson, N. S. (2006). Randomized trial of behavioral activation, cognitive therapy, and antidepressant medication in the acute treatment of adults with major depression. Journal of consulting and clinical psychology, 74(4), 658.

Yet another article and JCCP that authors tweaked its null findings into the appearance of being positive and went on to be cited hundreds of times,

 Bach, P., & Hayes, S. C. (2002). The use of acceptance and commitment therapy to prevent the rehospitalization of psychotic patients: a randomized controlled trial. Journal of consulting and clinical psychology, 70(5), 1129.

You can find my critical analysis of it here.

millersdaughter_reumpelstiltskinEvaluations of clinical trials by reviewers and subsequent readers depend heavily on what is said in the abstracts. If you want to spin a null trial into a published positive one, make sure that you take extra care with the abstract to emphasize the subgroup analyses, secondary outcomes, or invented variables over the null findings. If you can get away with it, do not even mention the null findings. And don’t describe the positive ones as post hoc or exploratory.

The foundation for sophisticated spinning is laid in the introduction and involves cherry picking from the literature and selectively citing studies that can be interpreted as predicting the particular outcome you now want to emphasize.

In the results section, is important to make a smooth transition from the reporting of null findings to the enthusiastic reporting of the positive findings.

You can probably even get away with starting this discussion section with a summary of the spun findings. If at all mentioned, you can leave the null findings with the primary outcomes to the limitation section and suggest that further research is needed to replicate them.

Many journals now require preregistration of clinical trials, including designation of the primary outcomes for the main analyses. But do not be intimidated by this requirement, because many journals do not enforce it. But be careful with PLOS One because it prompts Academic Editors and reviewers to examine published preregistration of trials and to make sure that what is reported in the manuscript submitted PLOS One corresponds what was promised in the preregistration.

A forthcoming article about mindfulness-based cognitive therapy (MBC T) to prevent relapse and recurrent depression provides another example of a successfully spun null findings transformed into a publication in JCCP. You can find the paper here.

In fairness to the authors, I should note that the abstract clearly states “allocated treatment had no significant effect on risk of relapse…” before stating “however severity of childhood trauma affected relapse. ..”

The conclusion of the abstract is

 MBC T provided significant protection against relapse for participants with increased vulnerability to history of childhood trauma, but showed no significant advantage in comparison to an active control treatment and usual care over the whole group of patients with recurrent depression.

Fair enough, but I doubt that most readers would classify this as a null trial on the basis of the abstract or the full paper. And I very much doubt that this would have been published in JCCP if the emphasis had not been given to this post-hoc finding for trauma.

How did the authors decide to focus on childhood trauma?

Nowhere is it said that it came from a planned analyses.

The trial was preregistered but there is no mention of childhood trauma in the variables of interest or in the plans for analysis.

The heading in the method section of the article reporting the trial “Assessment of Primary Outcomes” gives no mention.

The section “Statistical Analysis” states

 We then used a forward stepwise approach to derive a parsimonious model predicting risk of relapse from treatment allocated and stratifying and moderator variables including interactions between group and other variables. This model enabled us to examine potential interactions between predictor variables into identifying the strongest predictors amongst a set of variables that was likely to be correlated.

Translation: the authors got themselves a fishing license with no initial idea what was in the pond, but lots of room for exploring and a potentially large set of analyses, the size of which is never disclosed to readers. There is a high likelihood they would find at least something here, but that it will not replicate

The stepwise approach allows the authors to make use of phantom degrees of freedom and report significance levels that are utterly inaccurate, because they do not take into account the number of variables that were examined.

They report

 A childhood trauma by treatment group interaction, as its racial for increase of one standard deviation equals 0.42, 95% CI [0.26, 0.69]

But

 a site by treatment group interaction, has odd ratio equals 1.93 with 95% CI [0.74, 5.0]… This site by treatment group interaction, which emerged only in stepwise modeling, resulted from a greater reduction in risk of relapse in those allocated to MBC T in Oxford than in Bangor.

Translation: we did not predict either effects childhood trauma or whether the treatment was provided in Oxford or Bangor, but we found for both.

Although the effects for being treated in Oxford rather than Bangor seems substantially larger than whether someone had a childhood trauma, only the latter was mentioned in the abstract. Actually, it mattered more whether you are treated in Oxford or Bangor than to what treatment condition you were assigned!

The article closes with the take away message

In short, our findings add to the growing body of evidence that psychological interventions, delivered during remission, may have particular beneficial effects in preventing future episodes of depression, but maybe especially relevant for those at highest risk of relapse. The pattern of data that emerges from all existing trials suggest future research should now give particular attention to characterizing indices of vulnerability so to take the treatment characteristics that might address them can be clearly identified.

Nowhere does it say that the authors could not find an overall beneficial effect of mindfulness cognitive therapy on likelihood of a future episode of depression.

Nowhere does the does discussion caution that the effect for trauma emerged only in poststraw into gold hoc analyses with variables that were identified by stepwise regression equations capitalizing on chance. Nowhere does it say that if you want to reduce your likelihood of a recurrence of depression go to Oxford, not Bangor for treatment There is high likelihood that these post-hoc findings will not replicate. Indeed, the trial had not even been powered to find it. But read again the concluding paragraph to see how wonderfully this is spun.

Special thanks to Ioana Cristea for spotting the spin.