The futility of debating bad science in letters to the editor
My post at PLOS Mind the Brain summarizes criticisms of a meta-analysis of psychological interventions for depressive symptoms in cancer patients that was organized by the Society of Behavioral Medicine. The authors systematically searched the literature, but found too few studies to justify their sweeping conclusion: that strength of evidence for psychotherapy for depressive symptoms among cancer patients warrants widespread dissemination of existing treatments and implementation routine care.
Like a lot of professionals concerned about delivering effective psychosocial care to cancer patients, I wish that the evidence was that strong. But Ilook to the available evidence to see if my wish is justified. It is not. Finding that the available evidence is not extensive or strong can justify funding more research. Prematurely claiming that we have sufficient evidence can lead to these scarce research resources being shifted elsewhere and questions not getting answered.
I previously provided my concerns to the authors in a Letter to the Editor published in JNCI and the authors responded. It was not that long ago, but since then I have become convinced of the futility of letters to the editor as a form of post-publication peer review. The problems are that
- Journals place such stringent limitations on the number of words in a letter that at best only one or two criticisms can be delivered effectively.
- Whereas letters receive often stringent review, responses from authors are not necessarily peer-reviewed by anyone, as long as libel is avoided.
- The authors can write whatever they want, knowing that no further response from critics is allowed.
- A letter to the editor is usually published months after an article has appeared, even when they are submitted after the article was published. Readers are unlikely to bother to look up the original article or even to recall it.
- Readers newly accessing the article that has been critiqued may not even become aware that there has been a letter to the editor offering decisive criticism of what is claimed in the target article.
Restrictions on letters to the editor remain, but have become obsolete with the development of journal webpages and the possibility of rapid response without cutting into page allocations for the journal. Some restrictions on letters are simply a matter of inertia. Editors of traditional journals published in paper additions generally did not like letters to the editor because they consumed scarce page allocations and were unlikely to generate the citations that original article would. That is no longer a problem. A main source of resistance to letters of the editor remains, in my experience, that editors are quite defensive and loathe to accept letters that reflected badly on their decisions. And they consider publishing such letters disloyal to reviewers if the reviewers are made to look bad for overlooking obvious flaws.
Bottom line is that I no longer consider letters to the editor very effective contributions to post publication peer review. I once promoted them as a way of teaching critical appraisal skills, but regret it. Fortunately we now have personal and lab blogs and PubMed Commons and group blogs for junior persons like Mental Elf.
Anyway, in my letter to the editor, I criticized the authors’ misclassification of collaborative care interventions for depression as psychotherapy. Collaborative care interventions are designed to improve the overall quality of care for depression being provided, including the availability medication and the quality with which it is prescribed, monitored, and followed up. They involve reorganizing whole systems of care.
Lots more than psychotherapy is being provided in a collaborative care intervention and not all patients in the intervention arm get psychotherapy. And just who gets therapy within the collaborative care intervention is not determined by further random assignment.
Many of the patients assigned to the intervention arm receive antidepressant medication, either in conjunction with therapy or alone. Furthermore, many patients assigned to the control group in collaborative care intervention studies have to pay for any treatment and may have other difficulties accessing quality care besides not being able to pay for it. Comparisons between collaborative care interventions and control groups thus do not produce meaningful estimates of the effect sizes for psychotherapy. Among the 60 or so collaborative care studies and probably 30 or more systematic reviews and meta-analyses, I have never seen collaborative care considered to be psychotherapy for the purpose of calculating effect sizes.
The authors replied to my criticism:
First, we contend that including collaborative care RCTs …was well reasoned. Our goal (p. 991) was to examine the efficacy of RCTs testing various therapeutic approaches rather than specific psychotherapies. Collaborative care (CC) interventions are well suited for primary care (1) and are gaining traction in oncology (2). Secondary processes in CC, such as education about depression, are common components of psychotherapy (3). In the three CC trials, patients were randomly assigned to CC or usual care. We emphasized (p. 1000) that patients do not invariably receive psychotherapy in a CC model but rather can receive psychotherapy, medication, or both. Most CC patients received psychotherapy, with or without medication. Having treatment options better represents the naturalistic context and fosters successful dissemination to practice.
This reply brings to mind an expression I picked up from an old college housemate: you’re shuckin’ me, man.
Surely the authors know this is nonsense and would not say it in other contexts. Let’s ignore that the three collaborative care interventions that they included were not sustained in the clinical settings after conclusion of their demonstration research projects. That combinations of psychotherapy and medication occur in the natural environment are irrelevant to evaluations of whether psychotherapy is efficacious. That requires randomized trials in which the availability of psychotherapy is the key controlled variable. It is not in a collaborative care RCT.
I further criticized the inclusion of studies with sample sizes so small that it was statistically improbable that they would get a positive effect even if they were evaluating a potent active treatment.
Allowing RCTs with relatively small sample sizes, which coincides with our inclusion of pharmacologic studies and the well-documented knowledge of substantial attrition in pharmacologic RCTs for major depressive disorder (6), reflects a decision about which active debate exists in the meta-analytic literature (7). Our use of Hedges’ g, which corrects for small sample bias, and findings from our elected safeguards of examining publication bias, the fail-safe N, and whether the psychotherapeutic RCT effects varied as a function of trial attrition all suggest a stable overall effect size.
ineffective correction. As I noted in my longer blog post, meta-analysis experts reject the validity of fail-safe N. It is explicitly discouraged by training materials for the Cochrane Collaboration which is the first place that many researchers, clinicians, and policymakers go to find authoritative meta-analyses.
Finally, I challenged the authors’ including two separate effect sizes from the same study, violating the requirement for statistical independence.
The authors replied
As we stated (p. 992), because interventions were distinct, we calculated two separate effect sizes for trials containing two intervention groups, which violates the assumption of independent effect sizes. We conducted sensitivity analyses to address this issue; separate analyses including only the largest or the smallest effect size from those studies did not substantially influence the findings (p. 999).
Analyses with such small numbers of small studies do not correct a violation of the basic assumptions of the statistics of meta-analyses. Furthermore, if the authors are going to include two effect sizes from the same study, why not include three? As I noted in my longer blog posts, one of the conditions counted as an intervention group was supportive therapy, normally considered a control group in psychotherapy research. If it had been considered a control/comparison for CBT, the effect size CBT would have been negative because patients receiving supportive therapy actually had better outcomes. I would not want to make too much of this finding because of the trial being underpowered, but certainly once again, the authors are simply wrong and counting two, not one effect sizes from the three that were available. Their selection of the two serves their bias in seeking evidence consistent with a strong effect for psychotherapy.
I get frustrated again just reviewing the exchange that was precipitated by my letter to the editor. I think I should return to it the next time I get an overwhelming urge to write a letter to the editor, rather than just blog or post something at PubMed Commons.
Importantly, PubMed Commons allows ongoing, continuous peer review for the life of an article. Letters to the editor allow only a duel with authors guaranteed the last shot.
But forget two-shot duels, participate yourself in postpublication peer review by expressing your opinion about this article at PubMed Commons. I have, come see.
And for links to the actual RCTs discussed here, go here.