Reply to an author complaining about my critique of a RCT of CBT for an unrecognized mental disorder

My recent blog post at Mind the Brain, Danish RCT of cognitive behavior therapy for whatever ails your physician about you http://blogs.plos.org/mindthebrain/2016/12/07/danish-rct-of-cognitive-behavior-therapy-for-whatever-ails-your-physician-about-you/ seems well received, except that it drew a harsh response from an author of the RCT that I criticized.

He started with:

I am surprised by this blog – its timing, its false claims, and its author’s attitude towards me and my work.

In the scientific world, I live in, scientific studies are discussed within the scientific community in the scientific literature, not on random blogs. Why is this critique not put forward as a letter to the Editor? And why comes it now, years after the publication of our trial?

At Mind the Brain, I provided a detailed response. I encourage readers to go there and see both the full text of his complaint about my post and my response.

But comment threads tend to get ignored. So I’m reposting my reply to him here, with a lead-in.

The author is correct that I did not see much value to his administering what he called cognitive behavioral therapy to a diverse group of patients who mostly had diagnosed physical conditions. The patients had little in common with other except that their primary care physicians were frustrated with them and had referred them to the author’s clinic.

it-is-thought

This patient education material is pseudoscientific nonsense.

I really didn’t see the treatment as cognitive behavior therapy as I knew it. For a start, the treatment was much less a matter of working collaboratively with patients, than indoctrinating them with a simplistic, non-evidence-based model that much of their pain and distress was simply in their heads.

I also found fault with the weak experimental design in which this treatment was evaluated. For reasons I outline, the design would even have produced evidence that an unproven placebo complementary treatment “works.”

But there is a larger issue here that the author raises: why didn’t I simply write a letter to the editor, taking advantage of well established procedures for peer reviewing criticism of published work?

My answer pointed to what I believe is the untrustworthiness of what we find in the medical and psychological literature and the resistance of existing editorial processes to any self-correction.

I think our exchange conveyed a fundamental disagreement about whether the current system of editor-dominated peer review is working. From my perspective, the author’s clinical trial is quite deserving of criticism. Yet, I was pessimistic about the effectiveness of a letter to the editor, and and even whether the author would be able to veto its publication. The author would get the last word, and authors’ respond to criticism are often not peer reviewed. Anyway, who reads letters to the editor?

But a more serious problem is that with issues like the treatment of poorly accepted and defined diagnoses, a tight group of proponents accept each others’ papers that do not meet the standards of the larger scientific and professional community. When papers are accepted, these cliques vigorously defend against criticism and attack critics. An author group that has overlap with the that of the paper I criticized in this blog has been quite thuggish in its response to critics and to legitimate requests for its data.

The idealized view of editorial constraints that the author invokes in his criticism of me is naïve at best. I think it’s so important that we have blogs and other forms of post publication peer review to counter the control currently exercised by editors and cliques. These corrective mechanisms are not going away. We can expect them to be strengthened in the future and to have greater influence. That is, if we’re going to achieve a more trustworthy literature.

My reply to the author

Thank you for taking time out of your busy schedule to comment on my blog post. I hope this is just the beginning of a fruitful dialogue.

I’m a bit disappointed that you did not respond to the most central issue I raised about your trial design. Namely, you incorporated the same design features into your trial that were shown to be unable to distinguish between an evidence-based treatment of asthma with albuterol and treatment with sham acupuncture. The NEJM trial to which I refer in the blog showed that an objective measurement, in contrast, readily distinguished between albuterol and both the inhaler filled with an inert substance and the sham acupuncture. It’s unfortunate that there are no objective measurements in your study to make any such comparison. However, you did measure number of visits to primary care and mental health and found no differences between the patients assigned to your intensive active treatment and those assigned only two remaining routine care. This is troubling from a health services research point of view. Your treatment is quite expensive and yet led to no reduction in less intensive and less expensive services.

The choice of patients who remained in routine care as a comparison/control group for the active treatment is a poor one. There is a substantial imbalance in terms of the frequency and intensity of contact (especially with the over 30 hours with a psychiatrist provided to the intervention group), peer support, and encouragement of positive expectations. None of these features were present in your control comparison. Such an imbalance disallows claiming that the presumed active ingredients of your intervention were the source of any differences in outcomes.

I’m sure you have ready responses to these issues and I’ll certainly give you a forum for providing them. But it would seem this is an inadequate way to show your treatment works, as you claim.

I don’t know if I can satisfy the objections you raised to the blog, but I will attempt to do so.

First, my carefully documented, long read post was certainly not random. It had origins in a recent dialogue with a Danish journalist who alerted me to your study. I don’t know why he contacting me. When I examined your study, I recognized a number of issues relevant to ongoing discussions of flawed research at this heavily visited blog site. This study is a quite an interesting example from which readers can learn a lot.

You ask why I did not submit my critique as a letter to the editor. Welcome to the era of post-publication peer review. Its rapid development was prompted by the recognized limitations of existing journal-controlled peer review, including letters to the editor. It’s not clear that a letter to the editor would be accepted. It’s not clear that you wouldn’t have veto power over whether it got published. You would have the final word and I would have no opportunity to reply to your response. In contrast, we have the opportunity for dialogue here. I hope we can take advantage of it.

But I have specific additional reservations about attempting to engage you and your co-authors in any letters to the editor. One of your co-authors is an investigator in a highly similar study, which also switched outcomes after data were available. After a drawn out legal battle in which your co-author’s group spent nearly 250,000 British pounds, independent investigators were able to re-analyze the data with the original scoring of the recovery criteria (the SF36, which you also altered). There was a substantial drop in group differences, arguably the claim of effectively to the treatment was effectively eliminated. However the journal refused to consider any letter to the editor pointing out the discrepancy between original and switched recovery criteria. The editor’s response to a submission:

“Obviously the best way of addressing the truth or otherwise of the findings is to attempt to replicate them. I would therefore like to encourage you to initiate an attempted replication of the study. This would be the best way for you to contribute to the debate…Should you do this, then Psychological Medicine will be most interested in the findings either positive or negative.”

I think you would agree that the existing system of editor-controlled review is not working in this instance. You can read an extended account of this particular situation with your co-author here.

Note that your list of authors includes one who among the authors on another that group who have attempted to block criticism of their study, even resorting to lodging complaints with the universities of critics. You can see my documentation of their vilification one academic here and there is more. Personally, I unsuccessfully attempted to engage the author group in a live public debate. I then requested data for reanalysis that they had promised would be available as a condition for publishing in a particular journal. Not only did they refuse, but they denounced me as being vexatious for making the request. Over a year later, I am still waiting for the data. Something is rotten, not only in the State of Denmark, but elsewhere.

So I hope you understand my taking to the blog.

Although I have read your article a number of times, I fail to see where you have are responded to my criticisms. I would welcome you points out exactly where these criticisms have already been addressed.

In your reply, you refer readers to an “evaluated by independent researchers” in a Cochrane systematic review. However, the review is hardly a glowing endorsement. The review finds low to moderate study quality and modest effects of interventions. Almost all comparisons involving CBT to a poorly matched comparison/control group like yours, which stacked the deck. The one exception allowed comparison to an active treatment, there were no advantages for CBT.
Furthermore, there are serious problems in the independence of Cochrane from one of your co-authors. See if you agree with my analysis of “Probing an untrustworthy Cochrane review of exercise for “chronic fatigue syndrome”
I am also engaged in an ongoing debate with David Tovey, Cochrane Editor in Chief and Deputy Chief Director concerning Cochrane’s acceptance of switched outcomes, such as occurred with your trial, without consideration of the risk of bias. You can find his blog post here and my response here. Incidentally, he seems to have embraced blog post as an appropriate way of discussing these issues. I think the same issue of switched outcomes as a risk of bias is true of any meta-analysis involving your study. I think there are also issues about the idiosyncratic nature of your diagnostic criteria that would unlikely not be mentioned in a meta analysis.

You complain about me: “We try to help patients, based on the best evidence available. I do not know who James Coyle [sic] is trying to help with his blog.’

If you review my other blog posts about many other topics, I believe I have a consistent record of demanding high quality evidence be used to evaluate what is best for patients. For the many reasons that I noted in the blog, I believe that your trial falls short.

I agree that the issues raised in my blog post might serve to encourage patients to get second opinions before accepting treatment from your clinic, if not decline referral altogether. Declining a referral would definitely be the advice I would give to a family member, friend or colleague.

I’m unimpressed that your multimodal intervention has much similarity to CBT as it is practiced in the United States and elsewhere. I also strongly object to the strong initial indoctrination of patients with an outdated psychosomatic model . The only pathways you highlighted as causally from the mind to the rest of the body. I find your cartoon representations of this model simplistic and lacking evidence. It seems to display a condescending attitude towards patients and the legitimacy of their suffering and complaints.

You criticize me for claiming that you are proposing bodily distress syndrome as a psychiatric disorder. How else could your diagnosis be interpreted? You recommend more intensive psychiatric treatment that is typically provided for major depression or general anxiety disorder in clinical trials. You propose that the treatment be what you consider “cognitive behavior therapy.”

You suggest that I should be unconcerned with your outcomes switching. I believe I am on firm ground in suggesting that that outcomes switching is a bad research practice associated with claims for both pharmacological and psychotherapeutic interventions that proved false or exaggerated in any re-analysis of original outcomes or independent efforts at replication.

You provide no acknowledgment of the issues nor refutation of these general concerns about switched primary outcomes.

You accuse me of making the misleading claim that 50% of the patients in the control group deteriorated. Compare what I said with what you said on page 505 . Namely, “Over half (56%) of patients in the usual care group reported their physical health to be worse than before randomisation, which was the case for only a quarter (25%) of the STreSS group.”

I find this statement quite troubling and hardly an endorsement for the effectiveness of your approach.

You are correct that I missed your follow up study. For that I apologize. But now that I have had a chance to examine it, I do not find it very reassuring. These patients were all treated at your clinic and all had letters to their primary care physicians indicating your diagnosis of them with what you call “body distress syndrome.” Neither established physical nor psychiatric disorder had been an initial exclusion criterion. From an international point of view, it is curious that a documented physical condition is not an exclusion for your diagnosis of a functional somatic symptom disorder. You show that it is a mixed grop of patients but most continue to be physically ill. The failure of the patients to have new diagnoses may simply be due to your letters to their primary care physicians discouraging appropriate attention to their physical health concerns.. You state “Since we did not have access to primary are records, only diagnosis made in secondary and tertiary care were obtained.” That is truly unfortunate limitation, but secondary to this being a follow-up on diagnoses made by your own group. If your diagnostic criteria are going to get any traction in the international scientific and medical community, you need to obtain blinded independent diagnoses in a sample that is not been contaminated by your feedback to primary care physicians. Ideally, this new sample with not involved in the primary care physicians or the interview who has been tipped off about your hypotheses about body distress syndrome.

Without such independently obtained validation, I think you will have a hard time convincing others of the usefulness of applying an additional diagnosis of a functional somatic symptom disorder to a heterogeneous population with multiple confirmed physical illnesses and mental disorders.

You complain about my bringing up the issue of conflict of interest. First, you have a strong investment in a diagnosis and a treatment that have not yet obtained much international acceptance. Second, the credibility of your maintaining such a specialist clinic depends on your ability to attract referrals and to have evidence that what you provide is effective. This trial is your evidence. .

But there is a third issue: one of your authors acknowledged extensive conflicts of interest in publication that was basically concurrent with your article. Why was that conflict of interest not disclosed in the article with you? Your co-author also has a substantial stake in consulting with insurance companies seeking to reduce disability payments by disqualifying patients on the basis of them not being in treatment. Your trial, although flawed, has the appearance of producing such evidence. Finally, in refusing to share data your co-authors investigative group has consistently argued the transparency would risk reputational damage. Although I reject this has a consideration, their argument seems to knowledge that investigators have a lot in stake in evaluating treatments that they have personally developed.

I look forward to your response to this blog post. Although I live in Philadelphia, I often visit Europe. Actually, I was in Copenhagen for a week working on a meta-analysis with some epidemiologists just before I was contacted by the journalist. I could arrange a side trip to Copenhagen on my next visit to Europe. II invite you to engage me in a discussion in a public forum. We could take a poll of the audience before and after the debate, English style. I previously extended such an offer to one of your co-authors, and Simon Wessely assured me it would be accepted. Unfortunately, the debate never happened.

However, I think we have an opportunity to clarify each of our misunderstandings, and to inform and entertain a professional or lay audience.