What motivates someone to publish that paper without checking it? Laziness? Naivety? Greed? Now that’s one to ponder. – Neuroskeptic, Science needs vigilantes.
A recent large trial published in the British Journal of Psychiatry of cognitive behavior therapy for psychosis yielded thoroughly null findings. The abstract presents the findings as impressively positive, but a table revealed otherwise. Another table present seemingly impressive effect sizes, but they were not conventionally calculated.
Readers who rely on the freely available abstract for this pay-walled study will be misled. Careless and simply lazy researchers conducting meta-analysis who rely on the misleading table will compound the problem.
It would be appropriate for the British Journal of Psychiatry to issue a correction.
The study was conducted in China. All of the authors are Chinese, except the last two authors, Douglas Turkington and David Kingdon, who are English. One of them, Douglas Turkington, previously faced considerable controversy in social media when he published incoherent tables in an article reporting a study of cognitive behavior therapy in the United States. He and the editor of the journal in which the study was published, Journal of Nervous and Mental Disease, attacked critics and initially refused to issue a correction.
The abstract of the British Journal of Psychiatry article present seemingly impressively significant findings:
At the post-treatment assessment and the 12-month followup, patients who received brief CBT showed greater improvement in overall symptoms, general psychopathology, insight and social functioning. In total, 37.3% of those in the brief CBT plus TAU group experienced a clinically significant response, compared with only 19.1% of those in the TAU alone group (P = 0.003).
Brief CBT has a positive effect on Chinese patients with schizophrenia in the community.
The text of the article certainly presents the results in glowing terms:
Clinical significance of symptom changes In accordance with the principle of ITT and with the criterion of a 25% or greater reduction on the PANSS total score, 37.3% (41/110) of those in the brief CBT group experienced a clinically significant improvement compared with only 19.1% (21/110) of those in TAU alone group, and this difference was statistically significant (w2 = 8.983, P= 0.003). The number needed to treat (NNT)26 for improvement in overall symptoms was 6 (95% CI 4–13), which means that for every six patients treated with brief CBT there was one extra good clinical response over TAU.
However, a glance at the key time x treatment interactions presented in a table revealed null effects.
This unambiguous patterns of results is seemingly contradicted by a table with effect sizes. However, the effect sizes are calculated on within-condition pre-post differences, not the more conventional between group differences, which would have been much less impressive.
A 2015 post at my PLOS blog Mind the Brain, Sordid tale of a study of cognitive behavioral therapy for schizophrenia gone bad summarized the furor on the internet when some skeptics- Tim Smits, Stuart Ritchie, Daniel Lakens, and Keith Laws- identified some serious problems in tables in an article in Journal of Nervous and Mental Disease. The critics blogged about the flaws in the article and published letters to the editor.
Aside from gross sloppiness, among the problems critics pointed out was a confused and inaccurate presentation of effects, similar to what is now being done in the British Journal of Psychiatry article.
However, Turkington and the statistical editor for the journal, Dominec Cicchetti personally attacked the critics. Turkington then escalated and challenged his critics to come to his university to debate him.
I got fed up witnessing the attacks on the mellow, restrained post-publication reviewers. I blogged a second time and offered to come to Newcastle and debate Turkington.
Turkington replied that I should first establish my credentials by sending a link to my Google Scholar profile so he could establish if I had any standing in the field. He did not respond when I complied.
Dr. Turkington, I renew my offer to debate you. But really, before this silliness goes any further, Kamaldeep Bhui, Editor of British Journal of Psychiatry should issue a correction. I suggest he be nicer than Dominec Cicchetti in doing so.
I will soon be offering e-books providing skeptical looks at mindfulness and positive psychology, as well as scientific writing courses on the web as I have been doing face-to-face for almost a decade.
Sign up at my new website to get advance notice of the forthcoming e-books and web courses, as well as upcoming blog posts at this and other blog sites. Get advance notice of forthcoming e-books and web courses. Lots to see at CoyneoftheRealm.com.