A systematic review and meta-analysis claimed a moderate-to large-effect of mindfulness-based stress reduction (MBSR) among breast cancer patients for perceived stress, depression, and anxiety.
- The article recommended MBSR be considered an “option as part of their rehabilitation to help maintain a better quality of life in the longer term.”
- I screened the article and concluded that its conclusions were biased and estimates of the efficacy of MBSR were likely inflated. The quick exercise demonstrates tools that readers can readily apply for themselves to other meta-analyses and particularly to meta-analyses of mindfulness-based treatments, which are prone to low-quality you can read more about some tips for screening out bad meta-analyses from further consideration here.
- This exercise adds to the weight of concerns that we cannot trust the “scientific” mindfulness literature. We need the literature to be scrutinized by researchers not making money or having investment in the promotion of mindfulness-based treatments.
The article appears in a pay walled journal but available through a repository identified on Google Scholar. According to Google Scholar the 2013 meta-analyses has already been signed an impressive 90 times.
Zainal NZ, Booth S, Huppert FA. The efficacy of mindfulness‐based stress reduction on mental health of breast cancer patients: a meta‐analysis. Psycho‐Oncology. 2013 Jul 1;22(7):1457-65.
The abstract announces its conclusions:
On the basis of these findings, MBSR shows a moderate to large positive effect size on the mental health of breast cancer patients and warrants further systematic investigation because it has a potential to make a significant improvement on mental health for women in this group.
But the abstract also disclosed a paucity of data on which this conclusion was based:
Nine published studies (two randomised controlled trials, one quasi-experimental case–control study and six one-group, pre-intervention and post-intervention studies) up to November 2011 that fulfilled the inclusion criteria were analysed. The pooled effect size (95% CI) for MBSR on stress was 0.710 (0.511–0.909), on depression was 0.575 (0.429–0.722) and on anxiety was 0.733 (0.450–1.017).
I was skeptical already. We know from the comprehensive US Agency for Healthcare Research and Quality (AHRQ) report on mindfulness, Meditation Programs for Psychological Stress and Well-Being,that there are thousands of studies of mindfulness-based treatments, but few that are of adequate sample size and methodological quality. The exhaustive search produced 18,753 citations, but only 47 randomized controlled trials (RCTs; 3%) that included an active control treatment.
- Is the meta-analysis limited to RCTs? The answer should be “Yes, of course.” but the answer is “No.” Only a minority of the studies (2) are RCTs.
RCTs are preferred for evaluating psychological interventions over other evaluations of treatments .
Moreover, efforts to combine effect sizes from RCTs with those from non-RCTs are generally problematic and produce inflated estimates.
The problem with effect sizes obtained from non-RCTs is that they are likely to be exaggerated by a host of nonspecific factors. But to understand that, let’s first consider what an effect size from RCT provides.
The important principle is that treatments do not have the effect sizes, but comparisons between active treatments and control conditions occurring in RCTs do. Appropriate effect sizes obtained from an RCT are between-group differences in outcomes. A comparison control group allow some controlling for nonspecific factors and any natural improvement in outcomes that would occur with the passage of time. These are particularly important issues for studies of cancer patients, because it robust literature indicates that initial levels of psychological distress decline in the absence of treatment.
So, the within-group effect sizes available from non-RCTs can readily be adjusted and will be exaggerated estimates of the efficacy of the treatment, particularly when combined with effect sizes from RCTs.
We already know that evaluations of mindfulness-based treatments have a serious problem that control groups are typically inadequate and lead to exaggerated estimates of the efficacy of these treatments. Now these authors have compounded the problem but by combining estimates of efficacy from RCTs that are likely exaggerated with those from studies that don’t even have the benefit of between-group comparisons. The credibility of this meta-analysis is in serious jeopardy.
If I were simply searching the literature for an understanding of how effective mindfulness-based treatments are for cancer patients, I would simply move on and find another source
A broad search yielded few suitable studies.
The authors reported systematically searching nine electronic databases using the search terms ‘mindfulness’ or ‘mindfulness-based stress reduction’ and ‘breast cancer ‘and their efforts yielded 625 studies. That’s a lot, but they were able to quickly screen out most of them (n=592) based on examining titles and abstracts. Reasons for exclusion were:
- Not MBSR intervention (n=107)
- MBSR mixed with other intervention (n=14)
- Non cancer populations (n=310)
- Commentaries or review or systematic review or meta analyses (n=133)
- Psychometric measurement (n=28).
That left 33 articles, of which they were able to exclude 24:
- Mixed cancer populations (n=19)
- Not studying effect on mental health (n=2)
- Multiple publications (n=2)
So now we’re down to 9 studies. Personally, I would excluded all but the two RCTs.
Lengacher CA, Johnson‐Mallard V, Post‐White J, Moscoso MS, Jacobsen PB, Klein TW, Widen RH, Fitzgerald SG, Shelton MM, Barta M, Goodman M. Randomized controlled trial of mindfulness‐based stress reduction (MBSR) for survivors of breast cancer. Psycho‐Oncology. 2009 Dec 1;18(12):1261-72.
This was a study comparing 40 survivors of breast cancer assigned to MBSR to 42 survivors remaining in usual care.
Henderson VP, Clemow L, Massion AO, Hurley TG, Druker S, Hébert JR. The effects of mindfulness-based stress reduction on psychosocial outcomes and quality of life in early-stage breast cancer patients: a randomized trial. Breast cancer research and treatment. 2012 Jan 1;131(1):99-109.
The second study compared three groups: 53 early-stage breast cancer patients assigned to MBSR, 52 to a nutritional education program and 58 assigned to usual care.
These two RCTs at least met my usual criteria of having 35 patients per group, which means they had better than a 50-50 chance of detecting a moderate effect if it were present. But how was methodological quality taken into account?
- How was the methodological quality of the studies taken into account?
It was ignored.
It is important to consider methodological quality conducting a meta-analysis. Methodologically poor studies produce higher estimates of efficacy. We know from the report that most studies of mindfulness are of poor quality. We should be particularly concerned about whether investigators were appropriately blinded to randomization procedures so we did not influence patient assignment. We should also be concerned about whether data for all patients entering the trial were available at follow-up or that there was appropriate compensation for any loss. That would allow the gold standard intention-to-treat analyses. Particularly when conducted with cancer patients, studies often lose substantial numbers of patients to follow-up and lose any benefits of randomization.
The authors were already in trouble by including mostly nonrandomized trials, which have their own risk of bias. But the authors simply ignored any consideration of risk of bias, further damaging the credibility of their analyses.
Figure 3 of the article presents effect sizes for all nine studies included in the meta-analysis. We can see that the Lengacher et al, 2009 study did not have a significant effect on depressionor anxiety, only perceived stress. The Henderson et al, 2011 did not measure perceived stress or anxiety, only depression and the effect size was not significant.
Below, I have excerpted the display of effect sizes for perceived stress. As can be seen, the significant overall effect is driven by two small, nonrandomized trials. It’s not surprising that nonrandomized trials would appear to have larger effect sizes, given the manner in which their effect sizes are calculated.
So, we have a meta-analyses of nine studies, only two of which are RCTs. There are no ratings of methodological quality of the studies. Considering past mindfulness research, the methodological quality is expected to be poor and needs to be taken into account. Neither of the two RCTs had significant effects on the mental health outcomes and both were of at least minimally required sample size. The overall effect sizes are driven by small, underpowered, nonrandomized trials. A different conclusion would be reached by limiting consideration to the two randomized trials, but only two trials would not be a good basis for a meta-analysis
So, I’m inclined to dismiss the claims of this meta-analysis as extravagant and to excluded from further consideration. Case closed.
- Who are the authors? Might they have undeclared conflicts of interest?
The senior author, Felicia A. Huppert, is a Founder and Director – Well-being Institute and Emeritus Professor of Psychology at University of Cambridge, as well as a member of the academic staff of the Institute for Positive Psychology and Education of the Australian Catholic University. She is also author of the study of mindfulness training for schoolchildren that was featured in my last blog on the UK Mindful Nation report [http://blogs.plos.org/mindthebrain/2016/11/16/unintended-consequences-of-universal-mindfulness-training-for-schoolchildren/ ]. Recall that the Mindful Nation cited Professor Huppert’s study along with another one as sole support the efficacy of mindfulness training for students with “the lowest levels of executive control and emotional stability.” Yet are critical review of the study revealed that the pair of studies were methodologically poor studies with absolutely null results.
I’m frustrated repeatedly going to the literature and finding methodologically inferior mindfulness studies, which are then evaluated by merchants of mindfulness in flawed meta-analyses that conclude that mindfulness is highly effective and ready for dissemination. Schoolchildren, and in this case, cancer patients, are being misled. Clinicians and policymakers are being misled.
A high level of skepticism is warranted in approaching mindfulness literature, and glowing conclusions about its effectiveness, particularly from those having financial and professional interest in promoting mindfulness should be dismissed.
What can I (and you) do about this flawed review?
Psycho-Oncology is the official journal of the international Psycho-Oncology Society (IPOS). It is strongly biased toward presenting a positive view of what mental health professionals can provide cancer patients, ignoring weaknesses in the evidence. I previously reported my unsuccessful complaints about a biased review claiming that psychotherapy promotes survival of cancer patients that was published without any peer review. I also reported my failed efforts to publish a letter to the editor concerning a flawed meta-analysis of couples interventions for cancer patients. As indicated in the title of the blog post, I got shushed. The letter was initially invited and accepted, and then withdrawn because of the complaints of the author. The Editor then promoted the article that I was complaining about by offering free access to what was otherwise pay wall. Finally, the Journal does not accept letters to the editor or corrective actions undertaken more than six weeks after the article has been published.
But there’s still something I can do, I can post a comment at PubMed Commons detailing the shortcomings and on reliability of this meta-analysis.
I have done so and you can see the comment here. Now, when someone is doing a literature search and comes across the entry for the study on PubMed, they will be alerted that comment has been made and they can read my comment. And you can comment yourself.
Viva post-publication peer-review that is not controlled by editors!