Mindfulness research’s huge problem with uninformative control groups

Are enthusiasts protecting cherished beliefs about the power of mindfulness from disconfirmation?

Do any advantages of mindfulness training disappear in a fairly matched cage fight with a treatment of comparable frequency and intensity?

  • Very few of the 1000s of articles retrieved in a literature search with the keyword “mindfulness” represent advances in the limited evidence that mindfulness-based stress reduction (MBSR) is effective for physical health problems.
  • Only a few randomized controlled trials with appropriate control groups are available and they do not offer strong evidence for the efficacy of MBSR.
  • This blog post demonstrates how uninformative and misleading comparisons with no treatment or treatment as usual/routine care can be.
  • While the lack of adequately controlled studies could have initially reflected the naïveté of MBSR researchers, increasing acknowledgment of the problem suggests enthusiasts’ avoidance of confronting cherished beliefs with disconfirming evidence.
  • When cage fights are arranged between MBSR and appropriate active control groups, the alternative treatments are often shown to be superior and more cost-effective, even when MBSR enthusiasts are the referees.

A comprehensive systematic review and meta-analysis prepared for the US Agency for Healthcare Research and Quality (AHRQ)

Goyal M, Singh S. Sibinga EMS, et al. Meditation programs for psychological stress and well-being: a systematic review and meta-analysis. JAMA Intern Med. Epub Jan 6 2014. doi:10.1001/jamainternmed.2013.13018.

Reviewed 18,753 citations, and found only 47 trials (3%) with 3515 participants that included an active control treatment.

Mindfulness meditation programs had moderate evidence of improved anxiety (effect size, 0.38 [95%CI, 0.12-0.64] at 8 weeks and 0.22 [0.02-0.43] at 3-6 months), depression (0.30 [0.00-0.59] at 8 weeks and 0.23 [0.05-0.42] at 3-6 months), and pain (0.33 [0.03- 0.62]) and low evidence of improved stress/distress and mental health–related quality of life. We found low evidence of no effect or insufficient evidence of any effect of meditation programs on positive mood, attention, substance use, eating habits, sleep, and weight. We found no evidence that meditation programs were better than any active treatment (ie, drugs, exercise, and other behavioral therapies).

An accompanying commentary on the review asked:

The modest benefit found in the study by Goyal et al begs the question of why, in the absence of strong scientifically vetted evidence, meditation in particular and complementary measures in general have become so popular, especially among the influential and well educated…What role is being played by commercial interests? Are they taking advantage of the public’s anxieties to promote use of complementary measures that lack a base of scientific evidence? Do we need to require scientific evidence of efficacy and safety for these measures?

A reminder: treatments do not have effect sizes.

MBSR does not have an effect size. Rather, comparisons of MBSR to other conditions have effect sizes, which will vary greatly with the comparison treatment and population being studied.

Not just any comparison/control condition will do.

A comparison/control condition must be suitably matched with MBSR in terms of frequency and intensity of contact, positive expectations, and overall levels of support and attention. MBSR treatments typically involve weekly meetings, daylong workshops or retreats, and the expectations that patients will practice mindfulness daily.

Construction of an adequate control condition that matches these features can be challenging.

Comparisons of MBSR with wait list controls and no treatment control conditions produce exaggerated effect sizes for active treatments and may produce positive findings were no differences would be found with an adequate control group.

The domination of the MBSR literature by nonrandomized trials and randomized trials with inadequate control groups represents one contribution to an exaggeration of the efficacy of MBSR.

Demonstrating how uninformative and even misleading poorly chosen control groups can be.

 Spirometry_NIHA study published in NEJM that did not evaluate MBSR nonetheless demonstrates how misleading poorly chosen control groups can be, especially for physical health outcomes.

Wechsler ME, Kelley JM, Boyd IO, Dutile S, Marigowda G, Kirsch I, Israel E, Kaptchuk TJ. Active albuterol or placebo, sham acupuncture, or no intervention in asthma. New England Journal of Medicine. 2011 Jul 14;365(2):119-26.

 This randomized, double-blind, crossover pilot study involved screening 79 patients, of whom 46 with mild-to-moderate asthma met the entry criteria, and were randomly assigned to one of four study interventions. An inhaled albuterol bronchodilator was compared to one of three control conditions placebo inhaler, sham acupuncture, or no intervention. Figure 4 from the article presents subjective outcomes for two self-report measures, perceived improvements in asthma symptoms on a visual-analogue scale and perceived credibility of treatment.

percent change subjectivePatients reported substantial improvement not only with inhaled albuterol (50% improvement) but also with inhaled placebo (45%) and with sham acupuncture (46%). In contrast, the improvement reported with no intervention was only 21%. The difference in the subjective drug effect between the active albuterol inhaler and the placebo inhaler was not significant (P=0.12), and the observed effect size was small (d=0.21). With respect to the placebo effects, however, the difference between the two placebo interventions and no intervention was large (d=1.07 for placebo inhaler and d=1.11 for sham acupuncture) and significant (P<0.001 for both comparisons). Treatment credibility was high, and most patients believed that they had received active treatment (73% for double-blind albuterol, 66% for double-blind placebo inhaler, and 85% for sham acupuncture). The two double-blind conditions did not differ significantly from each other, but sham acupuncture was significantly more credible than both inhaler conditions (P<0.05).

Figure 3 from the article  presents the outcomes for an objective measure physiological responses – improvement in forced expiratory volume (FEV1), measured with spirometry   to each intervention (albuterol inhaler, placebo inhaler, sham acupuncture, and no intervention) across the three study visits.

percent chane objective

The mean percent improvement in FEV1 was 20.1±1.6% with inhaled albuterol, as compared with 7.5±1.0% with inhaled placebo, 7.3±0.8% with sham acupuncture, and 7.1±0.8% with the no-intervention control. There were no significant differences between the three inactive interventions, none of which resulted in the degree of improvement observed with active albuterol. The difference in drug effect between the albuterol inhaler and the placebo inhaler, as indexed by the difference in mean percent improvement in FEV1, was significant (P<0.001) and large (d=1.48). In contrast, the placebo effects did not differ significantly between the two placebo interventions and the no-intervention control (P=0.65 for the comparison of placebo inhaler with no intervention, and P=0.75 for the comparison of sham acupuncture with no intervention).

The authors concluded:

In this repeated-measures pilot study in which active-drug and placebo effects were assessed in patients with asthma, two different types of placebo had no objective bronchodilator effect beyond the improvement that occurred when patients received no intervention of any kind and simply underwent repeated spirometry (no-intervention control). In contrast, the subjective improvement in asthma symptoms with both inhaled placebo and sham acupuncture was significantly greater than the subjective improvement with the no-intervention control and was similar to that with the active drug.

Relevance to Studies of MBSR.

 Claims for the efficacy of MBSR depend heavily on RCTs comparing MBSR to waitlist. I’m unaware of comparisons of the standard waitlist control condition to more appropriate comparison/control conditions.  However, this unusual pilot study provides some suggestive evidence that a waitlist is seriously deficient when compared to credible comparison/control conditions for which patients are likely to have positive expectations.

We should be cautious in interpreting these results because we will be comparing effect sizes across different kinds of studies. But with this caution in mind, we can see that for subjective self-report measures, the large difference between placebo conditions with positive expectations and no treatment is certainly greater than the differences typically found between the MBSR and a waitlist.

The difference between a waitlist control group and a blinded control with blinding group with positive expectations is considerably greater than the difference between MBSR and a waitlist control group. This spells trouble for anyone wanting to crow about MBSR.

It’s not an unreasonable inference that comparison with more appropriate comparison/control conditions will eliminate any advantage of MBSR. I would welcome a direct test of this hypothesis by pitting MBSR against a placebo condition with positive expectations and another comparison control condition like waitlist or no treatment.

The contrast between results from subjective self-report and objective outcomes should be troubling to those needing to evaluate MBSR or other psychological interventions for clinical or health policy applications.  If one relies on studies with subjective self-report as the primary outcome, the risk is that differences for objective health measures will be missed and ineffective treatments will be accepted as effective. Ouch!

For a large proportion of studies of psychological interventions for chronic health conditions, the primary outcomes are indeed subjective self-report. Even when conceptually possible, objective measures of health conditions are either not included or they are deemphasized as secondary outcomes. The message of this study, again delivered with appropriate caution, is that we should not be generalizing from results obtained with subjective self-report to objective health outcomes.

In defense of MBSR researchers, they might not just be defending against his confirmation of a cherished belief. They may also be avoiding a threat to continued funding. An investigator who conducted such a trial and got the expected result would jeopardize getting further funding for MBSR trials.

Cage fights between MBSR and active control conditions.

 corey nelsonComparisons between MBSR an active control conditions are the real test of whether MBSR is effective and distinctively so. Such “cage fights” become particularly important when MBSR enthusiasts are not the referee. Investigator allegiance is an important determinant of outcome. Yet even when cage fights are refereed by investigators rooting for MBSR, the results can be disappointing.

In a recent blog post, I examined a trial of MBSR for smoking cessation that was published too late to be included in the comprehensive systematic review and meta-analysis.


The well-designed study

Vidrine JI, Spears CA, Heppner WL, Reitzel LR, Marcus MT, Cinciripini PM, Waters AJ, Li Y, Nguyen NT, Cao Y, Tindle HA. Efficacy of Mindfulness-Based Addiction Treatment (MBAT) for Smoking Cessation and Lapse Recovery: A Randomized Clinical Trial. Journal of Consulting and Clinical Psychology. 2016 May.

Compared mindfulness-based abstinence therapy (MBAT) to cognitive behavior therapy, which was closely matched for frequency and intensity of contact and credibility. The control/comparison group was four  5-10 minute individual counseling sessions. Although the comparison was lopsided in terms of frequency and intensity of meetings, there were no differences among the three groups. The authors did not emphasize that  a reason for the finding that all three groups received a nicotine patch with instructions.

Another new large study 

Daubenmier J, Moran PJ, Kristeller J, Acree M, Bacchetti P, Kemeny ME, Dallman M, Lustig RH, Grunfeld C, Nixon DF, Milush JM. Effects of a mindfulness‐based weight loss intervention in adults with obesity: A randomized clinical trial. Obesity. 2016 Apr 1;24(4):794-804.

Compared mindfulness training to a 5.5 month active control condition that was carefully matched.

To control of attention, social support, expectations of benefit, food provided during the mindful eating exercises, and home practice time in the mindfulness intervention, the control intervention included additional nutrition and physical activity information, strength training with exercise bands, discussion of societal issues concerning weight loss, snacks, and home activities. We controlled for a mindfulness approach to stress management by including progressive muscle relaxation and cognitive-behavioral training in the control group, although at a lower dose than in the mindfulness intervention.

There were no differences in weight loss between the two groups. Questions can be raised about how different the two treatments actually were, but part of the problem is that it is difficult to design such a treatment of comparable frequency and intensity of contact with credible content that does not overlap.

I would anticipate that comparisons between MBSR and appropriate active control conditions will be slow to accumulate. But the results at this point are not encouraging of the notion that MBSR is distinctively more effective than other active control conditions when delivered with the same frequency of contact, intensity, and positive expectations.

Leave a comment

Please note, comments must be approved before they are published