Systematic review shows no improvement in quality of mindfulness research in 16 years

Should we still take claims about mental health benefits of mindfulness with a grain of  salt? A systematic review by one of mindfulness training’s key promoters suggests maybe so.

saltCritics have been identifying the same weaknesses in mindfulness research for almost two decades. This review suggests little improvement in 16 years the quality of randomized trials for mental health problems.

This study examined 171 articles reporting RCTs for:

(a) active control conditions, (b) larger sample sizes, (c) longer follow-up assessment, (d) treatment fidelity assessment, (e) reporting of instructor training, (f) reporting of ITT samples.

What was missed

Whether articles reporting RCTs had appropriate disclosure of financial or other conflicts of interest. COI pose significant risk of bias, especially when they are not reported.

This article discloses authors’ interests. One of the authors, Richard Davidson is a prominent promoter of mindfulness training.  A Web of Science search of Davidson RJ and mindfulness yielded 26 articles from 2002 to 2016. It would be interesting to check in see if these consistent weaknesses in mindfulness research are mentioned in these articles. To what extent do RCTs with Davidson as an author had these weaknesses, like being underpowered?

Critic: You say financial interests or other investments in a treatment are a risk of bias. Yet, this article is critical of mindfulness research. Wouldn’t you expect a more positive appraisal of the literature because of the authors having a confirmation bias?

Not necessarily. Conflicts of interest are a risk of bias, but don’t discredit an author, They only alert readers to be skeptical. Furthermore, the weaknesses in this literature are so pervasive, it would be difficult to put a positive spin on them.  Besides calling attention to specific weaknesses that need to be addressed in future research can become part of a pitch for more research.

The article

Goldberg SB, Tucker RP, Greene PA, Simpson TL, Kearney DJ, Davidson RJ. Is mindfulness research methodology improving over time? A systematic review. PLOS One. 2017 Oct 31;12(10):e0187298.

End of paper conclusion:

In conclusion, the 16 years of mindfulness research reviewed here provided modest evidence that the quality of research is improving over time. There may be various explanations for this (e.g., an increasing number of novel mindfulness-based interventions being first tested in less rigorous designs; the undue influence of early, high-quality studies). However, it is our hope that demonstrating this fact empirically will encourage future researchers to work towards the recommendations here and ultimately towards a clearer and scientifically-informed understanding of the potential and limitations of these treatments.

From the abstract

Objectives

The current systematic review examined the extent to which mindfulness research demonstrated increased rigor over the past 16 years regarding six methodological features that have been highlighted as areas for improvement. These feature included using active control conditions, larger sample sizes, longer follow-up assessment, treatment fidelity assessment, and reporting of instructor training and intent-to-treat (ITT) analyses.

Data sources

We searched PubMed, PsychInfo, Scopus, and Web of Science in addition to a publically available repository of mindfulness studies.

Study eligibility criteria

Randomized clinical trials of mindfulness-based interventions for samples with a clinical disorder or elevated symptoms of a clinical disorder listed on the American Psychological Association’s list of disorders with recognized evidence-based treatment.

Study appraisal and synthesis methods

Independent raters screened 9,067 titles and abstracts, with 303 full text reviews. Of these, 171 were included, representing 142 non-overlapping samples.

Results

Across the 142 studies published between 2000 and 2016, there was no evidence for increases in any study quality indicator, although changes were generally in the direction of improved quality. When restricting the sample to those conducted in Europe and North America (continents with the longest history of scientific research in this area), an increase in reporting of ITT analyses was found. When excluding an early, high-quality study, improvements were seen in sample size, treatment fidelity assessment, and reporting of ITT analyses.

Conclusions and implications of key findings

Taken together, the findings suggest modest adoption of the recommendations for methodological improvement voiced repeatedly in the literature. Possible explanations for this and implications for interpreting this body of research and conducting future studies are discussed.

Competing interests

RD is the founder, president, and serves on the board of directors for the non-profit organization, Healthy Minds Innovations, Inc. In addition, RD serves on the board of directors for the Mind and Life Institute. This does not alter our adherence to PLOS ONE policies on sharing data and materials

The variables examined in the systematic review

Six methodological features that have been recommended in criticisms of mindfulness research [10–12. 14]. These include: (a) active control conditions, (b) larger sample sizes, (c) longer follow-up assessment, (d) treatment fidelity assessment, (e) reporting of instructor training, (f) reporting of ITT samples.

…We graded the strength of the control condition on a five-tier system. We defined specific active control conditions as comparison groups that were intended to be therapeutic [17]. More rigorous control groups are important as they can provide a test of the unique or added benefit a mindfulness intervention may offer, beyond non-specific benefits associated with the placebo effect, researcher attention, or demand characteristics [11,14]. Larger sample sizes are important as they increase the reliability of reported effects and increase statistical power [11]. Longer follow-up is important for assessing the degree to which treatment effects are maintained beyond the completion of the intervention [10]. Treatment fidelity assessment allows an examination of the degree to which the given treatment was delivered as intended [12]. Treatment fidelity is commonly assessed through video or audio recordings of sessions that are coded and/or reviewed by treatment experts [18]. We coded all references to treatment fidelity assessment (e.g., sessions were recorded and reviewed, a checklist measuring adherence to specific treatment elements was completed). Relatedly, reporting of instructor training increases the likelihood that the treatment that was delivered by qualified individuals [12], which should, in theory, influence the quality of the treatment provided. Lastly, the reporting of ITT analyses involves including individuals who may have dropped out of the study and/or did not complete their assigned intervention [12]. Generally speaking, ITT analyses are viewed to be more conservative estimates of treatment effects [19,20], and are preferred for this reason.