This slide presentation comes from a Critical Appraisal Class that I teach in the Health Psychology Section at the University Medical Ctr., Groningen (UMCG). It analyzes a meta-analysis that I particularly liked for its transparent, straightforward analysis of the literature concerning interventions for families of adult patients with chronic illnesses. You can find the actual article here.
Meta-analysis has replaced the randomized clinical trial (RCT) as the highest form of evidence for the efficacy of interventions. It’s been recognized that individual RCTs are imperfect, often carry substantial risk of bias, and tend to contradict each other in their evaluations of interventions. Meta-analyses provide the hope of overcoming the biases of individual trials by providing a summary overall effect size for an intervention taking into account what individual trials have found.
As I’ve noted in articles in numerous blog posts, meta-analyses often do not live up to these expectations. They are often badly conducted, with a deliberate bias in order to arrive at a particular conclusion, regardless whether it is justified by the evidence. RCTs in health and clinical psychology evaluating interventions are often undersized (underpowered) and share particular methodological problems that carry a high risk of bias. Simply combining them in a meta-analysis only compounds the problems, it does not overcome them.
I’ve shown published meta analyses, often in high impact journals
With effect sizes incorrectly calculated so that one psychotherapy is found superior to all others.
Deliberately skewed by anti-abortion activists in order to frighten women about the mental health effects of abortion.
Exaggerating the efficacy of couples interventions for cancer patients.
Conducted by professional organizations to promote reimbursement of the services provided by the members.
Slanted to support an author’s pre-existing view that antidepressants are no better than a placebo.
Falsely claiming that acupuncture and yoga are as efficacious as antidepressants and psychotherapy for the treatment of depression.
One can get a sense from my writings that I do not have a high opinion of meta-analyses. I certainly find flaws in them when I examine them closely.
However, this slide presentation discusses a meta analysis that is refreshingly different.
What I do in the slides is map correspondence of statements in the article with items from the Quality of Reporting of Meta-Analyses (QUORUM) which has now become the Preferred Reporting Items for Systematic Reviews and meta-analyses (PRISMA). If you are interested, it can be an informative exercise to first read the article and form your own opinion, and then see what I have to say in the slide presentation.
It’s important to note that QUORUM and PRISMA are checklists to evaluate what is reported in an article about systematic reviews or meta-analyses. They do not evaluate whether the meta-analyses is done well or even appropriately. An article about a badly done meta analysis can score high points for its reporting. Readers at least have the information to evaluate for themselves what was done and how it was interpreted.
AMSTAR is the only validated checklist for evaluating the quality of what was done in a meta analysis. Another useful exercise would be to apply AMSTAR to evaluate this meta-analysis.
The authors of this meta-analyses had to contend with the literature consisting of small, imperfect trials. They did not gloss over this, but tried to document problems and take them into account in their conclusions. Unlike meta-analyses that I have complained about in my blogging and in letters to the editor, this one doesn’t jump to making a sweeping recommendation that these interventions are ready to be rolled out and widely implemented. Rather, the authors concede that their review probably has greater implications for encouraging more research rather than deciding about dissemination of the existing interventions.
I wish I could find more meta analyses in psychology so carefully done and transparently reported.