- A well-orchestrated publicity campaign for a Lancet Psychiatry article promoted the view that locked inpatient wards are ineffective in reducing suicide.
- This interpretation is not supported by data in the actual paper, but plays to some entrenched political stances and prejudices.
- Hype and distortions in conventional and social media about this article are traceable directly to quotes from the authors in press releases from Lancet and from their university.
- Mental Elf posted a blog the day the embargo on reporting this study was lifted. The blog post and an associated Twitter campaign generated lots of social media attention. Yet, there is no indication that the blogger went beyond what was in press releases or compared the press releases to what was in the actual article.
- Not many of the re-tweets and “likes” were likely from people who had read the original research.
- The publicity orchestrated for this study raises issues about the ethics of promoting clinical and public policy with claims of being evidence-based when the audience does not have the ability to evaluate independently the claims by actually reading the peer-reviewed article.
I obtained the article from interlibrary loan and the supplementary material from the authors. I appreciate the authors’ immediate responsiveness to my request.
[I delayed this blog post for a week because of indications that the article would be released from behind the pay wall, but apparently it has not been freed.]
In this blog post I identify important contradictions between the authors’ claims in the article and what they promoted in the media. The contradictions are obvious enough that someone other than the authors – the Lancet Psychiatry editor and reviewers – should have immediately caught them.
Spoiler: Claims supposedly based on sophisticated multivariate techniques that were applied to data from hundreds of thousands of patients were actually based on a paltry 75 completed suicides. These were a subsample of at least 174 that occurred in 21 hospital settings in the course of 15 years. Throwing way a chunk of the data and the application of multivariate analyses to such a small, arbitrarily chosen subsample is grossly inappropriate. Any interpretations are likely to be invalid and unreliable.
No one else seems to be commenting on these key features of the study, nor the other serious problems of the study that I uncovered when I actually examine the paper and supplements. Join me in the discovery process and see if you agree with me. Please let me know if you don’t agree with my assessment.
The promotion of the study can be seen as a matter of ideologically-driven mistreatment of data with the intention of promoting clinical and public policies that put severely disturbed persons at risk for suicide.
Regardless of where one stands as to whether severely disturbed persons should be prevented from hurting or killing themselves, this attempted manipulation of public policy should be viewed as objectionable.
In presenting what may be controversial points, I’ll start with editorials that were easily accessible. I’ll then delve into the paywalled article itself.
The press release from the authors’ University of Basil
This press release, Psychiatry on closed and open wards: The suicide risk remains the same provided limited details of the study, but misrepresented the study’s finding of risk for suicide as being based on 350,000 patients.
The study’s last author declared his agenda in promoting the study:
Focus on ethical standards
“Our results are important for the destigmatization, participation and emancipation of patients, as well as for psychiatric care in general,” comments last author Undine Lang, Director of the Adult Psychiatric Clinic at UPK Basel. The results will also have an influence on legal issues that arise when clinics adopt an open door policy. In future, treatment should focus more on ethical standards that ensure patients retain their autonomy as far as possible, says Undine Lang. Efforts should also be made to strengthen the therapeutic relationship and joint decision-making with patients.
The press release from The Lancet
Distributed while the article was still embargoed, Locking doors in mental health hospitals does not lower suicide rate provided more details of the study, but more editorializing grounded in direct quotes from the authors:
Locking the doors of mental health hospitals does not reduce the risk of suicide or of patients leaving without permission, according to a study published in The Lancet Psychiatry.
Authorities around the world are increasingly using locked-door policies to keep patients safe from harm, but locked doors also restrict personal freedom.
European countries tend to follow traditional approaches in caring for patients in psychiatric care, because there has been little evidence so far that one method is better than another.
Similar outcomes whether doors are open or locked.
Of 349,574 patients, they selected 72,869 cases from each hospital type, or 145,738 cases altogether. Creating matched pairs enabled a direct comparison between hospitals.
Translation: to prepare the data for the statistical analyses the authors had planned, they threw away 203,836 cases, or 58.3% of the available cases.
And they concluded:
Findings revealed similar rates of suicide and attempted suicide, regardless of whether a hospital had a locked door policy or not. Furthermore, hospitals with an open door policy did not have higher rates of absconding, either with or without return. Patients who left an open door hospital without permission were more likely to return than those from a closed facility.
The press release next raised a dramatic question. But could these data answer it?
Do locked doors unnecessarily create a sense of oppression?
Given the similarity of outcomes between the two types of hospital, the researchers propose that an open door policy might be preferable.
“These findings suggest that locked door policies may not help to improve the safety of patients in psychiatric hospitals, and are not generally successful in preventing people from absconding. In fact, a locked door policy probably imposes a more oppressive atmosphere, which could reduce the effectiveness of treatments, resulting in longer stays in hospital. The practice may even lend motivation for patients to abscond.” -Dr. Christian Huber, of the Universitäre Psychiatrische Kliniken Basel, Switzerland
Of course, the study did not assess anything like “sense of oppression” and so cannot answer this question. As we will see when I discuss what I found in the actual paper, Dr. Huber’s characterization of his findings is untrue. Patients on locked wards did actually not have longer stays.
Since each hospital serves a specific location, there was no chance of higher-risk patients being allocated to hospitals with locked wards. This reduced the risk of bias.
This is also not true. An unknown proportion of the hospitals, probably most, had both locked and unlocked wards. There could easily have been strong selection bias by which patients was referred to a locked ward. We are not told whether patients could be referred into other catchment areas, but this information would be useful in interpreting the authors’ claims.
The authors warn that an open door strategy might not be appropriate everywhere, as mental health care provision differs in other ways, too, for example, how many beds are available, the percentage of acutely ill patients, and how long they are treated for.
Germany has around 1.1 psychiatric care beds for every 1,000 people, compared with 0.5 beds per 1,000 in the United Kingdom and 0.3 in the United States. Where there are fewer beds, patients who receive treatment are more likely to be severely ill and more at risk.
So, Germany has more than 3 times the beds/100 people than the USA and more than the twice the availability of beds in the UK. We can learn from other sources:
Germany is one of the countries with highest expenditure for mental health care in the world. However, in contrast to other western European countries, psychiatric treatment in Germany is still mainly provided by psychiatric hospitals, outpatient clinics and office based psychiatrists and only rarely by community mental health teams. As mental health policy, except the provision of pharmaceutical treatment, is the responsibility of the federal states, no national mental health plan exists. Therefore, community mental health care systems vary widely with regard to conceptual, organisational and economic conditions across the country. Moreover, the fact that different components of community mental health care are funded by different payers (and on different legal bases) hampers coordination and integration of services.
Studies largely conducted in other countries with organizations of care different than in Germany have consistently concluded that Assertive Community Treatment (ACT) programs are effective in reducing the need for inpatient treatment.
In order to keep the level of psychiatric inpatient treatment and institutional care as low as possible these services should be provided by multi-professional community mental health teams organized according to the principles of Assertive Community Treatment (ACT).
ACT programs keep persons with psychosis from being placed in psychiatric inpatient units like those studied in the Lancet Psychiatry and they lead to shorter hospital stays.
The Lancet Psychiatry article makes no mention of ACT in Germany. My inference is that implementation was not widespread during the study. If there are ACT programs in Germany, their influence on this data set is through an invisible hand.
Inpatient psychiatric beds are quite scarce in the US, even for patients and families willing to pay out of pocket. To deal with demand that is not met by psychiatric facilities, the Los Angeles jail has become the largest locked facility. Whether it not it was the intention of the Lancet Psychiatry, the ideology with which it is infused has served to make inpatient beds less available in the United States and greater reliance on jails instead of least restrictive, and more supportive settings for protecting persons with psychosis.
Inpatient hospitalizations in the United States are much shorter than in Germany. In some states, the mean length of stay is five days. Hospitalization has different goals in the US- only stabilization of the patient’s condition.
The means of killing oneself are also different between the US and Germany. Firearms are much more readily available in the US than in Germany, suggesting different means-restriction strategies for reducing suicide.
So, I cannot see the generalizability of the findings from the Lancet Psychiatry study to the US – or the UK, for that matter. Can you?
The Mental Elf: Locked wards vs open wards: does control = safety?
The Mental Elf advertises itself as offering “no bias, no misinformation, just what you need.” Its coverage of the study occurred the same day the embargo was lifted. Its coverage uncritically echoed what was in the press releases, adding some emotional and ideologically-driven amplification.
The reason usually given for wards being locked is that the people within them need to be kept safe; safe from harming themselves and safe from committing harm to others. Of course these are very real fears, but they are often wrongly magnified by a still sadly stigmatising media and public perception of severe mental illness.
There is certainly an uneasy tension between the Mental Health Act Code of Practice and the reality of locking up severely ill mental health patients, which is brought into sharp focus when we consider the lack of evidence for locked wards. The literature is primarily made up of expert opinion that insists safety is paramount, but fails to provide any compelling evidence that locking people up actually increases safety.
Let’s examine Mental Elf’s claim of the lack of “any compelling evidence that locking people up actually increases safety.” Presumably, he is referring to the lack of RCTs.
I have been a scientific advisor to experimental studies like the US PROSPECT study and quasi-experimental European studies attempting to test whether suicidality could be reduced. Any such studies suffer from the serious practical limitation that suicide is an infrequent event. But to say there is no compelling evidence for restricting opportunities for acutely suicidal persons to hurt themselves is akin to BMJ’s spoof systematic review finding no evidence from RCTs that parachutes reduce deaths when jumping of planes.
Neither RCTs nor the propensity analyses of administrative data that Mental Elf favors can produce “compelling data.” As I will soon show, this study displays the pitfalls of propensity analyses.
We can systematically examine the contextual circumstances of particular deaths by suicide when they do occur, and make suggestions whether some sort of means restriction, including access to a locked inpatient unit would have made a difference. We can also hold professionals in a decision making capacity legally responsible when they fail to avail themselves of such facilities, and we should.
The Mental Elf wrapped on a rousing, uncritical, and ultimately nonsensical note:
This is a novel and compelling study, conducted in Germany, but very relevant to any Western country that has a secure system for mentally ill inpatients.
Our obsession with security and safety in an ever more dangerous world is justified if you watch the TV news channels for any prolonged period of time. The world is after all full of war, terrorism, violent crime, child abuse; or so we’re led to believe.
I spent a very enjoyable day at City University last week, participating in the #COCAPPimpact discussions, which included some rich and very constructive conversations about therapeutic relationships. It doesn’t take much to appreciate that relationships (therapeutic or otherwise) are stronger and more equitable on open wards.
The Mental Elf website claims (8/5/20016) 215 responses to this post. All but a very few were approving tweets that did not depend on the tweeter having read the study.
The reference to TV news channels is at the level of evidence of a Donald Trump tweet in which he refers to something he saw on TV.
Taking a look at the actual article and its supplementary information.
Christian G. Huber, Andres R. Schneeberger, Eva Kowalinski, Daniela Fröhlich, Stefanie von Felten, Marc Walter, Martin Zinkler, Karl Beine, Andreas Heinz, Stefan Borgwardt, and Undine E. Lang. Suicide Risk and Absconding in Psychiatric Hospitals with and without Open Door Policies: A 15-year Naturalistic Observational Study. The Lancet Psychiatry, 2016 DOI: 10.1016/S2215-0366(16)30168-7
At the time of the media campaign, most people who wanted to access the article could only obtain its abstract, which you can click here .
Why were there only 75 suicides being explained?
Much ado is being made of 75 suicides that occurred over a 15 year period across 21 hospitals. Suicides are an infrequent event, even in high risk populations. But why were only 75 available for analysis from a sample that initially consisted of 350,000 in this amount of time?
Let’s start with the 350,000 admissions that are misrepresented as “cases” in the official press releases. The article states:
The resulting dataset contained 349 574 hospital admissions from 177 295 patients.
Presumably, a considerable proportion of these patients had multiple admissions over the 15 years. Suicides were probably concentrated in the group with multiple admissions. But some patients had only one admission. Moreover, some patients may have been admitted to different types of facilities – locked versus unlocked – on different occasions. Confusion is being generated, bias is being introduced, and valuable information is being lost about the non-independence of observations – i.e., admissions.
How many suicides occurred among these 349 574 hospital admissions? Readers cannot tell from the article. Table 4 states that multivariate analyses were based on predicting 79 suicides. Yet, going to supplementary materials, Table S1 indicates that the analyses were done without the matching requirements imposed by propensity analyses, there were 174 suicides to it explain. The authors aren’t particularly clear, but it appears that in order to meet the requirements of their propensity analysis, they threw away data on most of the suicides.
The exaggerated power of propensity analyses
The authors extol the virtues of propensity analyses:
We used propensity score matching and generalised linear mixed-eﬀects models to achieve the strongest causal inference possible without an experimental design. Since patients were not randomly allocated to the diﬀerent hospital types, causal inference between hospital type and outcomes might be biased—potential confounders could aﬀect both the probability of relevant outcomes and the probability of a case having been admitted to a speciﬁc hospital type. The propensity score of patients reﬂects their probability of having been admitted to a hospital with an open-door policy rather than one with a locked-door policy.15 By matching cases from both hospital types based on their propensity score, datasets with similar distributions of confounders can be generated. These allow stronger causal inference when analysed.15
A full discussion of propensity analyses is beyond the scope of this blog post. I worry that I would lose a lot of readers here if I attempted one. But here is a very readable, accessible source:
Glynn RJ, Schneeweiss S, Stürmer T. Indications for propensity scores and review of their use in pharmacoepidemiology. Basic & Clinical Pharmacology & Toxicology. 2006 Mar 1;98(3):253-9.
It remains unclear whether, and if so when, use of propensity scores provides estimates of drug effects that are less biased than those obtained from conventional multivariate models. In the great majority of published studies that have used both approaches, estimated effects from propensity score and regression methods have been similar.
Use of propensity scores will not correct biases from unmeasured confounders, but can aid in understanding determinants of drug use and lead to improved estimates of drug effects in some settings.
One problem with applying analysis of propensity scores to the data set used in the Lancet Psychiatry is that there was a great deal of difficulty matching the admissions to different settings. Moreover, because it was an administrative data set, there are numerous unmeasured, but particularly crucial confounds that could not be included in the propensity matching or in the generalised linear mixed-eﬀects model analyses thereafter. So, in using propensity analysis, the authors threw way most of their data without been able to achieve adequate statistical control for confounds.
We calculated propensity scores for all cases based on a model that included all clinical characteristics before admission as exploratory variables (age, sex, marital status, housing situation, living together with others, employment situation, main diagnosis, comorbid substance use disorder, comorbid personality disorder, comorbid mental retardation, self-injuring behaviour before admission, suicidal ideation before admission, suicide attempt before admission, type of admission, and voluntary admission). These calculations were done on a complete case basis, therefore 36 300 (10·4%) cases with missing covariate were excluded.
There is the temptation to ask “what is the harm in adjustments that involve the loss of only 10.4% of cases, particularly if better statistical control is achieved?” Well,
Overall, 72,869 pairs of matched cases could be created, resulting in a total matched set consisting of 145,738 cases from 87,640 individual patients for the analyses themselves.
So, the authors have lost a nonrandom selection of more than half the admissions with which they started, and they’ve lost the nonindependence of observations in this shrunken data set. Just look at the ratio of 145,738 “cases” to the 87,640 individual patients from which they came. There is a lot of valuable data being suppressed concerning the fate of individual patients when hospitalized in different settings.
How complete is the data available for matching and control of statistical confounds?
We calculated propensity scores for all cases based on a model that included all clinical characteristics before admission as exploratory variables (age, sex, marital status, housing situation, living together with others, employment situation, main diagnosis, comorbid substance use disorder, comorbid personality disorder, comorbid mental retardation, self-injuring behaviour before admission, suicidal ideation before admission, suicide attempt before admission, type of admission, and voluntary admission.
Recall that the effectiveness statistical controls assumes that all relevant variables have been measured with perfect precision. Statistical control is supposed to eliminate crucial differences among patients so they can be assumed to be otherwise equivalent in likelihood of being admitted to a locked or unlocked ward for the basis of analysis and interpretation. Statistical control is supposed to equip us to make “all-other-things-being equal” judgments about the effects of being in a locked or unlocked ward.
Zero in on main and comorbid diagnoses. What kind of statistical voodoo can possibly be expected to level other differences between patients at higher risk for suicide like the 49% minority with schizophrenia spectrum or affective of disorder versus the others at considerably lower risk? How does it help that this large minority of higher risk patient is thrown in with lower risk patients with organic mental disorder (dementia or mental retardation) and “neurotic, stress-related and somatoform disorders”?*
If there’s any rationality to the German system of care (and I assume there is), at least some crude risk assessment would guide patients with lower risk into less restrictive settings.
And then there is the question of substance use disorder, which was the primary diagnosis for 67,811 (25·5%) of the patients going into locked facilities and 14,621 (18·7%);
Substance use disorder was the comorbidity for another 100 128 (36·9%) going into locked facilities and 28 363 (36·2%) going into unlocked facilities. Issues for substance use disorder and exit security on psychiatric wards are very different than for patients without such disorders. These issues in relationship to absconding or dying by suicide are not going to be sorted by entering diagnosis into a propensity analysis or generalised linear mixed-eﬀects model analyses of a data set shrunken by matching in a propensity analysis.
I conclude that the data set is much less impressive and relevant than it first appears. There are not a lot of suicides. They occur in a heterogeneous population in a length of time in which the patterning of circumstances associated with these characteristics likely changed. Because it was the administrative data set, there were restricted opportunities for matching of patients or control of confounds. Any substantive interpretation of multivariate results requires dubious, unsubstantiated assumptions.
But more importantly, the data set does not provide much evidence for the ideologically saturated claims of the authors or their promoter, Mental Elf. They can pound their drums, but it is not evidence that they are announcing. And patients and their families in both Germany and elsewhere could suffer if the recommendations are taking seriously.
*The “neurotic, stress-related and somatoform disorders” admissions to inpatient units are a distinctly German phenomenon. Persons from the community claiming “burnout” can be admitted to facilities overseen by departments of psychotherapy and psychosomatics. There is ample insurance coverage for what can be a spa-like experience with massage and integrative medicine approaches.