Talking back to the authors of the Northwestern “Blood test for depression” study

translational psychiatry[Update 9/25/2014]This post critiques the press coverage of a recent article in Translational Psychiatry concerning whether a blood test for depression would soon be available. A critique of the bad science of the article itself is now available at PLOS Mind the Brain.

Judging from the extraordinary number of articles in the media, as well as the flurry of activity on Twitter, a recent study coming out of Northwestern University is truly a breakthrough in providing a blood test for depression.

Unfortunately, the many articles in the media have a considerable, almost copy/paste redundancy. Just compare them to the Translational Psychiatry article’s press release. In many instances, there is more than churnalism going on, there is outright plagiarism. Media coverage offers very few demurs or dampening qualifications on what the authors claim. How do journalists put their names on such lack of work?

Similarly, the tweets appear to be retweets of just a couple of messages, although few are labeled as retweets.

I had my usual doubts as to whether the journalists or tweeters have actually read the article. Journalists could always have gone for second opinions to Google Scholar and looked up similarly themed articles and then maybe contacted the authors of similar articles for comments. Journalists could also have loaded the abstract of the Translational Psychiatry article into EtBlast and gotten dozens of recommendations for relevant experts based on text similarity. I see no evidence that this was done.etblast

There must be something intimidating about an article that claims to be testing not for genes, but for gene transcripts associated with depression. Shuts down the critical faculties. Lacking relevant expertise, journalists and tweeters may be inclined to simply defer to the claims of the authors and not further scrutinize the text or tables of the article with whatever relevant knowledge they do have. If this had been done, they might have found things that they could understand that would be very relevant to evaluating the credibility of this article.

Almost all of the hype that is been written about this Translational Psychiatry article originates with its authors, either in the article itself, the press release, or the well-crafted soundbites provided to the media. Yet, some of the latter are simply excerpted from the press release and made to look like the quotes arose in an interview. I promised a full thorough demolition of the article, and that will be forthcoming. However, here, I will analyze some of the statements attributed to two of the authors in the press. There is a fascinating logic, an ideology even to the statements that is of interest in itself. But you also can take this blog post as a teaser for a soon to arrive blog post at PLOS Mind the Brain in the next week or two.

Keep in mind as we scrutinize what the authors say about their study, just how modest it is. The study started by comparing 32 primary care patients participating in a clinical trial to 32 control persons match for age, ethnicity/race, and gender. Five of the primary care patients were lost to follow-up and another five were lost from the 18 month blood draws. Of these last 22 remaining patients, nine were classified in remission of their depression, and 13 not in remission.

So basically we are talking about some exceedingly small samples and comparisons of subsamples. These shrink to a comparison of 9 patients in remission and 13 not in remission for any statements about prediction of treatment outcome. In any other context, how could anyone who knows anything about clinical research accept the results of such analyses?

Furthermore, if we want to talk about any differences observed at baseline versus what was seen at follow-up, it could well be attributed to simple selective loss to follow-up. This is just one of the many alternative explanations of results reported for these data that cannot be adequately tested because of the small sample sizes. The articles talks about utilizing multivariate statistical controls, but that is statistical malpractice in a sample this size that is highly likely to produce spurious findings.

The authors make a number of statements about predicting remission from cognitive behavior therapy, but from the beginning of the study and into follow-up, all of the patients were getting cognitive behavior therapy and considerable proportion getting antidepressants as well. That is no small complication. It is generally assumed that predictors of response to antidepressants should be different than predictors of response to psychotherapy, but there is really no opportunity to examine this within this confounded small sample.

The two authors quoted by name in media coverage are

Eva RedlEva Redei, PhD, Professor in Psychiatry and Behavioral Sciences and Physiology at Northwestern’s Feinberg School of Medicine in Chicago.

David.mohrDavid C. Mohr, PhD, Professor of Preventive Medicine and Director of the Center for Behavioral Intervention Technologies at the Feinberg School of Medicine at Northwestern University.

From an article in Medscape, Blood Test Flags Depression, Predicts Treatment Response:

We were pleased with these findings, including finding biomarkers that continued to be present after people were effectively treated,” co–lead author David C. Mohr, PhD, professor of preventive medicine and director of the Center for Behavioral Intervention Technologies at the Feinberg School of Medicine at Northwestern University in Chicago, Illinois, told Medscape Medical News.

Dr. Mohr noted that essentially, these are markers of traits ― and may show that certain people have a predisposition to the disorder and can be followed more carefully.

Maybe, maybe not, Dr. Mohr. Aside from your modest sample size and voodoo statistics, it is unclear how clinically useful a trait marker would be. After all, we already have a trait marker in neuroticism, and while it is statistically predictive, it does not do all that well in terms of clinical applications. And the alternative of course is simply to have a discussion with patients as the particular symptoms they have and whether alternative explanations can be ruled out.

Recall, Dr. Mohr, this “trait marker” as you assumed it to be, is occurring in the mildly to moderately depressed sample. Clinical depression is a recurring episodic condition, and this “trait” is not going to be expressing itself in a full-blown episode much of the time.

“Abundance of the DGKA, KIAA1539, and RAPH1 transcripts remained significantly different between subjects with MDD and…controls even after post-CBT remission,” report the investigators.

Well, maybe, but it seems a stretch to make such claims from such limited evidence. The 3 transcripts remaining significant after remission are based on the 9 patients who remitted. Three is different than the 9 of 20 transcripts that differed at baseline, but we don’t know if this is a matter of loss to follow up or remission. And even this reduced number of significant differences, 3, is still statistically improbable, given the small sample size, even assuming an effect is present. The authors have no business interpreting their data to the press in this fashion.

In addition, these transcripts “demonstrated high discriminative ability” between the 2 groups, regardless of their current clinical status, thus appearing to indicate a vulnerability to depression.

The authors have no business claiming to have demonstrated “high discriminative ability” with such a small sample. Notoriously, such findings do not replicate. There is always a drop in the performance statistics from such a small sample when replication is attempted in nine seconds sample. Comparison with an earlier paper, reveals that the authors have not even replicate the findings from their earlier study of early onset depression in the present one and that does not bode well.

“This clearly indicates that you can have a blood-based laboratory test for depression, providing a scientific diagnosis in the same way someone is diagnosed with high blood pressure or high cholesterol,” said Dr. Redei.

Maybe someday we will have a blood-based laboratory test for depression, but by themselves, these data do not increase the probability.

“Clinically, simplicity is important. The primary care setting is already completely overburdened. The more we can do to simplify the tasks of these caregivers, the more we’re going to be able to have them implement it,” said Dr. Mohr.

Of all the crass, premature and inaccurate statements I find in this article, this one tops the list. Basically, Dr. Mohr is making a pitch that the blood test he is promoting will free primary care clinicians from having to talk to their patients. All they need to do is give the blood test and prescribe antidepressants.

From A Blood Test for Depression Shows the Illness is not a Matter of Will

“Being aware of people who are more susceptible to recurring depression allows us to monitor them more closely,” said David Mohr, Ph.D., co-lead author of the study in a press release. “They can consider a maintenance dose of antidepressants or continued psychotherapy to diminish the severity of a future episode or prolong the intervals between episodes.”

This advice is not only premature, it is inappropriate for a mild to moderately depressed sample treated in primary care, where monitoring and follow-up are either nonexistent or grossly inadequate. Dr. Mohr’s suggestion if it were taken seriously, would lead to overdiagnosis and overtreatment or prolonged treatment without follow-up and him and re-evaluation.

In general, these authors seem cavalier in ignoring the problems of overdiagnosis. Elsewhere, Dr. Redei is asked about it and gives a flippant response:

There’s a lot of concern about overdiagnosis for psychiatric illnesses already. How do you think your findings might affect that issue?

[Dr. Redl] People who worry about overdiagnosis — they are probably right, and they are probably wrong. Because there is potentially a problem with underdiagnosis, too. In the elderly, for example – we say, “Oh, you’re just old. You don’t have any energy, and you don’t want to do anything — you’re just old.”

From Blood Test Spots Adult Depression: Study

The blood test’s accuracy in diagnosing depression is similar to those of standard psychiatric diagnostic interviews, which are about 72 percent to 80 percent effective, she said.

It is irresponsible rubbish to claim that the study showed that these measures of gene expression were as accurate as current interview methods. The study involved comparing 20 different measures of gene expression to an interview by a bachelor level interviewer using a less than optimal interview schedule that did not allow for explain any questions or probe of the patient’s response. It certainly would not of been allowed in a study for which the data were to be submitted to the US FDA (FDA). And there was no gold standard beyond that.

Additionally, if the levels of five specific RNA markers line up together, that suggests that the patient will probably respond well to cognitive behavioral therapy, Redei said. “This is the first time that we can predict a response to psychotherapy,” she added.

Again, Dr. Redei, you are talking trash that is not justified by the results of your study. The sample is quite small and most of the patients who receive cognitive behavior therapy also received medication.

The delay between the start of symptoms and diagnosis can range from two months to 40 months, the study authors pointed out.

“The longer this delay is, the harder it is on the patient, their family and environment,” said lead researcher Eva Redei, a professor in psychiatry and behavioral sciences and physiology at Northwestern’s Feinberg School of Medicine in Chicago.

“Additionally, if a patient is not able or willing to communicate with the doctor, the diagnosis is difficult to make,” she said. “If the blood test is positive, that would alert the doctor.”

Perhaps, Dr. Redei, you need to be reminded that you are studying mildly to moderately depressed primary care patients, not an inpatient or suicidal sample. What is the hurry to treat them? Current guidelines in much of the world have become conservative about initiating treatment too quickly. In both the United Kingdom and the Netherlands, there is a recommendation for first trying watchful waiting, simple behavioral activation homework, or Internet-based therapy before starting something more intensive like antidepressant therapy or psychotherapy. Certainly if a patient has multiple recurrent episodes or a history of sudden suicidality, a different strategy would be recommended.

And in what clinical situations does Dr. Redei imagine having to initiate treatment when a patient is not able or willing to communicate with the doctor? Would treatment be ethical under those circumstances and how would it receive the necessary monitoring?

From: First ‘Blood Test for Depression’ Holds Promise of Objective Diagnosis

“Currently we know drug therapy is effective but not for everybody and psychotherapy is effective but not for everybody, ” Mohr said. “We know combined therapies are more effective than either alone but maybe by combining therapies we are using a scattershot approach. Having a blood test would allow us to better target treatment to individuals.

Again, Dr. Mohr, this is a widely shared hope by some, but your current study in no way advances us further to achieve this hope of a clinical tool.

In all of the many media stories available about the study, there was little dissent or skepticism. One important exception was

Newsweek’s First ‘Blood Test for Depression’ Holds Promise of Objective Diagnosis

Outside experts caution, however, that the results are preliminary, and not close to ready for use the doctor’s office. Meanwhile, diagnosing depression the “old-fashioned way” through an interview works quite well, and should only take 10 to 15 minutes, says Todd Essig, a clinical psychologist in New York. But many doctors are increasingly overburdened and often not reimbursed for taking the time to talk to their patients, he says.

Essig says it’s “a nice little study” but has no clinical usefulness at this point. That’s because it involved such a small sample of people and because the researchers excluded many patients that real clinicians would see on a daily basis, he says.

“It’s moving basic knowledge incrementally forward, but its way to soon to say it’s a ‘blood test for depression,’” Essig says.

“Depression is not hard to diagnose, even in a primary care setting,” he adds. “If physicians were allowed by health-care delivery systems to spend more time talking with their patients there would be less need for such a blood test.”

Amen, Dr. Essig, well put.

Northwestern Researchers Develop RT-qPCR Assay for Depression Biomarkers, Seek Industry Partners

One has to ask why would these mental health professionals disseminate such misleading, premature, and potentially harmful claims? In part, because it is just quite fashionable and newsworthy to claim progress in an objective blood test for depression. Indeed, Thomas Insel, the director of NIMH is now insisting that even grant applications for psychotherapy research include examining potential biomarkers. Even in the absence of much in the way of promising, clinically useful biomarker candidates, there are points to be scored in grant applications that cite pilot work moving in that direction, regardless of how unjustified the claims are. As John Ioannidis has pointed out, fashionable areas of research are often characterized by more hype and false discoveries than actual progress.

However, comments in one article clearly show that these authors are interested in the commercial potential of their wild claims.

Now, the group is looking to develop this test into a commercial product, and seeking investment and partners, Redei said.

“The goal is to partner to move this as far as possible into the clinic,” Redei said. “There are [other assays] coming behind it, so I would like to focus on [those] … but then this one can move on. For that, I absolutely need partners [and] money, that’s the bottom line,” she said.

Redei envisions developing this assay into a US Food and Drug Administration-approved diagnostic, rather than a laboratory-developed test. “If it’s FDA approved, then any laboratory can do it,” she said.

“I hope it is going to result in licensing, investing, or any other way that moves it forward,” she said. “If it only exists as a paper in my drawer, what good does it do?”