Is something rotten in brain stimulation research?

quick takes

Rotten-Apple-Hard-Carbon-Sodium-Ion-BatteriesWhat other area of neuroscience are vulnerable to these possible problems?

Or areas of psychology dominated by underpowered studies with considerable investigator flexibility in the conduct of the study and the reporting of results?

 

This post is a Quick Take on a fascinating blog post by Neurocopaie that I just read. I was motivated to pursue some of the pseudonymous blogger’s links into the open access articles that were referenced. You may be too. I also highly recommend subscribing to this blogger, as I do.

The original blog post cites some articles about transcranial direct current stimulation (tDCS). The results of these articles can only be viewed as suggestive, not definitive in identifying what could be pervasive, serious problems in the tDCS literature. But taken together, these articles could be indicative of something rotten going on in this literature that is also going on in other neuroscience literatures. But why stop there? I think there is food for thought here worthy of further discussion and broader testing of other areas of research.

Who or what is Neurocopaie?

Neurocopiae is about the science and art of today’s fMRI research. Content is captured through the lens of a junior faculty scientist before it is heavily preprocessed so that everyone can enjoy the beauty of the blobs. Opinions expressed @neurocopiae tend to be at the upper end of the reproducibility scale.

The post.

Amping up control? Bad research practices and poor reliability raise concerns about brain stimulation

It hasn’t been a very good week for proponents of the popular brain stimulation method called transcranial direct current stimulation (tDCS). tDCS is a non-invasive technique that uses electrodes to deliver weak current to a person’s forehead. Numerous papers have claimed that tDCS can enhance mood, alleviate pain, or improve cognitive function. Such reports have sparked interest in tDCS at a broader scale. When you enter tDCS in the youtube search, you will find DIY tutorials on how to assemble a device so that you can amp up your brain at home. Including enthusiastic reports of the resulting changes in brain function. To put it in Richard Dawkins’ words: Science? It works, bitches. In particular, it works when you know what the outcome should be.

Great quote from a linked Science news piece in the blog.

Traced back to Science, it is even pithier in context:

The tDCS field is “a sea of bullshit and bad science—and I say that as someone who has contributed some of the papers that have put gas in the tDCS tank,” says neuroscientist Vincent Walsh of University College London. “It really needs to be put under scrutiny like this.”

The PLOS ONE article that is cited.

Questionable science and reproducibility in electrical brain stimulation research

We invited 976 researchers to complete an online survey. We also audited 100 randomly-selected published EBS papers. A total of 154 researchers completed the survey. Survey respondents had a median of 3 [1 to 6, IQR] published EBS papers (1180 total) and 2 [1 to 3] unpublished ones (380 total). With anodal and cathodal EBS, the two most widely used techniques, 45–50% of researchers reported being able to routinely reproduce published results. When asked about how study sample size was determined, 69% of respondents reported using the sample size of published studies, while 61% had used power calculations, and 32% had based their decision on pilot data. In contrast, our audit found only 6 papers where power calculations were used and a single paper in which pilot data was used. When asked about questionable research practices, survey respondents were aware of other researchers who selectively reported study outcomes (41%) and experimental conditions (36%), adjusted statistical analysis to optimise results (43%), and engaged in other shady practices (20%). Fewer respondents admitted to engaging in these practices themselves, although 25% admitted to adjusting statistical analysis to optimize results. There was strong agreement that such practices should be reported in research papers; however, our audit found only two such admissions. The present survey confirms that questionable research practices and poor reproducibility are present in EBS studies. The belief that EBS is effective needs to be replaced by a more rigorous approach so that reproducible brain stimulation methods can be devised and applied.

The NeuroImage study.

Test-retest reliability of prefrontal transcranial Direct Current Stimulation (tDCS) effects on functional MRI connectivity in healthy subjects

This is the first study investigating the test-retest reliability of prefrontal tDCS-induced resting-state functional-connectivity (RS fcMRI) modulations.

Analyses of individual RS-fcMRI responses to active tDCS across three single sessions revealed no to low reliability, whereas reliability of RS-fcMRI baselines and RS-fcMRI responses to sham tDCS was low to moderate.

Back to the Neurocopiae blog post.

Amping up control? Bad research practices and poor reliability raise concerns about brain stimulation

But how often do we see comprehensive assessments of the reliability of new techniques in human neuroscience at all? These studies are commonly seen as less interesting and they won’t help you get a paper in a fancy journal unless you trash the whole field. If you only trash the reliability of your newly developed paradigm or method, it might be difficult to claim that the effects somewhere around the edge of the significance threshold truly warrant any enthusiastic conclusions that attract future citations.

Still, in differential psychology, everyone would expect you to conduct and report such basic assessments first before you make any strong claim about individual differences. My take-home from the tDCS story is that basic checks of the reliability of every method in neuroscience should always come first and that they should be reported as well. They are important. Now we have kids building tDCS devices based on youtube tutorials and we know very little about what it does and what it does not do reliably. The call for more focus on reliability of outcomes might seem trivial, but think of your favorite fMRI studies and how often you have seen a discussion of the reliability of the method itself. It’s not a lot.

eBook_Mindfulness_345x550I will soon be offering e-books providing skeptical looks at mindfulness and positive psychology, as well as scientific writing courses on the web as I have been doing face-to-face for almost a decade.

Sign up at my new website to get advance notice of the forthcoming e-books and web courses, as well as upcoming blog posts at this and other blog sites.  Get advance notice of forthcoming e-books and web courses. Lots to see at CoyneoftheRealm.com.