If alternative medicine wants to be taken more seriously, the studies must be better designed and be put in the proper context.

UK’s The Telegraph reported last month that a study published in the journal Complimentary Therapies in Clinical Practice showed that reflexology was “as effective as pain killers.” It’s a bold claim.

However, this claim is backed up by nothing in the study. In fact, all the methodological flaws encourage a reflexive rejection of the study’s conclusions.

No Control, No Power

You don’t have to be a scientist to know what questions to ask about a study. Some of the most basic are “What was the sample size?” and “Was it double-blinded?” Even these basic questions can tell you a lot about what researchers find.

The reflexology study had a sample of 15 participants, most of them women, and each received both experimental conditions (we will come back to this point later on). If 15 sounds like a small number to you, that’s because it is. In fact, because the statistical analyses they were using looked at group averages, this small number gets broken down even further. With so few participants, this study does not have the power to comment on very much. In larger studies, vexing variations between individuals “cancel out” to hit on some average value. Whether this study hit on something interesting or not, we wouldn’t be able to tell—values are lost in the large variations between so few people.

 

To control for possible placebo effects, the researchers used transcutaneous electrical nerve stimulation (TENS) as the “sugar pill” comparison to reflexology. But the famous “sugar pill” experimental design comes from the idea that subjects should not be able to tell the difference between one pill and another. In this study, every subject could easily tell the difference between a massage of the foot and some electrodes placed on the wrist. And this brings in other problems. Because each subject, and each researcher, knew what treatments were given, there was effectively no blinding. Blinding is the best way to avoid the pernicious biases that tend to creep into studies like this. Needless to say, an unblinded study is far less persuasive.

And what of the TENS treatment that was supposed to act as a placebo? One systematic review concluded that there is “no benefit of TENS compared with placebo.” Another review found that “evidence for the efficacy of…is limited and inconsistent,” in regards to treating chronic back pain. The New England Journal of Medicine concluded that “treatment with TENS is no more effective than treatment with a placebo, and TENS adds no apparent benefit to that of exercise alone,” also referring to treating chronic back pain.

So, according to much larger studies, there is no reason to believe that TENS does much for pain. TENS could then effectively be a placebo, but the authors of the reflexology study mixed and mangled this fact by using a “fake” TENS treatment as a control for the reflexology treatment. What does this all mean? It means that there was no actual control group (a group who didn’t receive any treatment) and the placebo control for the reflexology treatment made no sense.

But the study did indeed find something. The researchers used an ice-bowl test—where participants held their hands under the water until they could no longer—to determine if there were any changes in pain tolerance or pain threshold. After receiving reflexology or the fake TENS, the subjects plunged their hands into a bowl of water filled with ice until they could no longer hold their hands in. The subjects would then take a 30-minute rest (during which time they were, perplexingly, allowed to talk with the researcher) before doing the test another four times.

After doing some statistical analyses, the study concluded that there was a significant difference between the average pain threshold (the time it took for a subject to feel some pain) for subjects during one of the tests, and a significant difference in pain tolerance (how long the subjects could hold their hands in the icy water) during three of the tests. Looking at the numbers, this amounts to a difference of 4-5 seconds in pain threshold and 25-75 seconds in pain tolerance from the baseline measurements.

Both of these measurements sound impressive, until you remember the problems we have already gone through. With only 15 people, it’s exceptionally hard to sort the signal from the noise. Statistically significant differences could be all noise; there just aren’t enough data to know. Combine this with the fact that none of the tests were blinded, there was no control group (a group that did the ice test but had no TENS or reflexology), and that simply yelling out obscenities provides greater pain threshold and tolerance in ice tests; you get a weak study indeed.

And the problems continue. The same subjects received both treatments, one week apart, in this study. This invites numerous variables in to mess up a study that larger experiments try to control for (i.e., separate groups of people per treatment). For the pain tolerance tests, the supposed placebo TENS test significantly decreased tolerance. If it were truly a placebo test, you wouldn’t expect such an effect. These problems echo mainstream medicine’s main critique of alternative medicine—the studies just aren’t up to scientific snuff.

Of course, this whole critique overlooked the obvious fact—reflexology does not work. Three different systematic reviews [PDF] of the studies on reflexology all come to the same conclusion: “The best evidence available to date does not demonstrate convincingly that reflexology is an effective treatment for any medical condition.” Most studies of “complimentary” therapies have the same problem. Statistics can find interesting connections, but those connections only mean something if there is some reason to believe that the connection is real. There is no plausible biological or physiological reason to think that this map shows us where to press on the foot to help a person’s spleen.

Unless you are okay with saying that blurted obscenities and placeboes are “as effective as painkillers,” reflexology doesn’t have anything on oxycodone. Or sugar pills.

Kyle Hill is a JREF research fellow and popular science writer who contributes to Scientific American, Wired, io9, and Popular Science. He writes daily at the Science-Based Life blog and you can follow him on Twitter here.