Not even bad science

Once in a while, one comes across a published study which is so bad that one cannot even consider it to be bad science, rather it's so completely wrong, that it has nothing to do with science at all.

I recently came across one such study.

For some strange reason, I had clicked on a homeopathy hashtag in Twitter, showing me the latest tweets about homeopathy. This is not something I recommend, as the stupidity is out in force there. Anyway, one of the most recent tweets at the time referred to a study demonstrating that homeopathy was as efficient as anti-depressants. Unsurprisingly, I took a closer look at the study.

The study is this one: Homeopathic Individualized Q-potencies versus Fluoxetine for Moderate to Severe Depression: Double-blind, Randomized Non-inferiority Trial by U. C. Adler et al. (read it at your own risk)

My last post was triggered by reading the study for reasons I will come into later, but let me just say for now that whoever was in charge of that study was obviously a true believer of homeopathy, and had very little grasps of science and evaluating results.

What makes me say that? Well, let's tackle the claim about the person in charge being a true believer and not understanding science. Passages like this one, should clearly demonstrates why I say that.

Hahnemann's dynamization gained support of physics: thermoluminescence emitted by ‘ultra-high dilutions’ (dynamizations) of lithium chloride and sodium chloride was specific of the salts initially dissolved, despite their dilution beyond the Avogadro number (11).

I think I can safely say that no physicist would agree that Hahnemann's idea of dynamization has gained the support of physics - as a matter of fact, physicists would call Hahnemann's claims pure nonsense.

Don't believe me? Well, this is what Hahnemann had to say on the subject of dynamization:

This remarkable transformation of the properties of natural bodies through the mechanical action of trituration and succussion on their particles (while these particles are diffused in an inert dry or liquid substance) develops the latent dynamic powers previously imperceptible and as it were lying hidden asleep in them. These powers electively affect the vital principle of animal life. This process is called dynamization or potentization (development of medicinal power), and it creates what we call dynamizations or potencies of different degrees.

In other words, this is the claim that diluting the substance makes it more potent.

The claim that physics supports this is based upon one article, Thermoluminescence ofultra-high dilutions of lithium chloride and sodium chloride (.pdf) by Louis Rey, which hasn't been replicated, and which certainly doesn't support Hahnemann's claims about the substance becoming more potent. There are also several people who points out problems with Rey's study.

So, all in all, a good scientist would most certainly not make claims like "Hahnemann's dynamization gained support of physics", since there is only one, non-replicated study which might support this claim - in terms of science, this amounts to an unsupported claim.

Still, even though the people in charge of the study are true believers, and don't really understand science, it doesn't mean that the study can't be useful - after all, if done properly, the results should speak for themselves. So, let's return to the study.

Basically, the study was conducted by assigning patients into two groups - one group which would receive Fluoxetine and a placebo, and one group which would receive the homeopathic remedy and a placebo.

I think you all can see the issue there. Since homeopathy is placebo, what this study is doing, is comparing Fluoxetine to placebo. Of course, the people conducting the study doesn't see it this way, but as I wrote in my last post: Until homeopaths can explain how homeopathy works in terms which doesn't mean that everything we know about chemistry, physics, and physiology is wrong, then we can safely reject their claims.

But let's for the sake of the argument accept that we are comparing two different types of remedies for depression, and look at what the study concludes.

This sample consisted of patients with moderate to severe depression, because their mean MADRS depression scores were close to the 31 score cut-off for moderate and severe depression (28). Initially, 284 subjects were screened, 105 of them met the inclusion criteria, 14 out of them did not attend the first appointment, 91 were randomized and 55 completed the 8-week trial. A detailed flow chart of subject progress through the study is shown in Fig. 1.

So, out of 91 people, only 55 completed the 8-week trial. That's a drop-out rate of ~40%. That's a quite significant number, and would impact the reliability of the study. Let's see how they go into this later.

There were no significant differences between the proportions of excluded and lost for follow-up patients in the two groups (P = 0.99), though there was a trend toward greater treatment interruption for adverse effects in the fluoxetine group, as can be seen in Table 1.

Well, true, there was no significant difference between the proportions, but there were quite different reasons why people were excluded.

As they state, there was "a trend toward greater treatment interruption for adverse effects in the fluoxetine group", hardly surprising since they were given actual medicine, rather than placebo, which has side-effects. What's more surprising is that there were actually 3 people excluded from the placebo homeopathy group for adverse effects. Since there are no active ingredients in the homeopathic remedies, this must either be due to an negative placebo effect (called the nocebo effect), or due to a misdiagnosis of e.g. clinical worsening.

What they did leave out was the fact that there is a significant difference in the clinical worsening in the two groups. Among people receiving the medicine, there was one person excluded because of worsening (approximately 2.3% of the cohort), while among the receivers of homeopathic remedies, there were five who were excluded for this reason (more than 10% of the cohort, 10.4% to be more precise). Again, this is entirely in line with our knowledge of which group is receiving actual medicine, and it certainly is a significant enough difference for it to be taken into consideration when writing about the results, yet this was not done.

The rest of the result section goes on to analyzing the results of the people who made it through the 8 week period, yet doesn't address neither the high drop-out rate nor the differences in the reasons for exclusions.

Going to the discussion section, there are a couple of things which jumps out, particularly this paragraph:

A placebo-arm was not included in the present study because it was not authorized by the National Ethic Council. Although placebo interventions are associated with mean response or remission rates of ~35% (37,38), a placebo effect cannot be ruled out, since the homeopathic Q-potencies were compared with an antidepressant and ‘it is becoming more and more difficult to prove that antidepressants—even well-established antidepressants—actually work better than placebo in clinical trials’ (39). Nevertheless, it also has to be taken into consideration that the antidepressant-placebo difference seems to be smaller in the trials aiming at mild to moderate depression (40,41) and the present sample consisted of patients suffering from moderate to severe depression. Placebo-controlled studies would be recommendable to clarify these findings.

The first part of the paragraph is what caused the ranting in my last post. I presume that the National Ethic Council didn't allow a placebo-arm in the study, because it considered it unethical to submit people suffering from depression to placebo, yet by allowing this homeopathy study to go ahead, it submitted the very same type of people to something which we know is placebo. What the hell is wrong with these people? Are they really so gullible that they don't realize this?

And when reading the rest of the paragraph, it becomes clear that the people who conducted the study is aware that they were comparing homeopathic remedies to something which hasn't been shown to be any better than placebo. In other words, they are comparing one type of placebo, homeopathy, to something which might very well be another type of placebo, or which at the very least seems to have a very limited effect on top of the placebo effect. And we are supposed to be impressed that homeopathy is as effective?