A clinical trial of hydroxychloroquine with 30 patients (15 treated 15 controls) has been completed in Shanghai. It is the first genuine randomized trial. It reports no evidence that hydroxychloroquine works at all.
It is true, that given the principal outcome measure defined in advance, the trial has no power. Not low power, 0 power. In a hypothetical, if all patients treated with hydroxychloroquine became healthy immediately with no symptoms and no detectable virus, then the report would be that there was not a statistically significant diference in the principal outcome measure for the treated and control subgroups.
The principal outcome measure was “can virus be detected 7 days after treatment starts”.
the answer was yes for 2 people in the treated group and 1 person in the control group.
Given that 14 out of 15 people in the control group had no detectable virus, the best outcome for hydroxychloroquine would have been 15 out of 15 in the treated group. Again a hypothetical, what if all the treated patients were assessed as cured after a week (best possible value of the principle outcome measure). This would reject the null that the probabilities were the same against the 1 sided alternative that treatment was better at the 50% level. It would reject the null against the two sideded alternative at the 100% level (not a typo).ù
So exactly zero power. Not low 0, zero, nada, niente.
With the benefit of hindsight, the researchers write that they could have designed the trial better. This does not mean that mistakes were made. When in a crisis, one has to act and must not make sure that one doesn’t do anything which is clearly suboptimal with the benefit of hindsight. That would imply sitting around thinking. They didn’t have time for that.
The secondary outcome measures provide statistically insignificant evidence that one is better without hydroxychloroquine. As noted by the authors, none of this evidence is strong enough to affect best practice of medicine (I still think that all patients without counterindications should be given hydroxychloroquine (I am not a doctor)).
At the clinical trials register, it is tagged “completed”, but the results are not yet uploaded (given the absolutely rigid standard format this takes the time of someone who is probably very busy).
The results are reported here.
googling for the link above, I found Hydroxychloroquine Is Ineffective In Treatment Of Patients Hospitalized With Covid-19, According To Small Controlled Trial From Shanghai
To that headline I say no No NOOOOOOO. Failure to reject the null is not a finding that the null is the truth. that would only be the case if all tests had power 100%. Since this test happens to have power 0%, the error is extreme. The error of rejecting the null is universal. It is a simple mistake – a failure to understand the Neyman Pearson framework.
Since I am a big fan of the alternative in this case, it is a delicate time to point out that the headline is simply incorrect. But it is.
Ugg, I hate that people are reporting failure to reject the null as rejecting the alternative. You are completely right to point that out.
It’s only sightly unfortunate that their main endpoint failed as a metric. It looks like there are a bunch of secondary endpoints as well.
I don’t see an english version, but plugging some sections into google translate does not look encouraging. Forgive me if I’m getting anything wrong due to translation.
Mean time to negative throat swab:
Mean corse of disease:
Lung image improvement after 3 days:
Control: 7 of 15
Treatment: 5 of 15
Okay, so the observed treatment group looks to have done worse on relevant metrics than control. Maybe there were some group differences. There were fewer (9) men in the treatment group vs control (12), which breaks in favor of treatment since men are at higher mortality risk. Treatment had more hypertensive cases (5 vs 3), and fewer COPD (0 vs 1). Ages were roughly comparable. So all in all, no obvious group differences that would favor control.
As you (and the authors) say, the patients had mild disease. We frankly don’t care too much about that other than what it can tell us about severe cases. What the world needs is something that reduces hospitalization times and mortality rates.
We need bigger faster randomized studies. Run them in the acute hospitalized/icu populations. Run them as sequential designs with daily statistics released publicly.