Medicaid Austin Frakt, Aaron Carroll and Kevin Drum are Good for the USA
by Robert Waldmann
An important study of the effect of Medicaid on health was published in the New England Journal of Medicine. The study was based on a genuine experiment where some people were given Medicaid and other people weren’t based on a lottery. Unfortunately, the results were communicated with a NEJM press release and not just the published article. The results as received by the press is that Medicaid did not cause significant effect on recipients’ health (except for significantly lower depression) which was interpreted as the study providing evidence that Medicaid does not improve health.
This means that somehow someone rejected the alternative.
Hero bloggers Austin Frakt and Aaron Carroll try to get valid use of statistics boots on before the error runs around the world. Read their important post. Then read Kevin Drum’s important post where he links to their important post and also presents, you know, the data people are arguing about. The damage is done and can’t be fully undone, but I hope that this will be a case in which the medium of blogging undoes some of the damage due to publication by press release. The actual authors in the actual article explain the issues very well (read the quote in the Frakt and Carroll post). The NEJM makes only an abstract of the article available for free to non journalists such as us.
In fact the raw data show better average health in the Medicaid recipient sub group compared to the control group. The estimated benefits are not large enough to be STATISTICALLY significant, because the number of people diagnosed with diabetes and hypercholesterolemia were low. Arithmetic tells me that 308 or 309 people were diagnosed with diabetes in the Medicaid group and 64 people were diagnosed with diabetes in the control group. Note this effect difference is statisically significant at all confidence levels ever used in the history if statistics (p < 0.1%). The question not answered by the study is whether current standard of care treatment of diabetes has any beneficial effects. This was not and is not an open question.
It isn’t really surprising that the effects of proposing treatment to 308 or 309 out of 6387 in the sample of people offered a chance to apply for Medicaid rather than 64 of 5842 people in the control sample does not cause statistically significant improvement in outcomes after 2 years averaged over all people whether diagnosed or not.
The sample of people needing care which they might or might not receive was very small. This means that, by itself, the experiment doesn’t contain enough evidence to cause the FDA to approve sale of statins or insulin. But no one sensible person would question current medical practice based on a new study which provided some evidence that current treatments work, but not enough to constitute proof if one ignores all the other data.
The probability of diagnosis of diabetes was vastly greater for the Medicaid group. After reading the data (I read them in Drum’s post) one can only conclude that Medicaid fails to improve health if one believes either that current treatment of diabetes is ineffective (in spite of massive evidence from other studies that such treatment improves health) or that the diagnoses in the study were incorrect (but the same tests are valid for assessing health) or that there is some offsetting health cost of Medicaid (maybe moral hazard as people who love needles eat sugar so they can get diabetes which they know will be diagnosed so maybe they can take insulin which is such fun). Any reasonable person looking at the raw data would conclude that Medicaid improves health.
The problem here is a combination of the first do no harm standard in the academic literature which mandates very cautious conservative claims combined with the gross missunderstanding of interpreting “statistically insignificant” to mean “nonexistent” or “small” or “purple” or “round” or well something other than “statistically insignficant”.
As a person diagnosed with Type II diabetes I am puzzled about why diagnosis (most diabetes diagnoses are Type II) does not lead to significant drop in blood sugar. In fact that outcome is so unlikely that it should lead to an immediate investigation itself — before we can make anything of the effect of expanded Medicaid.
Medicaid — I have Medicaid too but also Medicare — has so little acceptance from normal medical practices that all it accomplishes — which is terrific! — is allow indigent patients — I’ve been there — to attend the country hospital clinics or go to the ER and not get huge bills they cannot pay — which catastrophically impact on their credit ratings (if they have any) and thereby job and housing opportunities. Medicaid basically asks already squeezed doctors to take pennies on the dollar — can’t do it.
A big question here — thinking about the unthinkable — is if Medicaid patients don’t get much lower level of care than normal in Oregon — has to be asked. Also unthinkable: are some (not most!) Medicaid patients in Oregon so dysfunctional that they don’t utilize care effectively — possibly same dysfunction that has some on Medicaid in the first place?
First and foremost question that needs to be answered: what could possibly lead to no effect on blood sugar among 600+ diagnosed patients?
One finding, of the d’oh! variety but still important, was that bankruptcy as a result of health costs in the Medicaid group dropped to negligible levels.
This may or may not affect the recipients directly (I think it would, but set that aside.) but it would directly affect the halo of friends, relatives, community social programs, and so on, onto whose shoulders would fall the burden of supporting the bankrupt person, depress the housing and auto markets indirectly by putting those assets on the market prematurely, and all the other effects of bankruptcy.
Interesting Google Scholar search on “health consequences of bankruptcy” here: http://scholar.google.ca/scholar?as_ylo=2009&q=health+consequences+of+bankruptcy&hl=en&as_sdt=0,5&as_vis=1 , articles posted after 2009.
Denis there was a decline in blood sugar in the people offered Medicaid. The point estimate has the expected sign.
Before going into the boring statistics discussion again, I want to note that there is a first person explanation of the problem. One of the actual participants commented on Brad DeLong’s blog explaining how it took him a while to adapt to having insurance. He also said his doctor told him he adapted quicker than most. I stole his comment and posted it here
It’s a very clear explanation of why 2 years is early to measure blood cholesterol and hemaglobin glycosylation even though it is plenty of time for the effects to show up *after* regular treatment of the conditions begins. Now as he notes, this guy is one of the success stories, but he explains basically how effective care for him started about 6 months later than the article authors guessed it would.
The study does show that Medicaid enrollees are getting a lot more care (the diagnoses are one sign but also number of office visits and stuff).
OK boring statistics stuff.
The failure to reject the null of no effect probably occured because the estimate was extremely imprecise. This is partly due to the fact that the authors used winning the lottery and being eligible for Medicaid as an instrument. Only about 26% of the lottery winners actually signed up for Medicaid. So the estimate is look at the difference between the rate of diagnosis among lottery winners vs lottery losers then divide by 0.26 assuming the lottery only mattered via Medicaid (the only thing it could affect). this is the right thing to do, but it is costly.
Basically it is very possible that the null hypothesis of no effect was not rejected even though there is a large effect. The confidence intervals of the estimated effects clearly show this (as was noted by the authors in the article).
Now 2 years should have been long enough for improvement to be large enough that the null would be rejected in spite of the low precision of the estimate. I personally don’t know when the participants signed up for Medicaid and when their diabetes was diagnosed (it might be in the article but I have only read parts of the article the abstract, table 2 and the bit I just paraphrased above — the article is behind a paywall).
I get Drum’s point about stat language — add to that how little people with insurance take care of themselves and you probably have a better comparison.
Some of Frakt and Carrol is confusing: “although we did not find a significant change in glycated hemoglobin levels, the point estimate of the decrease we observed is consistent with that which would be expected on the basis of our estimated increase in the use of medication for diabetes.” Does this mean that “we” did not expect a significant change with the use of medication? ??
“The percent of people with diabetes with a high A1C went from 5.1% off Medicaid to 4.2%.” How is that consistent with no significant change in blood sugar.
Blood sugar levels change at once with medication, BTW.
I think the best endorsement for the program may be your patient who lost 50 pounds with regular medical advice — can see that sort of thing permeating through many lives.
His loss may even have been due to medication. I lost 50 pounds in 50 weeks with Exenatide (first as Byetta twice daily and now with Bydureon once a week).
Probably a great program; I just wish someone would break out blood sugar stats across a spectrum of patients now that the worry has been raised.
two things are not clear to me.
first, why was the incidence of diabetes so much lower in the control group?
second, i don’t see any statement that the people not given medicaid did not get treatment from some other source.