Welfare Reform Kills ?
This is an update of this post in which I expressed immense confidence that welfare reform killed people in Florida .
The post is based on
https://www.ncbi.nlm.nih.gov/pubmed/23733981
Abstract
During the 1990s reforms to the US welfare system introduced new time limits on people’s eligibility to receive public assistance. These limits were developed to encourage welfare recipients to seek employment. Little is known about how such social policy programs may have affected participants’ health. We explored whether the Florida Family Transition Program randomized trial, a welfare reform experiment, led to long-term changes in mortality among participants. The Florida program included a 24-36-month time limit for welfare participation, intensive job training, and placement assistance. We linked 3,224 participants from the experiment to 17-18 years of prospective mortality follow-up data and found that participants in the program experienced a 16 percent higher mortality rate than recipients of traditional welfare. If our results are generalizable to national welfare reform efforts, they raise questions about whether the cost savings associated with welfare reform justify the additional loss of life.
It’s not in the abstract, but they also analysed a larger data set and got a larger point estimate of 26% higher deaths due to participation in welfare reform.
The authors have since conceded that they unreasonably underesimated the standard errors of their point estimate. They used cluster robust standard errors with only 2 clusters (2 counties). This is not valid (the estimate of the variance of the point estimate of 16% more deaths is biased down). A reader noticed (as I should have) that the large difference between 16% and 26% would be extremely unlikely if the analysis had been correct.
using a reasonable fixed effects estimator (without the cluster robust consistent but biased down standard errors) they get
In the article we also presented combined results including participants in both Escambia and Alachua Counties, again controlling for year of birth, year of assignment, and site location and clustering the standard errors on location. The point estimate for that analysis is 1.26 (95 percent CI: 1.10, 1.45). Without clustering the standard errors around location, while controlling for location fixed effects as well as the other covariates, the new point estimate is 1.26 (95 percent CI: 0.96, 1.66).
So the more reasonably estimated stardard errors are roughly twice as large as the biased down ones. This means that the null of no effect (ratio of mortality rates =1) isn’t rejected at the 5% level. It is close. But the p-level of a t-statistic of a bit less than 4 is tiny (hugely significant).
The corrected standard errors imply evidence that welfare reform killed people, but not strong evidence. Hence the question mark in the updated title.
Like the authors, I apologize. I should have read the paper more carefully.
I thank Douglas Hess @douglasrhess for pointing out the published correction
The results of the analysis, as corrected (or not) hinge on the applicability of US data to the Florida Counties.
“To estimate the average number of life-years lost in the experimental group, we generated an unabridged life table of the US population using data from the Centers for Disease Control and Prevention. We adjusted this table to reflect the increase in mortality for people in the experimental group compared to those in the control group and subtracted the difference before and after adjustment. This process is described in detail elsewhere.[note 21].
The authors do not show the process “detailed elsewhere” or cite any evidence of the degree to which this process has been subjected to critical review.
I cannot help but believe that the US population data CANNOT BE applied on a county by county basis .. .or even a state by state basis without detailed mortality actuarial tables for each county in the time frame of the experimental results as a function of income.
Thus it seems to me that the foundation of the experimental study is missing the primary aspect upon which the comparative mortalities between the experimental and control group are based.
Sorry… I omitted the link to the original paper and the excerpted statement from it.
http://content.healthaffairs.org/content/32/6/1072.full#ref-21
under the heading “Statistical Analysis”
I should explain why this omission bothers me.
I spent a career running experiments large and small in advanced R&D and in multiple mfg’ing locations. In any experimental result based on a statistical inference there are always two sides… a group that will benefit by accepting the results and a group that will not benefit.
Thus experimental results will always be subject to critical questions by one or the other group. Because of this, the results must necessarily be able to be clearly and unequivocally robust enough to withstand even the most ardent adversaries.
It was my experience when designing an experiment or reviewing other experimental designs, the simplicity of assumptions which would have a material effect on the conclusions of experimental results …. especially the validity of confidence intervals.
Assumptions always minimize the effort so are prone to overuse and lack of scrutiny by whomever is conducting or funding the experimental effort..
I cannot count how often I decided, a-priori, to check an assumption.. my own or from prior work by others, with a separate evaluation and most often a separate experiment or set of experiments, finding far more often than not that the assumptions were invalid or significantly different than experimental results to confirm or not the assumption.
This always required more effort, more costs, and longer elapsed time than I or other experimental design preferred to spend, and more often than not I had to go to great lengths to convince others of the necessity of running the pre-experiments to determine whether the assumptions in the primary experiment were valid or applicable.
When it was the case that I was unable to obtain the support for running the pre-experiments or collect the data necessary to evaluate the applicability of the assumptions, I was at least able to show that there was little or even no value in running the primary experiment at all since the results could not be used to make a decision (involving millions of dollars) with any more confidence than a coin toss.
In other words, why spend time and effort to run an experiment that no matter what the results they were no better than a coin flip based on assumptions that were equally unknown. What such experiments do is give cover to the proponents of making some change in which the assumption used will insure the results of the experiment will always come out favoring the change. In other words, a politically necessary “experiment” which has no objective reality foundation at all.