This is an update of this post in which I expressed immense confidence that welfare reform killed people in Florida .
The post is based on
During the 1990s reforms to the US welfare system introduced new time limits on people’s eligibility to receive public assistance. These limits were developed to encourage welfare recipients to seek employment. Little is known about how such social policy programs may have affected participants’ health. We explored whether the Florida Family Transition Program randomized trial, a welfare reform experiment, led to long-term changes in mortality among participants. The Florida program included a 24-36-month time limit for welfare participation, intensive job training, and placement assistance. We linked 3,224 participants from the experiment to 17-18 years of prospective mortality follow-up data and found that participants in the program experienced a 16 percent higher mortality rate than recipients of traditional welfare. If our results are generalizable to national welfare reform efforts, they raise questions about whether the cost savings associated with welfare reform justify the additional loss of life.
It’s not in the abstract, but they also analysed a larger data set and got a larger point estimate of 26% higher deaths due to participation in welfare reform.
The authors have since conceded that they unreasonably underesimated the standard errors of their point estimate. They used cluster robust standard errors with only 2 clusters (2 counties). This is not valid (the estimate of the variance of the point estimate of 16% more deaths is biased down). A reader noticed (as I should have) that the large difference between 16% and 26% would be extremely unlikely if the analysis had been correct.
using a reasonable fixed effects estimator (without the cluster robust consistent but biased down standard errors) they get
In the article we also presented combined results including participants in both Escambia and Alachua Counties, again controlling for year of birth, year of assignment, and site location and clustering the standard errors on location. The point estimate for that analysis is 1.26 (95 percent CI: 1.10, 1.45). Without clustering the standard errors around location, while controlling for location fixed effects as well as the other covariates, the new point estimate is 1.26 (95 percent CI: 0.96, 1.66).
So the more reasonably estimated stardard errors are roughly twice as large as the biased down ones. This means that the null of no effect (ratio of mortality rates =1) isn’t rejected at the 5% level. It is close. But the p-level of a t-statistic of a bit less than 4 is tiny (hugely significant).
The corrected standard errors imply evidence that welfare reform killed people, but not strong evidence. Hence the question mark in the updated title.
Like the authors, I apologize. I should have read the paper more carefully.
I thank Douglas Hess @douglasrhess for pointing out the published correction