They say sample size matters. A handful of sample points in a study doesn’t tell you much, because they could just be showing random variation. This is also true not when you’re looking at many studies. You need to look at lots of research that uses different methodologies and data sets to get a confident feel for the facts on the ground.
Jared Bernstein points us to exactly such an effort, looking at 64 studies on the employment effects of minimum-wage increases, with a wonderfully informative display:
Note: “se” refers to standard error; 1/se is a measure of statistical significance. The dots up high are generally more believable.
“Employment elasticity” is a measure of the impact of minimum-wage increases. A measure of -.1 (left of the zero line) suggests that 10% MW increase reduces employment by 1%.
All the high-statistical-significance studies put elasticity at zero: no employment effect.
There’s some clustering to the left of the line versus the right down at the bottom, suggesting a small negative employment effect, but none of those studies has high statistical significance.
And this doesn’t consider “file-drawer/publication bias”: studies that find no effect don’t get published, because researchers don’t submit them or journals don’t accept them for publication. The CBO explains this in its new report on minimum-wage effects (PDF). Emphasis mine.
…an unexpectedly large number of studies report a negative effect on employment with a degree of precision just above conventional thresholds for publication. That would suggest that journals’ failure to publish studies finding weak effects of minimum-wage changes on employment may have led to a published literature skewed toward stronger effects.
And that doesn’t consider (pas possible!) negative-effect researchers finding ways to get to that publishable statistical-significance level. (It’s curious that those finding a positive effect don’t display this anomaly…)
So at least, you can mentally add a whole lot more unpublished dots to that tall vertical line. At most, you can shift a bunch of those published dots on the left farther to the right, and down.
Cross-posted at Asymptosis.