Stat Testing: A (Too) Easy Crutch
Those of us who do a lot of survey research spend tons of time poring over statistics and reading data tables. And no matter what all the latest clever tools promise, there is no shortcut to reading page after page after page of data or tables or charts, and discerning the patterns or lack thereof.
BUT, researchers often resort to a handy trick: Look for those little asterisks or capital letters that flag statistically significant differences. Are there differences between men and women? Scan down the gender columns and look for flags. If you don’t see any, the answer is “no.”
Alas, there is huge problem with this handy trick! Scanning for statistically significant differences is an easy crutch that does not truly answer the question. Statistical significance does not tell us whether there are meaningful differences in our data. It only tells us whether the measured differences are “illusory” because of sampling.
I was reminded of this last week as I reviewed an employee experience survey data from which we had a full census of employees. A census means we surveyed 100% of the employees, and statistical significance is a measure of confidence when you don’t have access to 100%. As I searched the columns I thought, “Wait, are there no significant differences. Oh, right, we turned off testing because it makes no sense with a census.” And immediately I felt a little lost. How will I know whether group A is different from group B? The usual crutch was gone.
I was left making an informed decision based on the context of the project, the questions at hand, and my own experience about whether a difference of one or five or ten percentage points was enough to be meaningful. And truthfully, that’s exactly what I should always be doing when examining data and looking for differences.
With census data, every difference, even the smallest fraction of a percentage point, is “statistically significant.” So the question becomes: “Are the differences meaningful?” No statistics program can answer that question. Only you can.
