In this multi-part blog series, we will cover various topics relevant to the Federal Contractor community as it relates to statistical analyses. We begin our series with a topic that is often misunderstood by both practitioners and enforcement agencies.
This past year we heard some crazy interpretations regarding the meaning of a statistically significant standard deviation test. Our favorite was:
“A significant standard deviation means that there is a 95% chance that the difference is due to discrimination.”
As a result of such statements, we thought this might be a good time to briefly remind everyone about the meaning of the term, “statistically significant.” The idea behind statistical significance testing is that anytime you gather a sample of data (e.g., hiring rates for a janitorial position, average salary for accountants) and compare two groups (e.g., men and women), due to chance alone, it is extremely unlikely that the difference between the two groups is going to be exactly zero. For example, for a particular hiring requisition we hired 76% of White applicants but “only” 70% of Hispanic applicants. Is this 6% difference the result of chance or something more sinister? What would the numbers look like across ten requisitions?
When a difference between two groups is statistically significant (e.g., the difference in selection rates is greater than two standard deviations), it simply means that we don’t think the observed difference is due to chance. The greater the number of standard deviations, the less likely we are to believe the difference is due to chance. Some things to keep in mind:
- Because a standard deviation test is greatly affected by sample size, the number of standard deviations doesn’t say anything about the size of the group difference. For example, with 10,000 job applicants, a 1% difference in selection rates (e.g., 90% v. 89%) would exceed two standard deviations; however, a 20% difference with 40 applicants (e.g., 80% v. 60%) would not.
- A group difference that is flagged as being statistically significant using a standard deviation test may still have occurred by chance. If we ran 100 adverse impact analyses, we would expect five to be statistically significant by chance alone!
- A statistically significant standard deviation doesn’t imply discrimination: It simply provides some confidence that something might be going on and that we should explore the difference further.
Stay tuned for additional blogs. Also, please read our white paper to learn more about the different statistical significance tests used to analyze data for the purpose of identifying disparate impact (adverse impact) which is different from disparate treatment.
By Mike Aamodt, Principal Consultant and Yesenia Avila, Associate Consultant at DCI Consulting Group
