We have written several blogs (e.g., regarding the recent Apsley v. Boeing case) discussing the influence of sample size on the likelihood of observing statistically significant indicators in adverse impact analyses. As covered in those blogs, the larger the samples being analyzed, the higher the likelihood of observing statistically significant indicators of protected class subgroup differences, regardless of how different the pass rates are in practical terms. However, even in cases of small sample sizes, the differences between protected class subgroups may be statistically significant, and it is equally important (as it is with large samples) to evaluate the practical significance of the difference. In the following paragraphs, we discuss interpreting analysis results based on small pools or small numbers of selections.
In any case in which statistically significant indicators between protected class subgroups are found, it is critical to consider the practical significance of the difference. A common measure of practical significance is the shortfall, which is defined as the difference between expected selections and actual selections. The shortfall is calculated by (1) determining the number of selections that would have been expected from a group given the overall selection rate and (2) subtracting the number of actual selections from that group. This is often used in the EEO realm as an indicator of the magnitude of the adverse impact effect.
As an example of the importance of evaluating the magnitude of the shortfall when sample sizes are small, but selection rate differences are statistically significant, refer to the example data below:
Given an overall selection rate of 20%, it is expected that 2 females would have been selected from the 10 that applied. Only 1 was actually selected, yielding a shortfall of one female. The question here becomes: With a relatively small applicant pool and few selections made, does the shortfall of one female indicate a true problem of selection rate differences?
There have been a variety of uses of the shortfall utilized in the courts and by EEO enforcement agencies. The “flip-flop” standard has been endorsed by the courts and addressed by the Uniform Guidelines on Employee Selection Procedures (UGESP) as a viable assessment of whether small changes to underlying numbers (the selection decision data) would change the analysis results significantly. The UGESP Question and Answer (#21) illustrates this standard in saying:
Generally, it is inappropriate to require validity evidence or to take enforcement action where the number of persons and the difference in selection rates are so small that the selection of one different person for one job would shift the result from adverse impact against one group to a situation in which that group has a higher selection rate than the other group.
To demonstrate from our example above, the hiring of one more female and one fewer male would have resulted in a 0% difference in selection rates. Recently, a different interpretation of the shortfall figure was presented in the case of Apsley v. Boeing Co. (2012). The court determined in this case that if the shortfall was a low percentage of the overall number of selections, the analysis results were not deemed robust enough to support a claim of adverse impact. Use of the shortfall in this manner allows for taking the pool size into account, which may warrant wider use in future court decisions. In the above case, the shortfall is 20% of the total selections made.
With the backing of UGESP and case law precedent, contractors are encouraged to interpret adverse impact analysis results of small pools and small numbers of selections with additional practical significance measures.
By Jana Garman, Consultant and Kayo Sady, Senior Consultant at DCI Consulting Group
