Art Gutman recently summarized the ruling in Lopez v. City of Lawrence, a police promotion disparate impact case. In this case, a district court judge ruled in favor of the employer along multiple dimensions, and rejected aggregation of test-taker data across multiple municipalities and years. Disregarding several violations of the 4/5 rule due to insufficient sample size, the judge focused more on statistical significance tests, which showed no evidence of adverse impact. Even though combining data to achieve a larger sample may have produced a different result, the judge rejected aggregation both across municipalities (questioning the similarity of the applicant pools) and across years (questioning the possibility of duplicate records).
The issue of data aggregation is one that may be very important to federal contractors conducting adverse impact statistics across different units (e.g., job, location) and time periods. Below we describe two ways in which data aggregation decisions can affect results of contractor adverse impact analyses. Aggregating data from multiple groups is generally reasonable when the groups are “similarly-situated,” meaning the individuals and the circumstances surrounding the selection process are reasonably similar. However, combining data from dissimilar groups can lead to what is known as “Simpson’s Paradox.” When groups are dissimilar (e.g., qualification level among applicants, state of the economy among years), combining data under a single analysis can lead to the opposite conclusion when compared to conclusions from separate group analyses. This result could include scenarios where statistically significant disparities are masked by aggregation, and other scenarios where statistically significant disparities are inflated by aggregation.
For example, data from multiple contractor facilities could be combined into one table, resulting in statistically significant adverse impact against women compared to men. However, when analyses are conducted separately by facility, there may be no statistically significant adverse impact against anyone, and in fact females may be the higher selected group at some facilities. Such a finding would exemplify Simpson’s paradox, and in such cases considerable thought is needed to understand whether a “by facility” approach is more reasonable than an aggregate approach.
Data aggregation can also lead to trivial conclusions based on statistical significance tests when very large sample sizes are involved. When analyses involve groups that are extremely large, statistical significance will be triggered regardless of selection rates. As an example, even a 1% difference in selection rates will be statistically significant in very large pools. As such, federal contractors will likely find statistically significant results when pools include thousands of applicants. When data are aggregated across time or level, pools become larger and the results of adverse impact analyses are more likely to become statistically significant.
It is important for contractors to think carefully about how to best group individuals when analyzing personnel processes. Analyses should consist of groups of similarly-situated employees (or applicants) and should be reflective of the contractor’s selection process. For example, a contractor who uses well defined applicant requisitions may find that analyzing applicant data at the requisition level best mirrors reality. When submitting analyses for a compliance evaluation, contractors should use the group structure that best captures the contractor’s selection process. Contractors should not attempt to mask adverse impact through analysis groups. The goal is to identify potential barriers to equal employment for protected groups. Contractors should strive to create groups that best mirror the selection process. Additionally, for the reasons described above, contractors must be prepared to speak up when inappropriate data aggregation is used to make adverse impact claims during an audit.
By Rachel Gabbard, M.A.,HR Analyst and Eric Dunleavy, Ph.D., Principal Consultant, DCI Consulting Group
