In keeping with their focus on transparency, last week OFCCP released two FAQ documents that should be reviewed by all federal contractors. The first focuses on practical significance in the context of adverse impact measurement, which is one approach to evaluating disparities in employment outcomes. The second focuses on validating employee selection procedures, a topic of particular interest to Industrial-Organizational (I/O) Psychologists.
With regard to practical significance, OFCCP describes the notion as “a conceptual framework for evaluating discrimination cases developed primarily on statistical evidence that is the subject of increasing interest and discussion by some in the equal employment opportunity (EEO) field.” The guidance compares and contrasts this approach to statistical significance testing, provides some example measures of practical significance, and summarizes what the Uniform Guidelines on Employee Selection Procedures (1978) have to say on the topic. This is a concept that DCI’s Director of Personnel Selection and Litigation Support Services, Dr. Eric Dunleavy, has thought and wrote about often over the last 15 or so years. DCI was pleased to see some of the collaborative work he has done with Drs. Scott Morris and Fred Oswald1 cited by the agency and recommended as worth reading for those that are interested in learning more on the topic.
Perhaps most importantly, the agency notes that “OFCCP considers practical significance along with statistical significance and all other evidence gathered in the course of the investigation.” This consideration of both statistical and practical significance is generally consistent with standards in the social scientific research literature, and is particularly relevant for contemporary EEO analyses, where, through the use of technology, applicant and employee pools are often so large that statistical significance testing doesn’t tell us anything that we don’t already know—that observed differences in selection rates are beyond chance
The second FAQ focuses on selection procedures and validation research, and generally endorses standards from the Uniform Guidelines on Employee Selection Procedures (1978). The guidance defines and exemplifies selection procedures, explains the concept of validity, and provides some detail related to statistical analyses OFCCP would conduct that may identify disparities warranting validation evidence under the Guidelines. Although some contemporary science and practice have diverged from the Guidelines, they are still the federal regulation given the most deference in the context of equal employment opportunity and facially neutral selection procedures.
One point in particular from this FAQ is worth noting; some specific language demonstrates that the agency is paying attention to the use of artificial intelligence tools in employment decision making. In fact, the guidance document ends with the following Q&A:
Irrespective of the level of technical sophistication involved, OFCCP analyzes all selection devices for adverse impact. “If OFCCP discovers that a contractor’s use of an AI-based selection procedure is having an adverse impact at a contractor’s establishment, the contractor will be required to validate the selection procedure….”.
This FAQ underscores the necessity for federal contractors who use artificial intelligence to make employment decisions to be prepared for OFCCP to evaluate those tools. Validation research may be particularly important in this context.
By Sarah Layman, M.S., Senior Consultant at DCI Consulting Group
1Oswald, F.L., Dunleavy, E.M., & Shaw, A. (2017), "Measuring practical significance in adverse impact analysis" in Morris, S.B. & Dunleavy, E.M. (Eds.) Adverse impact analysis: Understanding data, statistics, and risk. New York: Routledge