We are back! After a brief hiatus, we continue our UGESP Series by exploring the UGESP concept of suitable alternative selection procedures addressed by the UGESP. This piece explores the difficulty in determining (a) whether two selection procedures are truly alternatives to one another and (b) what level of effort to identify a suitable alternative is considered reasonable.
Background
To the extent that a selection procedure results in adverse impact against a protected class (e.g., women, men), it is incumbent on the employer to (a) ensure that decisions based on selection procedure scores are valid and (b) evaluate the availability of alternative selection procedures that are equally valid for the intended purpose and result in lower adverse impact. As the UGESP state:
Where two or more selection procedures are available which serve the user's legitimate interest in efficient and trustworthy workmanship, and which are substantially equally valid for a given purpose, the user should use the procedure which has been demonstrated to have the lesser adverse impact… Whenever the user is shown an alternative selection procedure with evidence of less adverse impact and substantial evidence of validity for the same job in similar circumstances, the user should investigate it to determine the appropriateness of using or validating it in accord with these guidelines.
In short, if two selection procedures have similar levels of validity evidence but different levels of adverse impact, the procedure associated with lower adverse impact should be used. Thus, consideration of multiple selection procedures is an important aspect of quality validation studies. As stated in the UGESP:
…whenever a validity study is called for by these guidelines, the user should include, as a part of the validity study, an investigation of suitable alternative selection procedures and suitable alternative methods of using the selection procedure which have as little adverse impact as possible, to determine the appropriateness of using or validating them in accord with these guidelines.
Such guidance is sensible and should serve to result in the employment of selection procedures that maximize the validity of selection decisions while minimizing adverse impact against protected classes. However, two primary questions must be adequately answered to ensure that (a) selection procedures considered alternatives are truly alternatives and (b) adequate effort is devoted to identifying suitable alternatives. Those question are:
Is Procedure B truly an alternative to Procedure A? A suitable alternative is defined as a selection procedure that is substantially equally valid to a different selection procedure. Unfortunately, the UGESP and case law do not clearly address what constitutes substantially equally valid. As addressed in an earlier post, questions of validity refer to whether the decisions made based on a set of selection procedure scores are valid. That is, validity refers to whether the conclusions made based on a set of scores are justified. Thus, it is reasonably inferred that substantially equally valid refers to whether the validity research on each of the procedures is equally robust and indicates that:
Unfortunately, the idea of suitable alternatives is often over-simplified and thought to mean that two selection procedures have similar validity coefficients. Such a perspective ignores that the two procedures may be measuring substantially different characteristics and predicting substantially different aspects of work performance. For example, cognitive ability tests are typically associated with higher racial subgroup differences than are personality inventories, but in a particular situation, the validity coefficients associated with each may be similar. If one were to consider a personality inventory a suitable alternative for a cognitive ability test, one would be ignoring that the two selection procedures predict different on-the-job behaviors. In our view, such a perspective is inconsistent both with UGESP guidance and with contemporary selection research and guidance.
Take for example a police department that is using a reading comprehension test to ensure that its newly hired officers can understand the material presented in the academy (e,g., constitutional law, agency policies and procedures). The validation study shows that scores on the reading test predict performance in the academy as well as on-the-job performance in such dimensions as report writing, court testimony, and decision making. Because the test results in high levels of adverse impact, a civil rights group suggests that the department use a personality inventory instead of the cognitive ability test. The group’s argument is that both correlate .30 with academy performance but the personality inventory has less adverse impact. Is the personality inventory a reasonable alternative? No, because the two tests are tapping completely different characteristics.
If a particular selection procedure measuring cognitive ability is proposed to be used, other selection procedures measuring cognitive ability, but with smaller subgroup differences, should be considered as suitable alternatives. For example, if a traditional academic-based test measuring verbal and mathematical reasoning is being considered for selection purposes, a suitable alternative might be a test that measures reasoning via questions that ask about patterns in sets of symbols or diagrams. Each test is intended to measure reasoning ability, and the latter may be equally valid for selection into the target job yet may have smaller subgroup differences. This assumes that verbal and mathematical reasoning ability, per se, are not the characteristics critical to job performance, but rather the broader characteristic of reasoning ability is what really matters.
Essentially, a suitable alternative to a selection procedure under consideration is one that taps the same job-relevant characteristics but taps fewer job-irrelevant characteristics that exacerbate subgroup differences. Questions that require answers in the search for suitable alternatives include:
What is the burden to seek and identify an alternative? The UGESP note that use of a selection procedure producing adverse impact is lawful if, “a user has made a reasonable effort to become aware of such alternative procedures and validity has been demonstrated in accord with these guidelines...” They further note that:
…preliminary determination of the suitability of the alternative selection procedure for the user and job in question may have to be made on the basis of incomplete information…the investigation should continue until the user has reasonably concluded that the alternative is not useful or not suitable, or until a study of its validity has been completed.
But what is a reasonable effort? Thankfully, the UGESP Questions and Answers provide explicit guidance concerning the limits of a user’s effort and outline a series of steps to take to meet the burden.
The composite caveat One question that remains unanswered by UGESP and case law, to our knowledge, is whether a composite selection procedure, in which scores from different tests of different job-related characteristics are combined, presents a suitable alternative to just one test of one type of job-related characteristic. For example, it is unclear whether a composite selection procedure combining scores from a job-related cognitive ability test and a job-related personality inventory presents a suitable alternative to the cognitive ability test alone.
If the composite has stronger evidence of validity and smaller subgroup differences than the cognitive ability test alone, is the composite a suitable alternative? Perhaps it is because, unlike a situation in which a personality inventory is proposed as a suitable alternative to a cognitive ability test, the composite predicts both work behaviors associated with cognitive ability and work behaviors associated with personality.
A more complex scenario is one in which the composite has stronger evidence of validity but higher subgroup differences than the cognitive ability test alone. Such a scenario could arise if the personality test has subgroup differences, however small, in the same direction as the cognitive ability test. Because the two selection procedures are indicators of different characteristics, the subgroup differences may compound rather than average out. The question is whether the composite should be used because it has stronger validity evidence or whether the cognitive ability test should be used alone because it has smaller subgroup differences. Without clear guidance from the UGESP or case law, such a choice may come down to the values and objectives of the organization considering the selection procedures.
Of course, considerations of cost, timing, and other practical realities are relevant to whether investigations of available composites exceed the threshold of reasonable effort. Additionally, we have not addressed questions of how different components might be weighted, how to determine appropriate cut-scores, and whether different configurations of elements in a composite reflect suitable alternatives to one another. The answers to such questions are context dependent and are not readily explored in a single blog. What is clear from the UGESP, however, is that combining different selection procedures into composite measures is not prohibited by the suitable alternatives search requirement. The UGESP state:
Whenever the user is shown an alternative selection procedure with evidence of less adverse impact and substantial evidence of validity for the same job in similar circumstances, the user should investigate it to determine the appropriateness of using or validating it in accord with these guidelines. This subsection is not intended to preclude the combination of procedures into a significantly more valid procedure, if the use of such a combination has been shown to be in compliance with the guidelines” (emphasis in bold added).
Conclusion The requirement to seek and evaluate suitable alternative selection procedures can be a complex issue in practice. Unfortunately, guidance on the topic is relatively scarce, and there is widespread misinterpretation of what constitutes a suitable alternative and what efforts are required to seek out a suitable alternative. DCI’s Kayo Sady is co-editor of an upcoming handbook (to be published in 2014) that will provide concrete, practitioner-oriented guidance on a number of HR legal issues. The volume will include a thorough treatment of the suitable alternatives requirement including evaluation of the UGESP, case law, the research literature, case studies, best practice guidance, and attorney commentary.
by Kayo Sady, Ph.D, Consultant, Eric Dunleavy, Ph.D., Principal Consultant, and Mike Aamodt, Ph.D., Principal Consultant, DCI Consulting Group