In this installment of our UGESP series, we address an aspect of the validation process that presents significant EEO enforcement risk if overlooked: selection procedure implementation.
The purpose of a validation study is to collect evidence that supports the use of a particular selection procedure for a particular purpose. Typically, decisions are made based on a set of scores from a selection procedure. Decisions based on the set of scores are valid to the extent that the scores are meaningful and allow for appropriate inferences, and any factor that undermines the meaning of the scores also undermines the validity of any decisions based on the scores.
One aspect of the validation process that we have observed being overlooked fairly often is the implementation and use of a selection procedure once the developmental research has been conducted. Common post-development practices include establishing the intended use of the procedure, the administration and scoring protocols, and setting cut-scores to make actual employment decisions.
The UGESP are clear that the validation process is not finished once a selection procedure has been developed. Inconsistent or improper practices at the implementation stage create error in selection procedure scores, and as a result, make it more difficult to defend that the scores provide reliable information about an individual’s potential job performance. The questions below are not meant to be exhaustive but provide examples of the types of questions that need to be considered (and the answers to which need to be defensible) at the selection procedure implementation stage:
- For what jobs is it appropriate to use the selection procedure?
- For what purpose(s) is the selection procedure designed?
- Are there clear and comprehensive instructions to guide administration of the procedure?
- Is administration of the procedure standardized such that all individuals completing the process experience the same environmental factors (e.g., same noise levels, same amount of time, same provided materials)
- Is training provided to the administrators?
- Are cut-scores appropriate and set in reasonable ways?
- Are implemented cut-scores “reasonable and consistent with normal expectations of acceptable proficiency within the work force”?
- Are pass rates and sub-group differences measured and monitored over time?
by Kayo Sady, Ph.D., Consultant, Eric Dunleavy, Ph.D., Principal Consultant, and Mike Aamodt, Ph.D., Principal Consultant, DCI Consulting Group
