by Art Gutman Ph.D., Professor, Florida Institute of Technology
The case is M.O.C.H.A. v. City of Buffalo, decided in a majority ruling by the 2nd Circuit on July 30, 2012 [2012 U.S. App. LEXIS 15715]. M.O.C.H.A. (“Men of Color Helping All”) sued the Buffalo Fire Department (BFD), claiming that promotion exams to lieutenant administered in 1998 (and again in 2002) produced adverse impact on black firefighters, and that they were not job related and consistent with business necessity. There were three district court rulings in this case, but the one that is most important for present purposes is M.O.C.H.A. v. City of Buffalo (2009) [2009 U.S. Dist. LEXIS 20070], in which District Court Judge John T. Curtin of the Western District of New York, found that the test in question was job related and consistent with necessity based on a content validity analysis. This was the main ruling affirmed by a 2-1 majority of the 2nd Circuit, and it is the content analysis performed that seems rather questionable, given the nature of the job analysis that was used by the BFD.
As a starting point, there is nothing wrong with supporting a test with a well-conducted content validity strategy. Indeed, five parameters for content validity was laid down originally by the 2nd Circuit in Guardians v. Civil Service (1980) [630 F.2d 79], and includes:
(1) a suitable job analysis,
(2) reasonable competence in constructing the test itself
(3) content of the test is related to the content of the job
(4) content of the test is representative of the content of the job, and
(5) scoring system selects applicants who can better perform the job
These parameters have been endorsed by all other circuit courts that have examined content validity, and by the 2nd Circuit itself in Gulino v. N.Y. State Educ. Dep't (2006) [460 F.3d at 382]. As a starting point, there was no question that the test produced adverse impact, and the plaintiffs never asserted there were alternative methods with less or no adverse impact. Therefore, the only question in this case is whether the two majority judges correctly interpreted the Guardians precedent, or whether the dissenting judge had the correct interpretation.
The key issue is related to Guardians is parameter #1 (suitable job analysis). More specifically, did the defense expert, Dr. Wendy Steinberg, an associate personnel examiner with the Buffalo Civil Service Department, use a “suitable” job analysis strategy? Steinberg collected job specification form various NY fire departments by job title, assembling 190 firefighter tasks for various ranks that were reviewed by the Civil Service Department’s Fire Advisory Committee, who were deemed experts on administration of firefighter departments. She then used this list to create a statewide survey to identify the criticality of these tasks, and a second survey to assess critical knowledge, skills and abilities (KSAs) to perform these tasks. The job tasks and KSAs were then sent to every incumbent firefighter in NY, except for NY City and Rochester. The return rate for the job tasks was relative good in the 12 largest jurisdictions sampled (2,502 out of 5.934), but the return rate from Buffalo itself was poor (68 out of 833). Additionally, no Buffalo firefighter responded to the KSA survey.
The plaintiff’s expert, Dr. Kevin Murphy, a distinguished SIOP Fellow who has written extensively on test validity strategies, found Steinberg’s strategy deficient in two major respects. First, he argued that a suitable job analysis was not conducted, not only because of the lower response rates in Buffalo, but also because it is a mistake to assume that the “job is similar from one place to another, without any detailed analytic demonstration to support the assumption.” Second, he criticized Steinberg’s reliance on validity generalization studies because “validity generalization is a method widely used in criterion-related test validation studies, but has little relevance in content-related validation studies.” Based on these criticisms (among other things), Murphy concluded that because of weakness in the first Guardians parameters, there were also weakness in the other four parameters.
This is the short version (believe it or not). Steinberg and Murphy were not the only experts, and there are other issues that, for the sake of brevity (to some extent anyway) are less important for present purposes. To finish the story, the two majority judges (Walker and Raggi) found that the test satisfied the five Guardians parameters and affirmed the summary judgment of the district court. The dissenting judge (Kearse) wrote that with no "expert opinion that the statewide job analysis was suitable to the Buffalo Fire Department …. the district court allowed Buffalo to avoid liability for its use of a racially discriminatory test on the ground that Buffalo had proven content-validity without:
■ participating in the test preparation,
■ hiring an expert to advise it in advance whether the test, prepared solely through the efforts of others, would be suitably related to the job in question,
■ hiring an expert thereafter to evaluate the content-validity of the test given and to testify to its validity,
■ reference by the test's creator to any data to substantiate her "guess" and her "presum[ption]" that the data she received from others reflect Buffalo's undisclosed needs,
■ presenting any evidence that any of Buffalo's own knowledgeable personnel ever looked at the Exam materials to determine whether the areas in which questions were, or were to be, posed were material to the job in question, and
■ making any attempt to show that the weighting of the areas on the Exam reflected the requirements for that position in Buffalo.
Kearse also wrote:
And given the Supreme Court's recent decision in Ricci v. DeStefano, 129 S. Ct. 2658 (2009), holding that a municipality may not refuse to certify a test that has disparate racial impact unless the municipality has a strong evidentiary basis for believing that certification would subject it to disparate-impact liability, it strikes me that this affirmance--allowing Buffalo to avoid liability without having made any effort whatever to seek, verify, or defend the test's validity--will make it virtually impossible for a municipality not to certify for use a test that has clear discriminatory impact.
Sounds like the kind of split ruling that spurs Supreme Court reviews.