By: Dave Schmidt and Bre Timko
Overview
The roundtables were held by DCWP to help those seeking to comply with the law. Three separate roundtables were held for three different audiences: employers/business advocates, audit industry stakeholders, and civil/worker rights advocates. During these roundtables, representatives from DCWP provided an overview presentation about the law, followed by a live question-and-answer session. Many, but not all, questions that were addressed made their way into the FAQ produced by the DCWP in late June. In both the roundtable presentation and the FAQ document, much of the original law and rules were reiterated, however, there were also some very useful clarifications that are highlighted below:
- In the City: The single most useful clarification provided by the DCWP may be what the term “in the City” means for this law as this is critical to determining if compliance is required. FAQ I.4 specifies that the phrase “in the City” is based on both the candidate residency and the job location or associated office location. Specifically, this law applies if:
-
- “The job location is an office in NYC, at least part time. OR
- The job is fully remote but the location associated with it is an office in NYC. OR
- The location of the employment agency using the AEDT is NYC or, if the location of the employment agency is outside NYC, one of the bullets above is true.”
This is slightly different than what was conveyed in the roundtable sessions. However, the key takeaway is that NYC residents applying to jobs located outside of the City will not trigger the need to comply with this law (unless an employment agency located in the City is used to administer the assessment). While there are still some shades of grey here, this is substantially more well-defined than it was previously.
- AEDTs out of scope: FAQ I.6 notes that this law only applies to assessments that are used once an individual becomes a candidate for the job (i.e., “a person who has applied for a specific position by submitting the necessary information or items in the format required by the employer or employment agency”). It does not apply to any assessments used prior to this point (e.g., candidate outreach, scanning a resume bank to invite individuals to apply to a job).
- AEDTs in scope: The FAQ included nothing new related to what is considered an AEDT, so any prior determinations as to whether an assessment is an AEDT or not still hold. However, it is worth highlighting a subtle clarification DCWP made to the final rules, which was reiterated in FAQ I.5. Specifically, this law applies to more than just the final hiring/promotion decision. An AEDT used to make or substantially assist a decision about progression of a candidate anywhere in the selection process is covered by this law.
- Purpose of the law: In the roundtables (and echoed in the FAQ), the DCWP emphasized that this law is geared toward transparency to candidates who are NYC residents and requirements are about notifying and publishing audit results—it is not intended to interpret bias audit results, compel employers to stop using an AEDT, nor enforce anti-discrimination law. However, that may be done by others. Per FAQ VII.2, any complaints of discrimination will be referred to the NYC Commission on Human Rights. Further, FAQ II.2 emphasizes that employers need to be aware of and comply with any federal, state, or city anti-discrimination laws.
- Complaints about violations of this law: Information for candidates to register complaints related to this law was provided. According to the roundtable sessions, the DCWP plans to use the complaints as the primary mechanism for directing their investigative activities.
- Data flexibility: FAQ III.1 – 4 relates to allowing employers and auditors some flexibility around the data that can be used for the bias audit. However, decisions made regarding the data used for the audit should be sufficiently described in the audit report. Unfortunately, FAQ III.7 regarding what is a “statistically significant bias audit” also conveyed flexibility resulting in ambiguity where further clarity was needed.
- Unavailable or missing demographic data: Per FAQ III.5, if demographic data are not available or are insufficient, “test data” may be used. Unfortunately, FAQ III neglects to provide any clarification as to what comprises test data other than anything that is not historical data. However, FAQ III.8 indicates the use of test data requires an explanation of how it was derived. Finally, FAQ III.6 makes it clear that demographic information about candidates should not be imputed or inferred.
- Exclusions from analysis: Despite some use cases where the 2% rule outlined in the rules falls short of safeguarding against analyses based on samples sizes that are too small to be useful, the roundtables and FAQ III.9 reiterated that this is the only available mechanism to account for small sample and cell sizes.
- Responsibility for compliance: FAQ V.1 clearly states that, “Employers and employment agencies are responsible for ensuring they do not use an AEDT unless a bias audit was done. The vendor that created the AEDT is not responsible for a bias audit of the tool.” This is not to say that an employer cannot use a vendor-level audit (as indicated in FAQ V.2 and via use cases listed in the final rules), but ensuring the bias audit is done prior to use of the AEDT falls on the employer.
- Good faith efforts: In the roundtables, the DCWP noted that they are not looking to punish good faith efforts to comply with the law. This is certainly a welcomed sentiment for all employers diligently trying their best to determine if this law applies to them, and, if so, come into compliance.
Finally, it is notable that there were a number of key elements that the FAQ merely reiterates from the final rules issued in April. DCWP did not alter anything related to the:
- Definition of automated employment decision tool (nor the substantially assists clause)
- Definition of machine learning, statistical modeling, data analytics, or artificial intelligence
- What comprises a bias audit
- What constitutes an independent auditor
- Publication requirements
DCI will monitor and share any additional updates on this law, as well as activity related to federal, state, and local regulations relevant to the use of algorithmically-driven assessments in employee selection.
