What the AI System Risk Classification (EU AI Act) actually produces

HIGH-RISK under EU AI Act Annex III(4) (Employment, workers management, access to self-employment). The system ranks job applicants and recommends top 20% for human review, directly impacting hiring decisions. PROVIDER role carries CE marking, conformity assessment, technical documentation, and QMS obligations under Article 16-17. LIMITED-RISK transparency obligations under Article 50 if disclosure requirements apply. US state overlays: NYC Local Law 144 AEDT triggers algorithmic impact assessment and notice/feedback for candidates; EEOC AI guidance expects bias testing and non-discriminatory validation; California CPRA AI provisions may trigger impact assessments. GPAI obligations (Articles 51-55) apply if the system embeds a general-purpose LLM (requires provider disclosure if so). This requires legal counsel and a notified body to confirm final classification.

Annex III(4) Trigger: The system directly evaluates candidates and produces recommendations that inform hiring decisions—a protected characteristic under employment discrimination law (ECOA-adjacent in EU: Art. 21 Charter of Fundamental Rights, GDPR data minimization). The 0-100 scoring and top-20% recommendation create material risk of disparate impact based on protected attributes (gender, age, ethnic origin inferred from resume signals, national origin from name). At 50,000 applications/month across 30 customers, the scale amplifies any bias. Provider obligations under Article 43(2): CE marking and Declaration of Conformity (Article 47), technical documentation per Article 11+Annex IV, QMS per Article 17. Conformity assessment routes: Article 43 Internal conformity assessment (Annex VI) or notified body (Annex VII)—likely Annex VII given employment context. Article 6(3) exemptions unlikely: the system is not a preparatory task; it is intended to screen applicants, not merely detect patterns; and the deployer (HR team) relies on the score, not independent judgment. PROVIDER vs DEPLOYER: Provider (you) designs, trains, and licenses the model to HR clients. Deployer (client ATS system) carries responsibility for human oversight, data-input quality, log retention, and candidate notice under Article 25. NYC Local Law 144 AEDT: impacts deployer (HR client) primarily, but provider should document that the system can support client's obligations (bias audit, notice to candidates, record retention). EEOC AI guidance (2023): expects employment AI to demonstrate non-discriminatory performance across race, gender, age, national origin; internal bias testing and external validation recommended.

FOR INTERNAL LEGAL: (1) Is our intended use expressly 'hiring decision support' (screening prior to human review) or 'decision improvement'? The distinction affects Article 6(3) applicability. (2) Do we have contractual language requiring deployers to implement human oversight and candidate notice per Article 25, and to log usage per Article 72? (3) Which notified body should we engage for Annex VII conformity assessment—Notified Body [name] specializes in employment AI? (4) Do our license terms require deployers to comply with NYC Local Law 144 and EEOC expectations (bias audit, feedback mechanisms)? FOR NOTIFIED BODY INTAKE: (1) Per Annex VII, technical documentation must address: (a) training data representativeness across gender, age, ethnicity, national origin; (b) model card showing performance metrics disaggregated by these protected classes; (c) validation study design and results. Do we have these? (2) Risk management (ISO 14971 adapted): what are the identified failure modes (rank inversion for protected groups, correlated proxies for protected attributes) and mitigation (threshold adjustment, explainability, deployer override pathways)? (3) Human oversight architecture: does the system support human override, and does the deployer's ATS integration enforce this? (4) Post-market monitoring (Article 72): what KPIs (adverse hiring patterns, complaint frequency by demographic) will be tracked, and over what period?