The problem concerns the evidentiary infrastructure for proving algorithmic discrimination: how the disparate impact of high-dimensional machine-learning decision systems can be measured with statistically robust procedures, how those measurements map onto the legal categories of prima facie indirect discrimination and objective justification, and under what data-access and institutional arrangements such tests could produce legally reliable evidence. The problem remains open: mappings between EU non-discrimination law and fairness metrics have been shown to be necessarily incomplete, the dominant four-fifths-rule metric has been shown to distort its legal source, statistical audit methods are advancing without legal validation, and the newest formal frameworks (2024-2025) remain single-jurisdiction theoretical proposals that no court or regulator has accepted as an operational procedure. No accepted, validated procedure exists for establishing algorithmic disparate impact or adjudicating its objective justification in a legally reliable way.

Background

Algorithmic decision-making systems - supervised machine-learning models trained on historical data to predict outcomes such as loan default, job performance, or reoffending, with those predictions used to decide about people - are now common in employment, credit, insurance, policing, and welfare. A well-documented risk is that such systems reproduce past discrimination: even when a protected characteristic (ethnic origin, sex, religion, and similar grounds) is not an input, features correlated with it, called proxies (for example postal code correlating with ethnicity), can produce systematically worse outcomes for protected groups.

In Europe, the principal legal response is non-discrimination law, in particular the prohibition of indirect discrimination (in the United States the equivalent concept is called disparate impact): a practice that looks neutral but has disproportionately prejudicial effects on a group defined by a protected characteristic. Under EU law and the case law of the European Court of Human Rights, a claimant who establishes facts from which discrimination may be presumed - a prima facie showing, in practice a statistical comparison of outcomes between groups - shifts the burden of proof to the defendant. The defendant can escape liability with an objective justification: showing that the practice pursues a legitimate aim and bears a reasonable relationship of proportionality between means and aim. Data protection law adds supplementary instruments, notably Article 22 of the General Data Protection Regulation (GDPR), a right not to be subject to decisions based solely on automated processing with legal or similarly significant effects, and data protection impact assessments (DPIAs), which are documented risk analyses required for high-risk processing.

Zuiderveen Borgesius (2020) evaluated this legal protection against algorithmic discrimination in Europe and identified unresolved problems centred on proof. A victim must make a statistical prima facie showing, but algorithmic systems are opaque black boxes, often shielded by trade secrets, so neither victims, journalists, nor researchers can typically observe outcomes broken down by group; and the objective-justification defence is an open-ended, all-the-circumstances standard that yields no determinate procedure for deciding when a statistical disparity produced by a model with hundreds of input features is justified. The paper also flagged the institutional side: trade-secret and privacy rules restrict who may examine models and data, sensitive attributes are needed to measure bias but their collection is itself restricted, and 'more research and debate are needed on who should conduct such audits'.

In parallel, the machine-learning literature has produced a large inventory of fairness metrics - statistical summaries of group outcomes, such as selection-rate ratios or error-rate differences - together with results showing that different fairness criteria can be mutually unsatisfiable, so that any choice among them involves trade-offs that law would have to arbitrate (Zehlike et al. 2022). But almost none of these metrics was derived from, or validated against, the evidential requirements of non-discrimination law. The de facto standard, the four-fifths rule (a United States enforcement heuristic treating a selection rate below 80%80\% of the highest group's rate as evidence of adverse impact), was abstracted into the disparate-impact metric of popular fairness toolkits, and Watkins and Chen (2024) have shown that this abstraction distorts the legal rule it purportedly encodes. The unresolved question is whether the statistical and legal sides can be bridged at all, and how.

Problem Statement

The problem is to make the proof and adjudication of algorithmic discrimination operational, so that testing is simultaneously statistically sound and legally reliable. Concretely, a constructive answer must supply all three of the following jointly necessary components:

(1) Metrics and testing procedures. Develop metrics and statistical testing procedures capable of establishing, from realistic audit data, whether a high-dimensional algorithmic decision system has disproportionately prejudicial effects on a protected group - procedures that remain valid under the conditions of real audits: many candidate subgroups and metrics tested simultaneously, small per-group samples, ground-truth labels that are themselves biased, features acting as proxies for protected attributes, demographic data that are missing because their collection is legally restricted, and distribution shift over deployment time. A resolution must state the validity guarantee its procedure provides, and show that the guarantee holds under all of these audit conditions rather than only for a fixed, well-sampled setting.

(2) Doctrinal mapping (adjudication). Provide the mapping that connects those statistical findings to the legal categories of non-discrimination law in a named jurisdiction: which statistical facts constitute a prima facie showing of indirect discrimination sufficient to shift the burden of proof; when algorithmic differentiation counts as direct discrimination not admitting of justification; and how an objective justification - legitimate aim and proportionality - can be established or contested from evidence about a model, its training data, and its deployment, in a form that courts, regulators, and auditors can apply. A resolution must state the mapping as testable propositions tied to specific provisions and case law of the one named jurisdiction, and show how each proposition would be applied or contested in a concrete proceeding.

(3) Access and institutional arrangements. Specify under what data-access and institutional regime such tests can actually be performed and their results accepted as evidence: who may access the model, code, and sensitive-attribute data and under what safeguards, given the tension between data-protection restrictions on processing special-category data and the practical need for those attributes to measure bias, and which bodies possess the expertise, independence, and funding to conduct such audits. A resolution must name, for the named jurisdiction, who may access which artifacts under what safeguards, and which body has the competence, independence, and funding to conduct and certify such audits.

The common objective uniting the three components is a procedure whose output could function as evidence of algorithmic discrimination in legal proceedings. A constructive answer must satisfy all three components and their stated conditions together; an answer delivering only statistical machinery without legal reliability, or only doctrinal argument without executable testing procedures, does not solve the problem. A limitative answer also resolves the problem if it establishes, for the named jurisdiction, why no procedure meeting the common objective can exist.

References

  1. Frederik J. Zuiderveen Borgesius (2020). Strengthening legal protection against discrimination by algorithms and artificial intelligence. The International Journal of Human Rights 24(10):1572-1593. DOI 10.1080/13642987.2020.1743976. https://doi.org/10.1080/13642987.2020.1743976
  2. Raphaële Xenidis (2020). Tuning EU equality law to algorithmic discrimination: Three pathways to resilience. Maastricht Journal of European and Comparative Law 27(6):736-758. DOI 10.1177/1023263X20982173. https://doi.org/10.1177/1023263X20982173
  3. Sandra Wachter, Brent Mittelstadt, Chris Russell (2021). Why fairness cannot be automated: Bridging the gap between EU non-discrimination law and AI. Computer Law & Security Review 41:105567. DOI 10.1016/j.clsr.2021.105567. https://doi.org/10.1016/j.clsr.2021.105567
  4. Reuben Binns, Jeremias Adams-Prassl, Aislinn Kelly-Lyth (2023). Legal Taxonomies of Machine Bias: Revisiting Direct Discrimination. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT 2023), 1850-1858. DOI 10.1145/3593013.3594121. https://doi.org/10.1145/3593013.3594121
  5. Hilde Weerts, Aislinn Kelly-Lyth, Reuben Binns, Jeremias Adams-Prassl (2024). Unlawful Proxy Discrimination: A Framework for Challenging Inherently Discriminatory Algorithms. arXiv:2404.14050. https://arxiv.org/abs/2404.14050
  6. Elizabeth Anne Watkins, Jiahao Chen (2024). The four-fifths rule is not disparate impact: A woeful tale of epistemic trespassing in algorithmic fairness. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT 2024), 764-775. DOI 10.1145/3630106.3658938. https://doi.org/10.1145/3630106.3658938
  7. Jongbin Jung, Sam Corbett-Davies, Johann D. Gaebler, Ravi Shroff, Sharad Goel (2018). Mitigating Included- and Omitted-Variable Bias in Estimates of Disparate Impact. arXiv:1809.05651. https://arxiv.org/abs/1809.05651
  8. Ben Chugg, Santiago Cortes-Gomez, Bryan Wilder, Aaditya Ramdas (2023). Auditing Fairness by Betting. NeurIPS 2023; arXiv:2305.17570. https://arxiv.org/abs/2305.17570
  9. Harvineet Singh, Fan Xia, Mi-Ok Kim, Romain Pirracchio, Rumi Chunara, Jean Feng (2023). A Brief Tutorial on Sample Size Calculations for Fairness Audits. arXiv:2312.04745. https://arxiv.org/abs/2312.04745
  10. Yash Vardhan Tomar (2025). What Must a Fairness Audit Report When Demographic Data Is Incomplete? arXiv:2506.23033. https://arxiv.org/abs/2506.23033
  11. Holli Sargeant, Måns Magnusson (2025). Formalising Anti-Discrimination Law in Automated Decision Systems. Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (FAccT 2025), 181-194. DOI 10.1145/3715275.3732015. https://doi.org/10.1145/3715275.3732015
  12. Sasha Costanza-Chock, Inioluwa Deborah Raji, Joy Buolamwini (2022). Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem. In 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT 2022), 1571-1583. DOI 10.1145/3531146.3533213. https://doi.org/10.1145/3531146.3533213
  13. Lucas Michael Haitsma (2023). Regulating algorithmic discrimination through adjudication: the Court of Justice of the European Union on discrimination in algorithmic profiling based on PNR data. Frontiers in Political Science 5. DOI 10.3389/fpos.2023.1232601. https://doi.org/10.3389/fpos.2023.1232601
  14. Alessandra Calvi, Dimitris Kotzinos (2023). Enhancing AI fairness through impact assessment in the European Union: a legal and computer science perspective. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT 2023), 1229-1245. DOI 10.1145/3593013.3594076. https://doi.org/10.1145/3593013.3594076
  15. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Article 10 (data and data governance): bias examination under Article 10(2)(f)-(g) and exceptional processing of special-category personal data for bias detection and correction under Article 10(5). CELEX 32024R1689; ELI https://eur-lex.europa.eu/eli/reg/2024/1689/oj
  16. Meike Zehlike, Alex Loosley, Håkan Jonsson, Emil Wiedemann, Philipp Hacker (2022). Beyond Incompatibility: Trade-offs between Mutually Exclusive Fairness Criteria in Machine Learning and Law. arXiv:2212.00469. https://arxiv.org/abs/2212.00469
  17. Jose M. Alvarez, Alejandra Bringas Colmenarejo, Alaa Elobaid, Simone Fabbrizzi, Miriam Fahimi, Antonio Ferrara, Siamak Ghodsi, Carlos Mougan, Ioanna Papageorgiou, Paula Reyero, Mayra Russo, Kristen M. Scott, Laura State, Xuan Zhao, Salvatore Ruggieri (2024). Policy advice and best practices on bias and fairness in AI. Ethics and Information Technology 26(2), article 31. DOI 10.1007/s10676-024-09746-w. https://doi.org/10.1007/s10676-024-09746-w