Implementing the mutatis-mutandis (MM) comparator for discrimination testing: reliable generation of protected-attribute-independent comparators and robust characterization against the ceteris-paribus comparator

Alvarez and Ruggieri (2024/2026) reanalyzed discrimination testing — the legal-forensic practice of comparing a complainant's decision outcome with that of a similar 'comparator' profile differing in the protected attribute — and distinguished the standard ceteris-paribus (CP) comparator, which matches all non-protected attributes exactly, from the mutatis-mutandis (MM) comparator, which asks what the complainant's profile would have been without the protected attribute's influence on their other attributes. The MM comparator is currently realized only through counterfactual generation under an explicitly specified structural causal model (SCM), a route the authors themselves flag as knowledge-intensive and potentially non-scalable. The open problem is the implementation of the MM comparator as a general machine-learning problem: constructing, without a fully specified SCM, a generative adjustment that renders the comparator's non-protected attributes independent of the protected attribute (or an equivalent adjusted similarity); and robustly characterizing, across benchmark datasets, decision contexts, and protected attributes, how the choice of CP versus MM comparator changes discrimination-testing outcomes. No prior work implementing MM comparator generation without a fully specified SCM, and no robust multi-benchmark CP-versus-MM evaluation, is known in the literature to date.

Background

Individual discrimination testing is the practice, central to anti-discrimination law and to algorithmic-fairness auditing, of establishing prima facie evidence of discrimination by comparing the decision outcome for a complainant with the outcomes for similar profiles that differ only in the protected attribute (e.g., gender or race). The profile against which the complainant is compared is called the comparator. Situation testing (ST), introduced to data mining by Luong, Ruggieri, and Turini (KDD 2011), implements this with a k-nearest-neighbor search: for each complainant it builds a test group of similar protected individuals and a control group of similar non-protected individuals and compares decision outcomes between the groups. ST embodies a ceteris-paribus ('with all else equal', CP) notion of similarity: the comparator should match the complainant on every non-protected attribute. A structural causal model (SCM) is a formal causal model consisting of a directed graph over variables plus structural equations assigning each variable a function of its causal parents and exogenous noise; from an SCM one can compute counterfactuals — 'what would have been' — via Pearl's abduction–action–prediction procedure. Counterfactual fairness (Kusner, Loftus, Russell, and Silva, NeurIPS 2017) uses SCMs to require that a decision be unchanged in the counterfactual world where the protected attribute differs. Building on this, counterfactual situation testing (CST; Ruggieri and Alvarez, EAAMO 2023; Alvarez and Ruggieri, JAIR 2025) replaces the matched control group of ST with a test group generated from the complainant's counterfactual — the profile that would have obtained had the protected attribute been different, allowing the protected attribute's downstream effects on seemingly neutral attributes (e.g., address, occupation) to change with it. CST carries this out under an explicitly specified SCM, supports single and multidimensional (multiple and intersectional) protected attributes, and equips each claim with confidence intervals. Alvarez and Ruggieri's 'Mutatis Mutandis' paper (arXiv 2405.13693; Computational Intelligence, 2026) generalizes this landscape into a two-kind classification of comparators: the CP comparator (the standard idealized comparison, of which ST is an instance) and the mutatis-mutandis ('with the appropriate adjustments being made', MM) comparator, which requires a comparator representing the 'what would have been' of the complainant without the protected attribute's effects on the non-protected attributes, so that the complainant–comparator pair may legitimately differ on those attributes. The paper formalizes the MM comparator as a generative adjustment $g(\cdot)$ producing adjusted non-protected attributes $\tilde{\mathbf{X}}$ with $\tilde{\mathbf{X}}\perp A$ ($A$ the protected attribute), illustrates with one real-world experiment that the CP-versus-MM choice changes how many discrimination cases are detected, and explicitly leaves open how the MM comparator can be implemented in practice: SCM-based counterfactual generation 'requires sufficient knowledge about the SCM generating the data, which is complex and potentially non-scalable in practice'; fair representation learning (learning representations of neutral attributes that remove protected-attribute information while retaining downstream-useful information), variational autoencoders (generative models that learn a compressed probabilistic latent representation of the data and generate new samples from it), and causal normalizing flows (normalizing flows trained to transform between factual and counterfactual distributions without an explicit SCM) are named as candidate directions but not developed; an adjusted similarity or data-driven distance function is floated as an alternative route and equally left open; and the empirical evaluation remains a single illustrative experiment rather than a robust multi-dataset study. The authors also note, and their companion 'causal perception' line (AIES 2025; Álvarez, arXiv 2608.03917) begins to explore, that competing SCMs of the same system can yield different counterfactuals and hence different discrimination verdicts — underscoring that the reliability of MM-based testing is unresolved.

Problem Statement

As stated by Alvarez and Ruggieri (arXiv:2405.13693; DOI 10.1111/coin.70238), the MM comparator 'denotes a more complex object and its implementation offers an impactful venue for machine learning methods', and the paper leaves that implementation open. The problem is to implement the MM comparator for discrimination testing as a reliable machine-learning method. That implementation comprises two jointly posed subgoals: (1) Generation. Construct a generative adjustment $g(\cdot)$ mapping an individual's non-protected attributes $\mathbf{x}$ to adjusted attributes $\tilde{\mathbf{x}}$ such that the adjusted attributes are independent of the protected attribute $A$ ($\tilde{\mathbf{X}}\perp A$) while retaining the information needed to form meaningful test and control groups, without requiring a fully specified structural causal model; or, equivalently, construct an adjusted similarity (distance) function $\tilde{d}(\cdot,\cdot)$ that realizes the same 'what would have been' objective for comparator search. The SCM knowledge bottleneck (learning or eliciting a suitable causal model at scale across diverse decision contexts, and handling competing causal models) is the central difficulty this component must overcome. (2) Empirical characterization. Evaluate the CP versus MM comparator choice beyond the paper's single illustrative experiment, across multiple fair-ML benchmark datasets, decision contexts, and protected attributes, characterizing robustness, fairness–accuracy trade-offs, and the circumstances under which the two comparators yield materially different discrimination-testing outcomes.

References

1. José M. Álvarez, Salvatore Ruggieri. Mutatis Mutandis: Revisiting the Comparator in Discrimination Testing. Computational Intelligence 42 (2026); earlier version arXiv:2405.13693 (2024). DOI: 10.1111/coin.70238. https://doi.org/10.1111/coin.70238
2. Jose M. Alvarez, Salvatore Ruggieri. Counterfactual Situation Testing: From Single to Multidimensional Discrimination. Journal of Artificial Intelligence Research 82:2279-2323 (2025). DOI: 10.1613/jair.1.17935. https://arxiv.org/abs/2502.01267
3. Salvatore Ruggieri, Jose Manuel Alvarez. Counterfactual Situation Testing: Uncovering Discrimination under Fairness given the Difference. In Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO '23), ACM (2023). DOI: 10.1145/3617694.3623222. https://doi.org/10.1145/3617694.3623222
4. Jose M. Álvarez. Implementing Causal Perception: Competing SCMs and Situated Fairness. arXiv:2608.03917 (2026). https://arxiv.org/abs/2608.03917
5. Jose M. Alvarez, Salvatore Ruggieri. Toward A Causal Framework for Modeling Perception. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES) 8:166-178 (2025). DOI: 10.1609/aies.v8i1.36539; arXiv:2401.13408. https://doi.org/10.1609/aies.v8i1.36539
6. Matt J. Kusner, Joshua R. Loftus, Chris Russell, Ricardo Silva. Counterfactual Fairness. Advances in Neural Information Processing Systems (NeurIPS) (2017); arXiv:1703.06856. https://arxiv.org/abs/1703.06856
7. Binh Thanh Luong, Salvatore Ruggieri, Franco Turini. k-NN as an implementation of situation testing for discrimination discovery and prevention. In Proceedings of the 17th ACM SIGKDD Conference (KDD '11) (2011). DOI: 10.1145/2020408.2020488. https://doi.org/10.1145/2020408.2020488
8. Ludwig Bothmann, Philip A. Boustani, Jose M. Alvarez, Giuseppe Casalicchio, Bernd Bischl, Susanne Dandl. Privilege Scores. arXiv:2502.01211 (2025). https://arxiv.org/abs/2502.01211
9. Joe Germino, Nuno Moniz, Nitesh V. Chawla. Intersectional Divergence: Measuring Fairness in Regression. arXiv:2505.00830 (2025). https://arxiv.org/abs/2505.00830