Robustness and calibration of CORaL's significant-set inference for ranked-list comparison: gene dependence, step-size selection, and operating regimes

CORaL (Comparison of Ranked Lists; Antosh, Fox, Cooper and Neretti, 2013) compares two gene-expression experiments by using the full ranked lists of genes ordered by fold change: it infers, for each list, a 'significant set' size — the depth beyond which the two lists share no more overlap than expected by chance — by maximizing, over a grid whose granularity is set by a step size, the probability of reaching and stopping at each grid point, with per-step probabilities computed from exact 3x3 contingency-table tests that treat genes as independent. The paper acknowledged but did not resolve four linked issues: the independence approximation underlying its step probabilities (the Benjamini–Yekutieli correction controls false discovery under arbitrary dependence of the tested steps but does not correct the null model itself); the absence of an intrinsic rule for the step size beyond a heuristic geometric ladder; the lack of validation beyond gene-expression data; and benchmark evidence restricted to two simulation configurations and two real datasets. No citing work analyzes CORaL's statistical properties, and the successor RedRibbon (2024) resolved the analogous step-size and dependence-calibration problems only for the RRHO minimum-p-value statistic, not for CORaL's path-based reach-and-stop statistic. The open problem is a reliability characterization of CORaL's significant-set inference: quantify how gene–gene dependence affects the inferred set boundaries, provide an intrinsic data-driven step-size rule or establish that none exists, delimit the regimes of overlap, noise and list-length imbalance in which CORaL's advantage over competing rank-based methods holds, and establish whether the validated range extends beyond gene-expression data.

Background

A gene-expression experiment — measured by microarray (a chip that quantifies the expression of thousands of genes at once) or by RNA-seq (RNA sequencing, which counts transcript molecules) — measures how strongly each gene's expression changes between a treatment and a control condition; the standard per-gene summary is the fold change, the ratio of mean expression in treatment to that in control. Comparing experiments from different laboratories is complicated by the customary practice of selecting 'differentially expressed' genes through cutoffs on fold change or statistical significance: such cutoffs are arbitrary and can select genes by different effective criteria in different datasets because of differences in protocol and analysis. Rank-based comparison methods avoid cutoffs by using only the order of genes in each list, ranked by fold change.

Two established rank-based methods frame the problem. OrderedList (Yang et al., 2006) scores the similarity of two ranked lists as an exponentially weighted sum of prefix overlap counts and assesses significance by permutation. RRHO (Rank-Rank Hypergeometric Overlap; Plaisier et al., 2010) computes, for every pair of list prefixes (top $m$ genes of list 1, top $n$ genes of list 2), the hypergeometric probability of the observed overlap — Fisher's exact test, which gives the probability under random ordering of one list of seeing an overlap at least as large — and visualizes the significance landscape as a rank-rank map, summarizing it by the minimum-p-value point.

CORaL (Antosh, Fox, Cooper and Neretti, 2013) was designed to estimate more precisely the significant set: the number of genes at the top of each list that genuinely change in both experiments. CORaL walks down the two lists in steps. The top $m$ genes of list 1 and top $n$ genes of list 2 share some overlap $k$; advancing $m$ by $\Delta m$ and $n$ by $\Delta n$ changes the overlap by $\Delta k$, and the probability of a specific overlap pattern at a step is computed with an exact formula for a 3x3 contingency table (the four overlaps between the current and added list sections; Ghent, 1972), normalized by a Fisher's exact probability that fixes the starting overlap. A small step p-value means the step added unusually much shared signal. CORaL then selects the pair $(m,n)$ maximizing the product of two quantities: the total probability of all grid paths reaching $(m,n)$ (computed recursively, each step weighted by one minus its p-value) and the probability of stopping there. Step p-values are corrected for multiplicity with the Benjamini–Yekutieli procedure (Benjamini and Yekutieli, 2001), which controls the false discovery rate — the expected fraction of false positives among flagged steps — under arbitrary dependence among the tested hypotheses. The method's only tunable parameter is the step size, which sets the grid granularity.

On simulated lists with planted overlaps and on two real datasets (Pearson et al., 2008; Bauer et al., 2010), CORaL recovered planted significant-set sizes more accurately than RRHO and OrderedList. But the paper's own conclusions flag what remains unresolved. First, the contingency-table machinery 'makes an implicit approximation that genes are behaving independently', which is 'of course not always true': real expression data have correlated rank structure (co-expressed genes, shared regulation), and the Benjamini–Yekutieli correction controls the false discovery rate among the many tested steps without changing the null model from which each step probability is derived — the authors checked only, informally and diagnostically, that the method behaved satisfactorily on partially correlated data. Second, the step size is chosen by a heuristic ladder (e.g., 50, 100, 200, 400, 800 — take the first step size whose successor selects the same set to within one step) because small steps get stopped by random fluctuations while large steps skip structure; no intrinsic rule exists, and the heuristic is least reliable exactly when the scale of random fluctuations is comparable to the scale of the biological signal. Third, the closing suggestion that the method 'could be applicable to data from fields outside of gene expression and biology' carries no validation. Fourth, the benchmark evidence covers two simulation configurations and two real datasets, leaving the operating regimes (overlap size, noise level, list length, list-length imbalance) unexplored. Later work in the rank-rank overlap family — RRHO2 (Cahill et al., 2018) and most recently RedRibbon (Piron et al., 2024), which replaces the RRHO grid with an evolutionary minimum-p-value search and adds a correlation-aware permutation correction — has addressed resolution and dependence questions for the RRHO statistic, not for CORaL's path-based statistic.

Problem Statement

The source question, kept at the source's own generality, asks when CORaL's inference of significant sets is reliable and how its one calibration choice can be made intrinsic. CORaL takes two lists of genes ranked by fold change and returns the significant set sizes $(m,n)$ — the depths beyond which the two lists no longer share more overlap than chance — found by maximizing a reach-and-stop path probability computed from per-step 3x3 contingency-table p-values on a grid of granularity set by a step size, with steps corrected for multiplicity by the Benjamini–Yekutieli procedure. Four linked questions, sharing the single objective of establishing the reliability conditions and intrinsic calibration of this inference, remain open:

(1) Dependence robustness. The per-step probabilities are exact only under the null model in which genes are independent (exchangeable). Quantify how gene–gene dependence of controlled strength and structure (for example, co-expression blocks — groups of genes whose ranks move together — inducing correlated rank perturbations) affects CORaL's inferred significant-set boundaries: the calibration of nominal versus empirical significance, and the bias and variability of the selected $(m,n)$, relative to the independence approximation. Identify the dependence regimes in which the inference degrades, given that the Benjamini–Yekutieli correction addresses the multiplicity of step selection but not the null model itself.

(2) Intrinsic resolution rule. Replace the geometric-ladder heuristic for the step size with an intrinsic, data-driven rule that demonstrably separates biological signal from random fluctuations, where the class of admissible rules is any step-size selection determined by the two ranked lists alone and separation is scored on planted-signal ground truth — on simulated lists with a known planted significant set, the rule's inferred significant-set boundary must recover the planted boundary — or establish that no rule in this class achieves such separation within a specified overlap-and-noise regime; characterize the regimes — including the hard regime where the scale of random fluctuations is comparable to the scale of the biological signal — in which such a rule is or is not reliable.

(3) Operating regimes. The original evidence for CORaL's advantage over rank-based competitors covers two simulation configurations and two real datasets. Systematically delimit the regimes of overlap size, noise level, list length, and list-length imbalance in which CORaL's significant-set estimates are more accurate than competing rank-based methods (such as RRHO and OrderedList), and the regimes in which they are not.

(4) Generality. The source suggests, without validation, applicability to ranked-list data beyond gene expression. Establish, by at least one non-gene ranked-list demonstration or a principled argument from the two-list formulation, whether the validated operating range extends to other data types, within or outside biology, or document where it fails.

References

1. Antosh M, Fox D, Cooper LN, Neretti N. CORaL: Comparison of Ranked Lists for Analysis of Gene Expression Data. Journal of Computational Biology, 2013. DOI: 10.1089/cmb.2013.0017. URL: https://doi.org/10.1089/cmb.2013.0017
2. Plaisier SB, Taschereau R, Wong JA, Graeber TG. Rank-rank hypergeometric overlap: identification of statistically significant overlap between gene-expression signatures. Nucleic Acids Research, 2010. DOI: 10.1093/nar/gkq636. URL: https://doi.org/10.1093/nar/gkq636
3. Yang X, Bentink S, Scheid S, Spang R. Similarities of ordered gene lists. Journal of Bioinformatics and Computational Biology, 2006. DOI: 10.1142/S0219720006002120. URL: https://doi.org/10.1142/S0219720006002120
4. Benjamini Y, Yekutieli D. The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 2001. DOI: 10.1214/aos/1013699998. URL: https://doi.org/10.1214/aos/1013699998
5. Ghent AW. A method for exact testing of 2X2, 2X3, 3X3, and other contingency tables, employing binomial coefficients. American Midland Naturalist, 1972. DOI: 10.2307/2424485. URL: https://doi.org/10.2307/2424485
6. Bauer J, Antosh M, Chang C, Schorl C, Kolli S, Neretti N, Helfand SL. Comparative transcriptional profiling identifies takeout as a gene that regulates life span. Aging, 2010. DOI: 10.18632/aging.100146. URL: https://doi.org/10.18632/aging.100146
7. Pearson KJ, Baur JA, Lewis KN, Peshkin L, Price NL, Labinskyy N, Swindell WR, Kamara D, Minor RK, Perez E, Jamieson HA, Zhang Y, Dunn SR, Sharma K, Pleshko N, Woollett LA, Csiszar A, Ikeno Y, Le Couteur D, Elliott PJ, Becker KG, Navas P, Ingram DK, Wolf NS, Ungvari Z, Sinclair DA, de Cabo R. Resveratrol delays age-related deterioration and mimics transcriptional aspects of dietary restriction without extending life span. Cell Metabolism, 2008. DOI: 10.1016/j.cmet.2008.06.011. URL: https://doi.org/10.1016/j.cmet.2008.06.011
8. Chen Q, Zhou XJ, Sun F. Finding genetic overlaps among diseases based on ranked gene lists. Journal of Computational Biology, 2015. DOI: 10.1089/cmb.2014.0149. URL: https://doi.org/10.1089/cmb.2014.0149
9. Cahill KM, Huo Z, Tseng GC, Logan RW, Seney ML. Improved identification of concordant and discordant gene expression signatures using an updated rank-rank hypergeometric overlap approach. Scientific Reports, 2018. DOI: 10.1038/s41598-018-27903-2. URL: https://doi.org/10.1038/s41598-018-27903-2
10. Serra F, Romualdi C, Fogolari F. Similarity measures based on the overlap of ranked genes are effective for comparison and classification of microarray data. Journal of Computational Biology, 2016. DOI: 10.1089/cmb.2015.0057. URL: https://doi.org/10.1089/cmb.2015.0057
11. Donald MR, Wilson SR. Comparison and visualisation of agreement for paired lists of rankings. Statistical Applications in Genetics and Molecular Biology, 2017. DOI: 10.1515/sagmb-2016-0036. URL: https://doi.org/10.1515/sagmb-2016-0036
12. Sánchez-Pla A, Salicrú M, Ocaña J. An equivalence approach to the integrative analysis of feature lists. BMC Bioinformatics, 2019. DOI: 10.1186/s12859-019-3008-x. URL: https://doi.org/10.1186/s12859-019-3008-x
13. Luo D, Zhang C, Fu L, Zhang Y, Hu Y-Q. A novel similarity score based on gene ranks to reveal genetic relationships among diseases. PeerJ, 2021. DOI: 10.7717/peerj.10576. URL: https://doi.org/10.7717/peerj.10576
14. Hajdarovic KH, Yu D, Hassell L-A, Evans SA, Packer S, Neretti N, Webb A. Single-cell analysis of the aging female mouse hypothalamus. Nature Aging, 2022. DOI: 10.1038/s43587-022-00246-4. URL: https://doi.org/10.1038/s43587-022-00246-4
15. Piron A, Szymczak F, Papadopoulou T, Alvelos MI, Defrance M, Lenaerts T, Eizirik DL, Cnop M. RedRibbon: A new rank-rank hypergeometric overlap for gene and transcript expression signatures. Life Science Alliance, 2024; 7(2):e202302203 (published online December 2023). DOI: 10.26508/lsa.202302203. URL: https://doi.org/10.26508/lsa.202302203