A 2025 paper by Brilhante, Gomes, Mendonça, Pestana and Santos (Journal of Statistical Theory and Practice 19(2):29) formalizes 'fake' p-values — values reported as the minimum of two independent replicates and hence Beta(1,2)-distributed rather than Uniform(0,1) — together with the Mendel(m) mixture family for samples combining genuine and fake p-values in combined-test meta-analysis. The paper leaves open the twin problems: (1) statistically reliable detection and estimation — deciding whether fake p-values are present, estimating how many there are, and where possible identifying which ones, with quantified error rates; and (2) exact closed-form or computationally efficient and accurate null distributions and critical values for the standard combination statistics (Fisher, Stouffer, Pearson, logit, Tippett) under an arbitrary number of fakes and moderate-to-large numbers of studies, where closed forms are known only for at most two fakes and current simulation tables stop at n ≤ 26 with at most n/3 fakes.

Background

In meta-analysis by combination of p-values, n independent studies each test a null hypothesis H0kH_{0k} against an alternative, and their p-values p1,…,pnp_1,\dots,p_n are pooled into a single statistic T(p1,…,pn)T(p_1,\dots,p_n) for the overall null hypothesis H0∗H_0^* (all H0kH_{0k} true) versus the alternative that at least one is false. Under H0∗H_0^*, if every study is honestly reported, the p-values behave like independent Uniform(0,1) random variables, and many combination statistics then have exactly known null distributions: Fisher's method uses TF=−2∑k=1nln⁡PkT_F=-2\sum_{k=1}^n \ln P_k, which is chi-square with 2n2n degrees of freedom; Stouffer's statistic sums the standard-normal quantiles Φ−1(Pk)\Phi^{-1}(P_k) scaled by n\sqrt{n}; Pearson's method uses a product of the p-values; Tippett's method uses their minimum; and the logit combination statistic applies ln⁡(Pk/(1−Pk))\ln(P_k/(1-P_k)).

This classical theory assumes all reported p-values are genuine. The source paper builds on a long-running concern that dates to Fisher's 1936 re-analysis of Gregor Mendel's pea-breeding data, where Fisher judged the agreement between data and theory too close to be credible and suspected that only results agreeing with expectations had been reported. A modern, structural version of this selective-reporting mechanism works as follows: publication pressure favors significant results, so if a first experiment yields a non-significant p-value, a researcher may repeat the experiment once and report only the better of the two. Such a reported value is an observation of the minimum of two independent Uniform(0,1) variables, which follows the Beta(1,2) distribution (the beta distribution with density 2(1−x)2(1-x) on (0,1)(0,1), i.e. the smallest order statistic of two uniforms); the paper calls it a fake p-value. Fakes are stochastically smaller than genuine p-values, so their presence shifts the null distribution of any combination statistic and can invalidate nominal critical values.

To model mixtures of genuine and fake p-values, the paper introduces the Mendel(m) family: a Mendel(m) random variable has density mx+(1−m/2)m x + (1-m/2) on (0,1)(0,1) for m∈[−2,2]m\in[-2,2], so m=0m=0 is the uniform, m=−2m=-2 is Beta(1,2) (the all-fake side, mixing in a minimum-of-two component of weight ∣m∣/2|m|/2 for m<0m<0), and m=2m=2 is Beta(2,1), the maximum of two uniforms. The framework descends from a statistical model to explain the Mendel–Fisher controversy proposed by Pires and Branco (2010), and from earlier work estimating the overall proportion of non-uniform reported p-values (Brilhante, Pestana, Semblano and Sequeira, 2015).

What is already known within this framework (from the 2024–2025 work of Brilhante, Gomes, Mendonça, Pestana and Santos): Tippett's minimum statistic has an exact distribution for any known number nfn_f of fakes; Pearson's product statistic has exact density results for exactly one and for exactly two fakes; Fisher's statistic admits closed-form mixture expressions when all p-values are iid Mendel(m), via the n-fold convolution of finite mixtures, extending a convolution theorem of Sarabia, Prieto and Trueba (2012); and a uniformity characterization of Deng and George (1992), based on contractions of products and sums of random variables, extends to Mendel(m) variables and supports tests of independence. Beyond these cases, closed-form cumulative distribution functions are unavailable, and the current practical workaround is a set of simulation-based critical-value tables (Santos, Brilhante and Mendonça, 2024) covering only n=3,…,26n=3,\dots,26 studies and at most nf≤n/3n_f\le n/3 fakes, used to compare the decisions of several combination tests across contamination scenarios.

A large adjacent literature detects selective reporting at the level of a whole collection of published results rather than labeling individual p-values: the p-curve approach of Simonsohn, Nelson and Simmons (2014) tests whether reported significant p-values have the distribution expected under true effects; the excess-significance test of Ioannidis and Trikalinos (2007) asks whether significant findings occur more often than the studies' power predicts; the text-mining analysis of Head, Holman, Lanfear, Kahn and Jennions (2015) looks for an excess of p-values just below 0.05, a signature of p-hacking (collecting or selecting analyses until a non-significant result becomes significant). These methods condition on published, typically significant results and deliver a collection-level verdict; they do not model the full-support minimum-of-two mechanism studied here, do not identify which p-values in a given set are fake, and do not provide contamination-robust critical values for combination statistics.

Problem Statement

Within the model of the source paper: under the overall null hypothesis H0∗H_0^*, the nn independent p-values P1,…,PnP_1,\dots,P_n to be combined would each be genuine, i.e. Uniform(0,1), except that an unknown subset of them may be fake — each fake being the minimum of two independent Uniform(0,1) replicates and hence Beta(1,2) with density 2(1−x)2(1-x) on (0,1)(0,1) — so the sample is a mixture in the Mendel(m) family with unknown contamination. The open problem, whose two parts jointly serve one objective — a valid and informative overall decision on H0∗H_0^* from the combined statistic T(P1,…,Pn)T(P_1,\dots,P_n) when the presence, number, and location of fake p-values are all unknown — is:

(1) Detection and estimation. Develop statistically reliable procedures, with quantified error rates, for deciding whether fake p-values are present in an observed sample p1,…,pnp_1,\dots,p_n; for estimating their number nfn_f (equivalently, under exchangeable contamination, the fake-component weight ∣m∣/2=nf/n|m|/2=n_f/n of the Mendel(m) mixture); and, to the extent statistically achievable, for identifying which of the observed p-values are fake. This includes characterizing the achievable performance envelope under this two-component mixture model (what presence-detection, number-estimation, and labeling accuracies are possible, and what is impossible).

(2) Contamination-robust combination. Obtain exact closed-form, or computationally efficient and provably accurate, null distributions and critical values for the standard combined statistics — Fisher's TF=−2∑kln⁡PkT_F=-2\sum_k \ln P_k, Stouffer's statistic, Pearson's product statistic, the logit combination statistic, and Tippett's minimum — when an arbitrary number nfn_f of the nn p-values are Beta(1,2) fakes, covering moderate-to-large nn. This must go beyond the currently known closed forms for at most two fakes and beyond the simulation tables that cover only n≤26n\le 26 studies with nf≤n/3n_f\le n/3 fakes, and should either not require knowledge of nfn_f or make explicit how the critical values depend on it. (For Tippett's minimum an exact null distribution is already known for any known nfn_f, so for that statistic the open gap is specifically the unknown-nfn_f setting; for the remaining statistics the gap covers arbitrary known nfn_f as well.)

References

  1. Brilhante MF, Gomes MI, Mendonça S, Pestana D, Santos R (2025) Meta-analysis of Genuine and Fake p-Values. Journal of Statistical Theory and Practice 19(2):29. DOI 10.1007/s42519-025-00445-3. https://doi.org/10.1007/s42519-025-00445-3
  2. Fisher RA (1936) Has Mendel's work been rediscovered? Annals of Science 1(2):115-137. DOI 10.1080/00033793600200111. https://doi.org/10.1080/00033793600200111
  3. Pires AM, Branco JA (2010) A Statistical Model to Explain the Mendel–Fisher Controversy. Statistical Science 25(4):545-565. DOI 10.1214/10-STS342. https://doi.org/10.1214/10-STS342
  4. Brilhante MF, Pestana D, Semblano P, Sequeira F (2015) On the proportion of non uniform reported p-values. AIP Conference Proceedings 1648:540003. DOI 10.1063/1.4912749. https://doi.org/10.1063/1.4912749
  5. Brilhante MF, Gomes MI, Mendonça S, Pestana D, Santos R (2024) Two P or Not Two P: Mendel Random Variables in Combining Fake and Genuine p-Values. AppliedMath 4(3):1128-1142. DOI 10.3390/appliedmath4030060. https://doi.org/10.3390/appliedmath4030060
  6. Santos R, Brilhante MF, Mendonça S (2024) Tables with Critical Values for the Meta-Analysis of Genuine and Fake p-Values. arXiv:2410.04651. https://arxiv.org/abs/2410.04651
  7. Sarabia JM, Prieto F, Trueba C (2012) The n-fold convolution of a finite mixture of densities. Applied Mathematics and Computation 218(19):9992-9996. DOI 10.1016/j.amc.2012.03.060. https://doi.org/10.1016/j.amc.2012.03.060
  8. Deng L-Y, George EO (1992) Some characterizations of the uniform distribution with applications to random number generation. Annals of the Institute of Statistical Mathematics 44:379-385. DOI 10.1007/BF00058647. https://doi.org/10.1007/BF00058647
  9. Fleiss JL (1993) Review papers: the statistical basis of meta-analysis. Statistical Methods in Medical Research 2(2):121-145. DOI 10.1177/096228029300200202. https://doi.org/10.1177/096228029300200202
  10. Simonsohn U, Nelson LD, Simmons JP (2014) P-curve: A key to the file-drawer. Journal of Experimental Psychology: General 143(2):534-547. DOI 10.1037/a0033242. https://doi.org/10.1037/a0033242
  11. Ioannidis JPA, Trikalinos TA (2007) An exploratory test for an excess of significant findings. Clinical Trials 4(3):245-253. DOI 10.1177/1740774507079441. https://doi.org/10.1177/1740774507079441
  12. Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD (2015) The Extent and Consequences of P-Hacking in Science. PLOS Biology 13(3):e1002106. DOI 10.1371/journal.pbio.1002106. https://doi.org/10.1371/journal.pbio.1002106
  13. Faridani S (2025) When is p-hacking detectable? arXiv:2506.20035. https://arxiv.org/abs/2506.20035