Open questions from Serre's 'Quelques applications du théorème de densité de Chebotarev' (1981): Frobenius counting on thin sets, removal of GRH, Serre's uniformity problem, lacunarity of Fourier coefficients, and the Lang–Trotter asymptotic

In his 1981 IHÉS paper applying the Chebotarev density theorem to $\ell$-adic Galois representations, Jean-Pierre Serre proved a general upper bound for the number of primes whose Frobenius conjugacy class lies in a fixed lower-dimensional subset $C$ of a compact $\ell$-adic Lie group, and applied it to Fourier coefficients of modular forms and to elliptic curves. He then posed a series of explicit open questions and recorded related conjectural targets: (a) whether the counting function $\pi_C(x)$ for a proper analytic subset $C$ satisfies the sharp bound $O(x^{1/2}|\log x|)$, even under GRH; (b) whether the polylogarithmic bounds on the least prime distinguishing two non-isogenous elliptic curves, or on the least $\ell$ forcing surjectivity of $\rho_\ell$, can be made unconditional; (c) whether $\ell \ge 41$ alone implies surjectivity of $\rho_\ell$ (the strongest form of Serre's uniformity conjecture); (d) analogues for automorphic forms on other Fuchsian groups; (e) bounds on gaps between vanishing Fourier coefficients; (f) the Lang–Trotter asymptotic $P_{E,h}(x) \sim c_{E,h}x^{1/2}/\log x$ for the primes $p$ with $a_p(E) = h$ (fixed $h$), a conjecture of Lang and Trotter cited in the paper and discussed after Théorème 20; and (g) convergence of $x^{-1}M_f(x)$ for forms $f$ in the space $M(N,k,\omega)$ treated in Théorème 17. The task is to make progress on these open questions.

Background

Let $K$ be a number field and $\mathcal{G}$ a compact $\ell$-adic Lie group (a closed subgroup of $\mathrm{GL}_n(\mathbb{Z}_\ell)$, where $\mathbb{Z}_\ell$ is the ring of $\ell$-adic integers) of dimension $N$, arising as the image of a continuous Galois representation $\rho: \mathrm{Gal}(\bar{K}/K) \to \mathcal{G}$ unramified outside a finite set $S$ of places. For a closed subset $C \subseteq \mathcal{G}$ stable under conjugation, one studies the counting function $\pi_C(x) = \#\{\mathfrak{p} : N\mathfrak{p} \le x,\ \mathfrak{p} \notin S,\ \rho(\mathrm{Frob}_\mathfrak{p}) \in C\}$, where $\mathrm{Frob}_\mathfrak{p}$ denotes the Frobenius conjugacy class at the prime $\mathfrak{p}$ and $N\mathfrak{p}$ its absolute norm. When $C$ has positive Haar measure, the Chebotarev density theorem determines $\pi_C(x)$ asymptotically; the interesting regime is when $C$ is a lower-dimensional 'thin' subset defined by algebraic equations. Such a $C$ has Haar measure zero, so Chebotarev gives no information, and the conjectural truth (Lang–Trotter in the trace case) is of order $x^{1/2}/\log x$, far below what the known bounds reach.

Serre's Théorème 10 bounds $\pi_C(x)$ by $O(\mathrm{Li}(x)/\varepsilon(x)^{\alpha})$ — a power-of-logarithm saving over the logarithmic integral $\mathrm{Li}(x)$ — for $C$ of $\ell$-adic analytic dimension at most $d < N$, with $\alpha = (N-d)/N$ and $\varepsilon(x) = \log x\,(\log\log x)^{-2}(\log\log\log x)^{-1}$ (equations (64), (65)); under the Generalized Riemann Hypothesis (GRH) for the relevant $L$-functions, the bound is $O(\mathrm{Li}(x)/\varepsilon_R(x)^{\alpha})$ with $\varepsilon_R(x) = x^{1/2}(\log x)^{-2}$ (equations (66R), (67)). The implied constants depend on $K$, $S$, $\mathcal{G}$, $C$, $d$. The proof uses a large-sieve inequality (an inequality bounding a function on $\mathcal{G}$ by its values on Frobenius elements, analogous to the classical large sieve for residue classes). Théorème 12 improves the exponent to $\beta = (N-d)/(N-r/2)$, where $r = \inf_{s \in C} \dim\,\mathrm{Im}(\mathrm{Ad}(s)-1)$, the dimension of the image of $\mathrm{Ad}(s)-1$ on the Lie algebra of $\mathcal{G}$; one has $\beta \ge \alpha$, with strict inequality exactly when $r \ge 1$, i.e., when no conjugacy class contained in $C$ is finite. Under GRH, these results, combined with the effective Chebotarev bounds of Lagarias–Odlyzko and Oesterlé, become effective in the ramification data — the basis of the §8 applications.

The two main applications are: (i) to Fourier coefficients of modular forms — for a normalized eigenform $f = \sum a_n q^n$ (a simultaneous eigenvector for the Hecke operators) of weight $k \ge 2$ on $\Gamma_0(N)$, Serre studies $M_f(x) = \#\{n \le x : a_n(f) \ne 0\}$, the sets $R_0 = \{f : \lim\inf x^{-1}M_f(x) = 0\} = S_{cm}$ and $R_{1/2} = \{f : \lim\inf x^{-1}M_f(x)(\log x)^{1/2} = 0\} = \emptyset$ (Théorème 17, where $S_{cm}$ denotes the subspace of CM forms — those with $a_p(f) = 0$ for every prime $p$ inert in some imaginary quadratic field, Ribet's criterion; for these $M_f(x) \asymp x/(\log x)^{1/2}$ by Proposition 18 and Théorème 17 (ii), while for non-CM eigenforms Théorème 16 gives $M_f(x) \sim \alpha x$ with $\alpha > 0$) — and the gap function $i_f(n)$, defined in the problem statement; and (ii) to elliptic curves — for $E/\mathbb{Q}$ without CM (i.e., $\mathrm{End}(E) = \mathbb{Z}$), the distribution of the trace of Frobenius $a_p(E) = p + 1 - \#E(\mathbb{F}_p)$, counted by $P_{E,h}(x) = \#\{p \le x : a_p(E) = h\}$ for a fixed integer $h$, and the $\ell$-adic representation $\rho_\ell: \mathrm{Gal}(\bar{\mathbb{Q}}/\mathbb{Q}) \to \mathrm{GL}_2(\mathbb{Z}_\ell)$ attached to the $\ell$-power torsion of $E$.

The paper's §4.5 example — with $K = \mathbb{Q}(\sqrt{-163})$, $\ell = 41$, a one-dimensional $\ell$-adic group obtained from an abelian extension $E/K$ by class field theory, and $C = \{1\}$ — gives unconditionally $\pi_C(x) \sim x^{1/2}/\log x$ (equation (87)), showing that the GRH bound $O(x^{1/2}\log x)$ (equation (86R), the case $d = 0$, $\alpha = 1$ of Théorème 10 (ii)) is optimal in its $x$-exponent. This motivates question (88?): whether $O(x^{1/2}|\log x|)$ holds for every proper analytic subset. The paper's §8 results give, under GRH, polylogarithmic bounds on the least prime $p$ with $a_p(E) \ne a_p(E')$ for non-isogenous $E, E'$ (Théorème 21: $p < c_{34}(\log N_S\,\log\log 2N_S)^{12}$, where $N_S$ is the product of the primes in a finite set $S$ containing the primes of bad reduction; exponent 6 in the quadratic-twist case, Théorème 21'), and on the least $\ell$ guaranteeing surjectivity of $\rho_\ell$ (Théorème 22); both raise the question of what survives without GRH.

Problem Statement

Let $\mathcal{G}$ be a compact $\ell$-adic Lie group of dimension $N$ arising as the image of a Galois representation unramified outside a finite set $S$ of places of a number field $K$, and let $\pi_C(x)$ denote the number of primes $\mathfrak{p} \notin S$ with $N\mathfrak{p} \le x$ whose Frobenius conjugacy class lies in $C$; let $\mathrm{Li}$ denote the logarithmic integral. Serre (1981) proved $\pi_C(x) = O(\mathrm{Li}(x)/\varepsilon(x)^{\alpha})$ with $\alpha = (N-d)/N$ for $C$ of $\ell$-adic analytic dimension at most $d < N$ (Théorème 10, eq. (64)), where $\varepsilon(x) = \log x\,(\log\log x)^{-2}(\log\log\log x)^{-1}$ (eq. (65)); the exponent improves to $\beta = (N-d)/(N-r/2)$ when $r = \inf_{s\in C}\dim\,\mathrm{Im}(\mathrm{Ad}(s)-1) \ge 1$ (Théorème 12, eq. (101)); and under GRH the bound is $\pi_C(x) = O(\mathrm{Li}(x)/\varepsilon_R(x)^{\alpha})$ with $\varepsilon_R(x) = x^{1/2}(\log x)^{-2}$ (eq. (66R), (67)). The following specific questions from the paper are open. They are independent open problems.

(1) (§4.5, question (88?)) Does $\pi_C(x) = O(x^{1/2}|\log x|)$ as $x \to \infty$ hold for every proper analytic subset $C$ of dimension $d < N$ (unconditionally, or at least under GRH)? (Note: this is an upper bound with $|\log x|$, not the asymptotic $x^{1/2}/\log x$ of the example (87).) Serre remarks: 'je ne vois pas comment le démontrer, même sous (GRH)'.

(2) (§8, Remark 3 after Théorème 21', and Question 1 at the end of §8) Can the GRH hypothesis be removed from Théorèmes 21, 21' and 22 — i.e., can one prove unconditionally that the least prime $p$ with $a_p(E) \ne a_p(E')$ for non-isogenous curves $E, E'/\mathbb{Q}$ (bounded under GRH by $p < c_{34}(\log N_S\,\log\log 2N_S)^{12}$, where $N_S$ is the product of the primes in a finite set $S$ containing the primes of bad reduction), or the least $\ell$ with $\rho_\ell$ surjective (bounded under GRH by Théorème 22), is bounded by a power of $\log N_S$ (resp. of $\log N_E$, the conductor of $E$)? Serre: 'Il serait intéressant d'obtenir une majoration effective de $p = p(E, E', S)$ qui n'utilise pas (GRH). Je n'y suis pas parvenu.'

(3) (§8, second question) Does $\ell \ge 41$ alone imply that $\rho_\ell$ is surjective for every non-CM elliptic curve $E/\mathbb{Q}$? A positive answer would establish Serre's uniformity conjecture in its strongest form — the conjecture itself asserts only that there exists some $p_0$ such that the mod-$p$ representation of every non-CM $E/\mathbb{Q}$ is surjective for every prime $p > p_0$, and the question asks whether $p_0 = 37$ suffices. (The known failures of surjectivity for non-CM curves, e.g. images contained in a Borel subgroup — curves with a rational $p$-isogeny — occur only for $p \le 37$.) This remains open even under GRH.

(4) (§7.6, Q1) For $f$ in the space $M(N,k,\omega)$ of modular forms of weight $k \ge 2$, level $N$ and nebentypus $\omega$, treated in Théorème 17 (the sum of the cusp-form space and the Eisenstein series, including the $\mathbb{C}$-span of the normalized eigenforms), with Fourier coefficients $a_n(f)$ and $M_f(x) = \#\{n \le x : a_n(f) \ne 0\}$ (Théorème 16 gives $M_f(x) \sim \alpha x$, $\alpha > 0$, for a single non-CM eigenform, and Proposition 18 the order of growth for CM forms), does $x^{-1}M_f(x)$ have a limit as $x \to \infty$? Does $x^{-1}M_f(x)(\log x)^{1/2}$ have a limit for $f$ in the CM subspace $S_{cm}$?

(5) (§7.6, Q2) For a nonzero cusp form $f = \sum a_n(f) q^n$ of weight $k \ge 2$ that is not of CM type (i.e., $f \notin S_{cm}(N,k,\omega)$, the CM subspace of the source), let $i_f(n)$ be the largest integer $m \ge 0$ such that $a_{n+i}(f) = 0$ for all $0 < i \le m$ (the length of the run of vanishing coefficients starting at $a_{n+1}(f)$); Théorème 17 (i) gives $i_f(n) = O(n)$ (eq. (215)). Does the improvement (216?) $i_f(n) = O(n\,(\log n)^{-m})$ hold for every $m \ge 0$, or even (217?) $i_f(n) = O(n^{\delta})$ for some $\delta < 1$?

(6) (§7.6, Q3) Do the analogues of Théorèmes 15–17 — the bounds on $P_{f,h}(x) = \#\{p \le x : a_p(f) = h(p)\}$ for a polynomial $h$, the asymptotics of $M_f(x)$, and the determination of $R_0, R_{1/2}$ — hold for modular forms of weight $\ge 2$ (not necessarily integral) on other subgroups of $\mathrm{SL}_2(\mathbb{Z})$ or on other Fuchsian groups (discrete subgroups of $\mathrm{SL}_2(\mathbb{R})$ with finite-volume quotient)?

(7) (Implicit in Théorème 20 / Lang–Trotter, eq. (247?)) For a non-CM elliptic curve $E/\mathbb{Q}$ and a fixed integer $h$, is $P_{E,h}(x) \sim c_{E,h}\,x^{1/2}/\log x$, where $c_{E,h} \ge 0$ is the explicit Lang–Trotter constant (a product of local density factors, see Lang–Trotter, LNM 504)? Serre's bounds — $P_{E,h}(x) = O(\mathrm{Li}(x)/\varepsilon(x)^{1/4})$ unconditionally and $O(\mathrm{Li}(x)/\varepsilon_R(x)^{1/4})$ under GRH, with the exponent $1/4$ replaced by $1/3$ for $h \ne \pm 2$ and by $1/2$ for $h = 0$, giving under GRH $P_{E,0}(x) = O(x^{3/4})$ (eq. (246R)) — are, by his own remark, probably not optimal even under GRH, and the conjectural order $x^{1/2}/\log x$ is out of reach.

References

1. Jean-Pierre Serre, 'Quelques applications du théorème de densité de Chebotarev', Publications Mathématiques de l'IHÉS 54 (1981), 123–201, DOI 10.1007/bf02698692, https://doi.org/10.1007/bf02698692
2. Noam D. Elkies, 'The existence of infinitely many supersingular primes for every elliptic curve over Q', Inventiones Mathematicae 89 (1987), 561–567, DOI 10.1007/bf01388985, https://doi.org/10.1007/bf01388985
3. M. I. Knopp and J. Lehner, 'Gaps in the Fourier series of automorphic forms', in Analytic Number Theory, Lecture Notes in Mathematics, Springer (1981), 360–381, DOI 10.1007/bfb0096474, https://doi.org/10.1007/bfb0096474
4. Serge Lang and Hale Trotter, 'Frobenius Distributions in GL2-Extensions', Lecture Notes in Mathematics 504, Springer (1976), DOI 10.1007/bfb0082087, https://doi.org/10.1007/bfb0082087
5. J. C. Lagarias, H. L. Montgomery and A. M. Odlyzko, 'A bound for the least prime ideal in the Chebotarev Density Theorem', Inventiones Mathematicae 54 (1979), 271–296, DOI 10.1007/bf01390234, https://doi.org/10.1007/bf01390234
6. J. C. Lagarias and A. M. Odlyzko, 'Effective versions of the Chebotarev density theorem', in Algebraic Number Fields (Proc. Symp. London Math. Soc., Univ. Durham 1975), 1977, 409–464, Zbl 0362.12011, https://zbmath.org/3563269
7. Joseph Oesterlé, 'Versions effectives du théorème de Chebotarev sous l'hypothèse de Riemann généralisée', in Journées Arithmétiques de Luminy, Astérisque 61 (1979), 165–167, https://numdam.org/item/AST_1979__61__165_0/
8. Yuri Bilu and Pierre Parent, 'Serre's uniformity problem in the split Cartan case', Annals of Mathematics 173 (2011), 569–584, DOI 10.4007/annals.2011.173.1.13 (preprint arXiv:0807.4954), https://doi.org/10.4007/annals.2011.173.1.13
9. Éric Gaudron and Gaël Rémond, 'Théorème des périodes et degrés minimaux d'isogénies', arXiv:1105.1230 (2011), https://arxiv.org/abs/1105.1230
10. Pedro Lemos, 'Some cases of Serre's uniformity problem', arXiv:1808.03110 (2018), https://arxiv.org/abs/1808.03110
11. Samuel Le Fourn and Pedro Lemos, 'Residual Galois representations of elliptic curves with image contained in the normaliser of a non-split Cartan', arXiv:2002.02714 (2020), https://arxiv.org/abs/2002.02714
12. Lorenzo Furio and Davide Lombardo, 'Serre's uniformity question and proper subgroups of C_ns^+(p)', arXiv:2305.17780 (2023), https://arxiv.org/abs/2305.17780
13. Lorenzo Furio, 'Effective bounds for adelic Galois representations attached to elliptic curves over the rationals', arXiv:2412.10340 (2024), https://arxiv.org/abs/2412.10340
14. Jesse Thorner and Asif Zaman, 'An explicit bound for the least prime ideal in the Chebotarev density theorem', Algebra & Number Theory 11 (2017), 1135–1197, DOI 10.2140/ant.2017.11.1135 (preprint arXiv:1604.01750), https://doi.org/10.2140/ant.2017.11.1135
15. Jesse Thorner and Asif Zaman, 'A unified and improved Chebotarev density theorem', arXiv:1803.02823 (2018), https://arxiv.org/abs/1803.02823
16. Asif Zaman, 'Bounding the least prime ideal in the Chebotarev Density Theorem', arXiv:1508.00287 (2015), https://arxiv.org/abs/1508.00287
17. Jeoung-Hwan Ahn and Soun-Hi Kwon, 'An explicit upper bound for the least prime ideal in the Chebotarev density theorem', Annales de l'Institut Fourier 69 (2019), 1411–1458, DOI 10.5802/aif.3274 (preprint arXiv:1807.00508), https://doi.org/10.5802/aif.3274
18. Habiba Kadiri, Nathan Ng and Peng-Jie Wong, 'The least prime ideal in the Chebotarev Density Theorem', arXiv:1902.08640 (2019), https://arxiv.org/abs/1902.08640
19. Daqing Wan and Ping Xi, 'Lang–Trotter Conjecture for CM Elliptic Curves', arXiv:2109.14256 (2021), https://arxiv.org/abs/2109.14256
20. Stephan Baier, 'The Lang-Trotter Conjecture on Average', arXiv:math/0609095 (2006), https://arxiv.org/abs/math/0609095
21. A. C. Cojocaru and I. E. Shparlinski, 'Distribution of Farey Fractions in Residue Classes and Lang–Trotter Conjectures on Average', arXiv:0705.3861 (2007), https://arxiv.org/abs/0705.3861
22. Stephan Baier and Nathan Jones, 'A refined version of the Lang-Trotter Conjecture', arXiv:0801.3946 (2008), https://arxiv.org/abs/0801.3946
23. Hao Chen, Nathan Jones and Vlad Serban, 'The Lang-Trotter Conjecture for products of non-CM elliptic curves', arXiv:2006.11269 (2020), https://arxiv.org/abs/2006.11269
24. Min Sha and Igor E. Shparlinski, 'Lang-Trotter and Sato-Tate Distributions in Single and Double Parametric Families of Elliptic Curves', arXiv:1404.0182 (2014), https://arxiv.org/abs/1404.0182
25. Arvind Kumar, Moni Kumari and Ariel Weiss, 'On the Lang–Trotter conjecture for Siegel modular forms', arXiv:2201.09278 (2022), https://arxiv.org/abs/2201.09278
26. Chihiro Ando, 'The Lang-Trotter conjecture on average for genus-2 curves with Klein-4 reduced automorphism group', arXiv:2408.13695 (2024), https://arxiv.org/abs/2408.13695
27. Chihiro Ando and Shushi Harashita, 'The Lang-Trotter conjecture on average for genus-2 curves with S_3 reduced automorphism group', arXiv:2604.00822 (2026), https://arxiv.org/abs/2604.00822
28. Rainer Dietmann and Christian Elsholtz, 'Longer gaps between values of binary quadratic forms', arXiv:1810.03203 (2018), https://arxiv.org/abs/1810.03203
29. A. B. Kalmynin and S. V. Konyagin, 'Large gaps between sums of two squares', arXiv:1906.09100 (2019), https://arxiv.org/abs/1906.09100
30. Tristan Freiberg, Pär Kurlberg and Lior Rosenzweig, 'Poisson distribution for gaps between sums of two squares and level spacings for toral point scatterers', arXiv:1701.01157 (2017), https://arxiv.org/abs/1701.01157
31. Chun-Yin Hui, 'On distribution of supersingular primes of abelian varieties and K3 surfaces', arXiv:2504.08088 (2025), https://arxiv.org/abs/2504.08088