Veech groups beyond genus two: limit sets in higher genus, higher-degree trace fields, and cusp sets of triangle groups

The Veech group $\mathrm{SL}(X,\omega)$ of a translation surface — the group of derivatives of its affine automorphisms, a Fuchsian group — controls the renormalization of its straight-line flows and polygonal billiards. McMullen's 2003 study of Teichmüller geodesics of infinite complexity classified the limit sets of genus-two Veech groups and left open how the theory extends beyond its two anchors, genus two and real quadratic trace fields. The subsequent literature has settled the genus-two classification itself: lattice Veech groups arise exactly from torus covers, from the Weierstrass curves $W_D$ of double-zero eigenforms, and from the regular decagon, while every other genus-two Veech group with a hyperbolic element is infinitely generated. Three linked fronts remain open: the possible limit sets of non-elementary Veech groups of closed surfaces of genus at least three — in particular whether any such Veech group of the second kind exists and whether every finitely generated such Veech group is a lattice; the structure of Veech groups whose trace fields have degree at least three, where the Galois-flux reduction is unavailable; and the cusp arithmetic in trace degree at least three, notably whether a lattice Veech group with trace field $K$ whose cusp cross-ratios exhaust $\mathbf{P}^1(K)-\{0,1,\infty\}$ must have $\deg(K/\mathbb{Q})\le 2$ (McMullen's 2022 Question 3).

Background

A translation surface is a compact Riemann surface $X$ together with a nonzero holomorphic 1-form $\omega$; concretely it is obtained by gluing finitely many polygons in the complex plane along parallel sides by translations, with $\omega$ descended from the differential $dz$. The pairs of genus $g$ form a bundle $\Omega M_g$ over the moduli space $M_g$ of compact Riemann surfaces, and the locus of forms whose zeros have prescribed orders $d_1,\dots,d_k$ is the stratum $\Omega(d_1,\dots,d_k)$. The group $\mathrm{SL}_2(\mathbb{R})$ acts on $\Omega M_g$ by applying real-linear maps to the polygon charts. The stabilizer of $(X,\omega)$ under this action,

$$\mathrm{SL}(X,\omega)=\{A\in\mathrm{SL}_2(\mathbb{R}) : A\cdot(X,\omega)\cong(X,\omega)\},$$

is the Veech group: it is the group of derivatives of the orientation-preserving affine automorphisms of $(X,\omega)$, and it is a discrete subgroup of $\mathrm{SL}_2(\mathbb{R})$, i.e.\ a Fuchsian group (a discrete group of isometries of the hyperbolic plane, here the upper half-plane). The image in $M_g$ of the orbit $\mathrm{SL}_2(\mathbb{R})\cdot(X,\omega)$ is a complex geodesic for the Teichmüller metric. When $\mathrm{SL}(X,\omega)$ is a lattice, meaning a discrete group of finite covolume, this image is an algebraic curve called a Teichmüller curve, and the surface satisfies Veech's dichotomy: in every direction, the straight-line flow is either completely periodic (every trajectory closes up) or uniquely ergodic (every orbit is dense and there is a unique invariant transverse measure). Surfaces with lattice Veech group are called lattice surfaces or Veech surfaces (Veech 1989).

The limit set $\Lambda(\Gamma)\subset S^1$ of a Fuchsian group $\Gamma$ is the set of accumulation points on the circle at infinity of any orbit $\Gamma z$ with $z$ in the upper half-plane. The group is elementary if $\Lambda(\Gamma)$ contains at most two points and non-elementary otherwise; a non-elementary group is of the first kind if $\Lambda(\Gamma)=S^1$ and of the second kind if $\Lambda(\Gamma)$ is a proper closed subset. For a non-elementary Fuchsian group, being a lattice is equivalent to being finitely generated and of the first kind. Hence for Veech groups the classification of limit sets and the lattice question are one problem: a non-lattice Veech group is either elementary, infinitely generated of the first kind, or non-elementary of the second kind — and no example of the last type is known for closed surfaces.

The source of the present problem is McMullen's 2003 study of Teichmüller geodesics of infinite complexity, which settled the genus-two case: for $(X,\omega)$ of genus two, the limit set of $\mathrm{SL}(X,\omega)$ is either empty, a single point, or the full circle, and there are infinitely many genus-two surfaces with infinitely generated Veech group, such as the surface glued from an S-shaped chain of three squares of side lengths $1$, $1+a$ and $a$ for suitable irrational $a$. The proof introduced the Galois flux, defined as follows. An interval exchange transformation is a piecewise-translation self-map of an interval $I=\bigcup_j I_j$ that translates each subinterval $I_j$ by a real number $t_j$; it arises as the first return of a directional flow on a translation surface to a transversal. Its Sah–Arnoux–Fathi invariant $\sum_j|I_j|\otimes t_j$ (an element of $\mathbb{R}\otimes\mathbb{R}$ modulo the diagonal) classifies interval exchanges up to conjugacy and vanishes only in exceptional cases. When the translation lengths lie in a real quadratic field $K$ with nontrivial Galois automorphism $x\mapsto x'$, the flux $\mathrm{flux}(f)=\sum_j|I_j|\,t_j'$ is a one-dimensional projection of this invariant. A uniquely ergodic exchange has nonzero flux, so vanishing flux forces non-unique ergodicity, which by Masur's criterion drives the Teichmüller geodesic out of every compact set of moduli space and produces short saddle connections (geodesic segments joining zeros of $\omega$ with none in their interior); in genus two these force parabolic elements in the Veech group, and the Weierstrass points (the fixed points of the hyperelliptic involution, i.e.\ of the central symmetry of the double cover of the sphere) then generate enough parabolics to fill the boundary circle. The reduction of the general case to vanishing complex flux uses the Galois involution of the real quadratic trace field and has no known analogue over fields of higher degree. The paper closes by noting that infinitely generated examples appear to be ubiquitous in genus two, and that at the time the only known lattice cases with irrational trace field were those associated to the regular decagon.

In genus two the classification is now complete: lattice Veech groups arise exactly from branched covers of a torus ramified over torsion points (trace field $\mathbb{Q}$), from the components of the Weierstrass curves $W_D$, and from the decagon. Here $W_D$ is the locus of genus-two forms with a single double zero that are eigenforms for real multiplication by the quadratic order of discriminant $D$: real multiplication means an action of that order (a rank-two subring of a real quadratic field) as endomorphisms of the Jacobian, a principally polarized abelian surface, and an eigenform is one lying in a one-dimensional eigenspace for this action. The trace field $\mathbb{Q}(\mathrm{tr}\,\Gamma)$ of a Veech group $\Gamma$, i.e.\ the field generated over $\mathbb{Q}$ by the traces of its elements, has degree at most the genus of the surface, so trace fields of degree at least three first occur in genus three. For them the flux machinery is unavailable, while lattice examples do exist: Bouw and Möller constructed Teichmüller curves uniformized by the triangle groups $\Delta(m,n,\infty)$ for every $m<n$, whose trace fields are generally of degree greater than two.

The arithmetic side of the problem concerns triangle groups. The triangle group $\Delta(p,q,r)$ is the orientation-preserving subgroup of the group generated by reflections in the sides of a hyperbolic triangle with angles $\pi/p$, $\pi/q$ and $\pi/r$; a signature entry $\infty$ records an ideal vertex (angle zero), and the Hecke groups are the family $\Delta(2,q,\infty)$, with parameter $\lambda_q=2\cos(\pi/q)$. A cusp of such a group is an orbit of ideal vertices, equivalently the fixed point of a parabolic element (an isometry with exactly one fixed point on the boundary). The set of cross-ratios of cusps — the cross-ratio being the classical projective invariant of a quadruple of distinct boundary points — measures how close the non-arithmetic group comes to the arithmetic configuration over its invariant trace field $\mathbb{Q}(\mathrm{tr}\,g^2 : g\in\Gamma)$, a commensurability invariant (invariant under passing between commensurable discrete groups). Leutbecher determined the cusp cross-ratios for the Hecke groups of signatures $(2,n,\infty)$ with $n=5,8,10,12$, and Seibold for $(3,n,\infty)$ with $n=4,5,6$; the source paper extended this to signatures $(2,2n,\infty)$, $(3,n,\infty)$ with $n=4,5,6$, and $(2,5,\infty)$, via its Theorem A.1: a lattice Veech group with real quadratic trace field $K$ has cusp cross-ratios exactly $\mathbf{P}^1(K)-\{0,1,\infty\}$. The general quadratic case is now settled (Panti 2020; McMullen 2022). The open residue is degree at least three: height-descent arguments fail there because the Northcott property (finiteness of the set of algebraic numbers of given degree and bounded height) fails over fields whose group of units has rank at least two, and the Hecke groups $\Delta(2,q,\infty)$ of degree at least three are known to have nonperiodic directions in their trace fields (Arnoux–Schmidt).

Prym eigenforms in genus three, four or five are forms $(X,\omega)$ equipped with an involution whose associated Prym variety — the complementary abelian subvariety of the Jacobian determined by the involution — admits real multiplication with the symmetrized form as an eigenform; these loci are algebraic and closely related to strata of quadratic differentials. Bouillabaisse or Thurston–Veech surfaces are forms obtained from a pair of transverse measured foliations with a prescribed integral intersection matrix (Thurston's construction of pseudo-Anosov maps), whose Veech groups contain two transverse parabolic elements. Finally, for infinite-type translation surfaces — non-compact surfaces of infinite genus, outside the category of closed surfaces considered here — Veech groups of the second kind do exist (the Chamanara surface, built from a unit square whose sides are cut into segments of lengths $2^{-n}$, $n=1,2,\dots$, with parallel segments of equal length identified by translations), which delimits the question.

Problem Statement

The genus-two portion of the source question is resolved and is not part of this problem. The problem is the surviving open core, whose single objective is to extend the structure theory of Veech groups established in genus two and over real quadratic fields to closed translation surfaces and triangle groups in general. It comprises three linked strands—the sub-questions enumerated within a strand spell out that strand's general demand—and the strands are joined by this common objective.

1. Limit sets in genus at least three. Determine the possible limit sets of the Veech group $\mathrm{SL}(X,\omega)$ of a closed translation surface $(X,\omega)$ of genus at least three whose Veech group is non-elementary. In particular: (a) Does there exist such a surface for which $\mathrm{SL}(X,\omega)$ is of the second kind, i.e.\ its limit set is a proper closed subset of the circle containing more than two points? (b) Is every finitely generated non-elementary Veech group of a closed translation surface of genus at least three a lattice? A positive answer to (b) is equivalent to: every finitely generated non-elementary Veech group of a closed translation surface of genus at least three is of the first kind; a finitely generated second-kind example for (a) would refute (b). The genus-two answer (empty, a singleton, or the full circle) is known, as is its extension to some Prym eigenform loci in genus 3, 4, 5 and, partially, to bouillabaisse surfaces in $\mathcal{Q}(1,1,1,1)$; the question is open for the general strata. The scope is closed (finite-type) surfaces, since second-kind Veech groups of infinite-type surfaces exist (the Chamanara surface) and are understood as a different phenomenon.

2. Trace fields of degree at least three. Determine which of the genus-two dichotomies—the structure alternative that a Veech group with a hyperbolic element is either a lattice or infinitely generated, and the limit-set alternative that the limit set is empty, a single point, or the full circle—persist for translation surfaces whose Veech group has trace field of degree at least three (which first occurs in genus three, since the trace-field degree is bounded by the genus): in particular, whether infinitely generated Veech groups are ubiquitous among such surfaces, and what replaces the Galois-flux reduction of the source paper, which relies on the Galois involution of a real quadratic field. The lattice cases are known to exist (the Bouw–Möller triangle-group Teichmüller curves); the non-lattice structure theory outside the Prym eigenform and bouillabaisse loci is open.

3. Cusp arithmetic in degree at least three. Determine the cusp sets of triangle groups with invariant trace field of degree at least three, and more generally resolve the following question of McMullen (2022): if $\mathrm{SL}(X,\omega)$ is a lattice Veech group with trace field $K$ (which for a lattice Veech group coincides with the invariant trace field) and the cross-ratios of its cusps coincide with $\mathbf{P}^1(K)-\{0,1,\infty\}$, does it follow that $\deg(K/\mathbb{Q})\le 2$? The case of the Hecke groups $\Delta(2,q,\infty)$ is known; Winsor's 2026 results on the orbit structure of $G_{18}$ on $\lambda_q\mathbb{Q}(\lambda_q^2)$ show that the higher-degree orbit problem is rich but do not settle the question.

References

1. Curtis T. McMullen, Teichmüller geodesics of infinite complexity, Acta Mathematica 191 (2003), 191–223, DOI 10.1007/BF02392964, https://doi.org/10.1007/BF02392964
2. William A. Veech, Teichmüller curves in moduli space, Eisenstein series and an application to triangular billiards, Inventiones Mathematicae 97 (1989), 553–583, DOI 10.1007/BF01388890, https://doi.org/10.1007/BF01388890
3. Curtis T. McMullen, Billiards and Teichmüller curves on Hilbert modular surfaces, Journal of the American Mathematical Society 16 (2003), 857–885, DOI 10.1090/S0894-0347-03-00432-6, https://doi.org/10.1090/S0894-0347-03-00432-6
4. Curtis T. McMullen, Teichmüller curves in genus two: The decagon and beyond, Journal für die reine und angewandte Mathematik 582 (2005), 173–199, DOI 10.1515/crll.2005.2005.582.173, https://doi.org/10.1515/crll.2005.2005.582.173
5. Curtis T. McMullen, Teichmüller curves in genus two: Discriminant and spin, Mathematische Annalen 333 (2005), 87–130, DOI 10.1007/s00208-005-0666-y, https://doi.org/10.1007/s00208-005-0666-y
6. Curtis T. McMullen, Teichmüller curves in genus two: Torsion divisors and ratios of sines, Inventiones Mathematicae 165 (2006), 651–672, DOI 10.1007/s00222-006-0511-2, https://doi.org/10.1007/s00222-006-0511-2
7. Curtis T. McMullen, Dynamics of SL(2,R) over moduli space in genus two, Annals of Mathematics 165 (2007), 397–456, DOI 10.4007/annals.2007.165.397, https://doi.org/10.4007/annals.2007.165.397
8. Pascal Hubert and Thomas A. Schmidt, Infinitely generated Veech groups, Duke Mathematical Journal 123 (2004), 49–69, DOI 10.1215/S0012-7094-04-12312-8, https://doi.org/10.1215/S0012-7094-04-12312-8
9. Pascal Hubert and Thomas A. Schmidt, Geometry of infinitely generated Veech groups, Conformal Geometry and Dynamics 10 (2006), 1–20, DOI 10.1090/S1088-4173-06-00120-2, arXiv math/0410132, https://arxiv.org/abs/math/0410132
10. Erwan Lanneau and Duc-Manh Nguyen, Complete periodicity of Prym eigenforms, Annals of the École Normale Supérieure 49 (2016), 87–130, DOI 10.24033/asens.2277, https://doi.org/10.24033/asens.2277
11. Pascal Hubert, Erwan Lanneau and Martin Möller, Completely periodic directions and orbit closures of many pseudo-Anosov Teichmüller discs in Q(1,1,1,1), Mathematische Annalen 353 (2012), 1–35, DOI 10.1007/s00208-011-0670-3, https://doi.org/10.1007/s00208-011-0670-3
12. Ralf Lehnert, On the critical exponent of infinitely generated Veech groups, Mathematische Annalen 368 (2017), 1017–1058, DOI 10.1007/s00208-016-1462-6, arXiv 1404.2110, https://arxiv.org/abs/1404.2110
13. Eliot Bongiovanni, Extensions of finitely generated Veech groups, Algebraic & Geometric Topology 26 (2026), 989–1035, DOI 10.2140/agt.2026.26.989, arXiv 2406.11090, https://doi.org/10.2140/agt.2026.26.989
14. Irene I. Bouw and Martin Möller, Teichmüller curves, triangle groups, and Lyapunov exponents, Annals of Mathematics 172 (2010), 139–185, DOI 10.4007/annals.2010.172.139, arXiv math/0511738, https://arxiv.org/abs/math/0511738
15. Giovanni Panti, Decreasing height along continued fractions, Ergodic Theory and Dynamical Systems 40 (2020), 763–788, DOI 10.1017/etds.2018.55, arXiv 1802.09378, https://arxiv.org/abs/1802.09378
16. Curtis T. McMullen, Billiards, heights, and the arithmetic of non-arithmetic groups, Inventiones Mathematicae 228 (2022), 1309–1351, DOI 10.1007/s00222-022-01101-4, https://doi.org/10.1007/s00222-022-01101-4
17. Curtis T. McMullen, Triangle groups and Hilbert modular varieties, Journal of the Mathematical Society of Japan 78 (2026), 729–749, DOI 10.2969/jmsj/94179417, https://doi.org/10.2969/jmsj/94179417
18. Pierre Arnoux and Thomas A. Schmidt, Veech surfaces with nonperiodic directions in the trace field, Journal of Modern Dynamics 3 (2009), 611–629, DOI 10.3934/jmd.2009.3.611, https://doi.org/10.3934/jmd.2009.3.611
19. Pascal Hubert and Erwan Lanneau, Veech groups without parabolic elements, Duke Mathematical Journal 133 (2006), 335–346, DOI 10.1215/S0012-7094-06-13326-4, https://doi.org/10.1215/S0012-7094-06-13326-4
20. Frank Herrlich and Anja Randecker, Notes on the Veech group of the Chamanara surface, 2016, arXiv 1612.06877, https://arxiv.org/abs/1612.06877
21. Karl Winsor, Hecke Triangle Groups and Special Hyperbolic Elements, 2026, arXiv 2605.30064, https://arxiv.org/abs/2605.30064
22. Armin Leutbecher, Über die Heckeschen Gruppen G(λ) II, Mathematische Annalen 211 (1974), 63–86, DOI 10.1007/BF01344143, https://doi.org/10.1007/BF01344143
23. F. Seibold, Zahlentheoretische Eigenschaften der Heckeschen Gruppen G(λ) und verwandter Transformationsgruppen, Dissertation, Technische Universität München, 1985 (no DOI)