Realizable Hermitian-invariant symmetry groups $\Gamma_f$ of rational proper maps between balls: characterization and minimal realization cost

For a rational proper holomorphic map $f:\mathbb{B}^n\to\mathbb{B}^N$ between unit balls, the Hermitian invariant group $\Gamma_f$ collects the source automorphisms $\varphi$ that can be compensated by a target automorphism, $\tau\circ f=f\circ\varphi$. Grundmeier and Lebl (2024) proved that, after putting a non-linear map into a canonical normal form, $\Gamma_f$ is exactly the subgroup of $U(n)$ leaving invariant the single polynomial $\rho=|g|^2-\|p\|^2$ underlying $f=p/g$, and conversely that every subgroup of $U(n)$ defined by finitely many invariant polynomials occurs as $\Gamma_f$ of a polynomial proper map. The characterization problem therefore reduces to an invariant-theoretic question that remains open: give an intrinsic criterion for which closed subgroups of $U(n)$ are definable by finitely many invariant polynomials, hence realizable as $\Gamma_f$. The companion optimization problem, posed by D'Angelo and Xiao (2017), asks for the smallest source dimension for which there is a rational map with Hermitian invariant group isomorphic to a given $\Gamma_f$, and for the smallest degree of a rational map whose Hermitian invariant group is a given finite group; both questions are open in general. Related partial results concern possible target dimensions rather than these minima: for $\Gamma_f=U(n)$ the set of possible target dimensions is co-finite in each source dimension (D'Angelo 2022), and for sphere maps invariant under the three fixed-point-free cyclic representation types of the strict invariance group $G_f$ the possible numbers of terms exhibit Frobenius-type gaps (D'Angelo-Grundmeier-Lichtblau 2025).

Background

Let $\mathbb{B}^n=\{z\in\mathbb{C}^n:\|z\|<1\}$ be the unit ball. Its automorphism group $\operatorname{Aut}(\mathbb{B}^n)$ consists of the biholomorphic self-maps, all of them linear-fractional (a matrix-affine numerator divided by a linear functional). A holomorphic map $f:\mathbb{B}^n\to\mathbb{B}^N$ is proper when it maps the boundary sphere onto the boundary sphere; when $f$ is rational it is a quotient $f=p/g$ of a polynomial map $p:\mathbb{C}^n\to\mathbb{C}^N$ by a scalar polynomial $g$ in lowest terms, and boundary-smooth proper maps between balls are automatically rational (Forstnerič, building on Rudin's study of group-invariant maps).

To each rational proper map D'Angelo and Xiao attached several symmetry groups, the central one here being the Hermitian invariant group $\Gamma_f=\{\varphi\in\operatorname{Aut}(\mathbb{B}^n):\ \exists\,\tau\in\operatorname{Aut}(\mathbb{B}^N),\ \tau\circ f=f\circ\varphi\},$ the group of source automorphisms of $f$ that can be compensated by target automorphisms. It is a closed Lie subgroup of $\operatorname{Aut}(\mathbb{B}^n)$ with finitely many connected components. D'Angelo and Xiao proved: $\Gamma_f$ is noncompact exactly when $f$ is linear-fractional, in which case $\Gamma_f=\operatorname{Aut}(\mathbb{B}^n)$; $\Gamma_f$ contains the $n$-torus (diagonal unitaries) exactly when $f$ is spherically equivalent to a monomial map (each component a single monomial); $\Gamma_f$ contains a conjugate of the full unitary group $U(n)$ exactly when $f$ is spherically equivalent to an orthogonal sum of tensor powers $\lambda_1 z^{\otimes m_1}\oplus\cdots\oplus\lambda_k z^{\otimes m_k}\oplus 0$; and every finite subgroup of $\operatorname{Aut}(\mathbb{B}^n)$ occurs as $\Gamma_f$ of some rational proper map. Two maps are spherically equivalent if they differ by source and target automorphisms; this conjugates $\Gamma_f$, so realizability is naturally stated up to conjugacy. For the stricter invariance group $G_f=\{\varphi:f\circ\varphi=f\}$, D'Angelo-Lichtblau and Lichtblau gave the decisive characterization: a finite group admits a nonconstant rational proper ball map invariant under it exactly when the group is cyclic, fixed-point-free, and conjugate to one of three representation types (D'Angelo-Xiao 2018, Proposition 3.5): the cyclic group generated by $\eta I_n$ with $\eta$ a primitive $m$-th root of unity; the cyclic group of odd order generated by $\eta I_j\oplus\eta^2 I_k$ with $j+k=n$; or the cyclic group of order $7$ generated by $\eta I_j\oplus\eta^2 I_k\oplus\eta^4 I_l$ with $j+k+l=n$. These representation types constrain the strict invariance group $G_f$, which can be strictly smaller than $\Gamma_f$: for $f(z)=z^{\otimes m}$, $G_f$ is cyclic while $\Gamma_f=U(n)$ (D'Angelo-Xiao 2017, Example 7.1).

Lebl proved that every rational proper map of balls is spherically equivalent to one in normal form: $f(0)=p(0)=0$, the denominator $g=1+g_2+g_3+\cdots$ has no linear term, and the quadratic part $g_2=\sum_\ell\sigma_\ell z_\ell^2$ is diagonal. Grundmeier and Lebl used this to show that for a non-linear map in normal form all the natural symmetry groups sit inside unitary groups, and that $\Gamma_f$ is then exactly the subgroup of $U(n)$ preserving the underlying Hermitian polynomial $\rho(z,\bar z)=|g(z)|^2-\|p(z)\|^2,$ the real polynomial that encodes $f$ up to target automorphisms. Call a subgroup $\Gamma\le U(n)$ definable by finitely many invariant polynomials if $\Gamma=\{U\in U(n):\ \rho_j(Uz,\overline{Uz})=\rho_j(z,\bar z)\ \text{for}\ j=1,\dots,\ell\}$ for some real polynomials $\rho_1,\dots,\rho_\ell$ on $\mathbb{C}^n$. Their Theorem 1.5 identifies this class with the realizable one: if $f$ is a non-linear rational proper map in normal form, then $\Gamma_f$ is definable by the single invariant polynomial $\rho$ above; conversely, every subgroup of $U(n)$ definable by finitely many invariant polynomials equals $\Gamma_f$ for a polynomial proper map fixing the origin.

What the theorem does not provide is an intrinsic description of the definable class, that is, a group-theoretic or real-algebraic criterion telling which closed subgroups of $U(n)$ are definable and hence which compact groups occur as $\Gamma_f$. Nor does it address the cost of realization: D'Angelo and Xiao asked, for a given group, for the smallest source dimension and the smallest degree of a map realizing it. Partial answers exist. For $\Gamma_f=U(n)$, the set of possible target dimensions is co-finite in each source dimension, with the largest exceptional value found (computer-assisted) via Graham's theorem on complete polynomial sequences (D'Angelo 2022). For maps invariant under the three fixed-point-free cyclic representation types of $G_f$ described above, the possible numbers of terms, and hence target dimensions, exhibit Frobenius-type gaps (D'Angelo-Grundmeier-Lichtblau 2025). The general questions remain open.

Problem Statement

Overarching objective: determine which compact groups occur as the Hermitian invariant group $\Gamma_f$ of a rational proper map of balls of degree at least $2$, and at what minimal complexity each such group can be realized. By Grundmeier-Lebl Theorem 1.5 this is equivalent, on the algebraic side, to understanding which closed subgroups of $U(n)$ are definable by finitely many invariant polynomials. The problem has two linked parts.

(1) Characterization of the realizable class. Give an intrinsic necessary and sufficient condition, group-theoretic, representation-theoretic, or real-algebraic, under which a closed subgroup $\Gamma\le U(n)$ (with $n\ge 2$) is definable by finitely many invariant polynomials, i.e. $\Gamma=\{U\in U(n):\ \rho_j(Uz,\overline{Uz})=\rho_j(z,\bar z)\ \text{for}\ j=1,\dots,\ell\}$ for some real polynomials $\rho_1,\dots,\rho_\ell$ on $\mathbb{C}^n$; equivalently, under which $\Gamma_f=\Gamma$ (up to conjugacy in $\operatorname{Aut}(\mathbb{B}^n)$) for some rational proper map $f:\mathbb{B}^n\to\mathbb{B}^N$ of degree at least $2$. Within the class of all closed subgroups of $U(n)$, the finite groups; tori and block-unitary subgroups such as $U(k)\oplus U(n-k)$ arising from monomial, Whitney-type, and tensor constructions; the trivial group; and $U(n)$ itself are known to be definable, while the definability of the remaining closed subgroups — for instance subgroups acting irreducibly on $\mathbb{C}^n$ or transitively on the unit sphere, such as $SU(n)$ or the compact symplectic group — is open and forms part of the characterization.

(2) Minimal realization cost (D'Angelo-Xiao 2017, Section 6). Given $\Gamma_f$, determine the smallest source dimension for which there is a rational proper map $g$ between balls with $\Gamma_g$ isomorphic to $\Gamma_f$; and, given a finite group $G$, determine the smallest degree of a rational proper map $f$ with $\Gamma_f=G$. Throughout, the degree of a rational map $f=p/g$ in lowest terms is $\deg f=\max(\deg p,\deg g)$. These two minima quantify part (1): they measure how expensive a prescribed symmetry is.

References

1. Dusty Grundmeier and Jiří Lebl, Rational maps of balls and their associated groups, São Paulo Journal of Mathematical Sciences 18 (2024), no. 2, 1793-1806. DOI 10.1007/s40863-024-00430-x; arXiv:2402.03152. https://doi.org/10.1007/s40863-024-00430-x
2. John P. D'Angelo and Ming Xiao, Symmetries in CR complexity theory, Advances in Mathematics 313 (2017), 590-627. DOI 10.1016/j.aim.2017.04.014; arXiv:1703.09320. https://doi.org/10.1016/j.aim.2017.04.014
3. John P. D'Angelo and Ming Xiao, Symmetries and regularity for holomorphic maps between balls, Mathematical Research Letters 25 (2018), no. 5, 1389-1404. DOI 10.4310/MRL.2018.v25.n5.a2. https://doi.org/10.4310/MRL.2018.v25.n5.a2
4. Jiří Lebl, Exhaustion functions and normal forms for proper maps of balls, Mathematische Annalen 390 (2024), 3133-3154. DOI 10.1007/s00208-024-02837-5; arXiv:2212.06102. https://doi.org/10.1007/s00208-024-02837-5
5. John P. D'Angelo, Symmetries, rational sphere maps, and complete polynomial sequences, Complex Analysis and its Synergies 8 (2022), no. 2, Article 7. DOI 10.1007/s40627-022-00095-2. https://doi.org/10.1007/s40627-022-00095-2
6. John P. D'Angelo, Dusty E. Grundmeier, and Daniel A. Lichtblau, Invariant polynomials, gaps, and sparseness, preprint arXiv:2512.05892 (2025). https://arxiv.org/abs/2512.05892
7. Edgar Gevorgyan, Haoran Wang, and Andrew Zimmer, A rigidity result for proper holomorphic maps between balls, Proceedings of the American Mathematical Society 152 (2024), no. 4, 1573-1585. DOI 10.1090/proc/16717. https://doi.org/10.1090/proc/16717
8. John P. D'Angelo and Daniel A. Lichtblau, Spherical space forms, CR mappings, and proper maps between balls, Journal of Geometric Analysis 2 (1992), no. 5, 391-415. DOI 10.1007/BF02921298. https://doi.org/10.1007/BF02921298
9. Daniel Lichtblau, Invariant proper holomorphic maps between balls, Indiana University Mathematics Journal 41 (1992), no. 1, 213-231. DOI 10.1512/iumj.1992.41.41012. https://doi.org/10.1512/iumj.1992.41.41012
10. Franc Forstnerič, Proper holomorphic maps from balls, Duke Mathematical Journal 53 (1986), no. 2, 427-441. DOI 10.1215/S0012-7094-86-05326-3. https://doi.org/10.1215/S0012-7094-86-05326-3
11. Walter Rudin, Proper holomorphic maps and finite reflection groups, Indiana University Mathematics Journal 31 (1982), no. 5, 701-720. DOI 10.1512/iumj.1982.31.31050. https://doi.org/10.1512/iumj.1982.31.31050