Classification of the triples $(G,\pi_\lambda^G,\phi)$ with admissible restriction of a discrete series representation to an embedded $SU(2,1)$

For a connected noncompact simple matrix Lie group $G$ admitting discrete series, the paper of Márquez–Simondi–Vargas (Rev. Unión Matem. Argentina 60 (2019)) determines when a square-integrable irreducible representation restricts admissibly to one distinguished subgroup $H\simeq SU(2,1)$, the copy constructed by Gross–Wallach inside each quaternionic real form: this happens exactly for the quaternionic discrete series, and an explicit multiplicity formula is given. The same paper explicitly raises the general question: since $SU(2,1)$ embeds in a simple group $G$ in possibly several non-conjugate ways, determine all triples $(G,\pi_\lambda^G,\phi)$, over all such embeddings $\phi$, for which the restriction of the discrete series representation $\pi_\lambda^G$ to $\phi(SU(2,1))$ is admissible. Only the symmetric-pair case is classified in general (Kobayashi–Oshima), and the closest analogue, for $SL_2(\mathbb R)$ embeddings, was solved by Duflo–Galina–Vargas; the classification over all embeddings remains open.

Background

Let $G$ be a connected noncompact simple matrix Lie group and let $K\subset G$ be a maximal compact subgroup of the same rank as $G$, so that $G$ contains a compact Cartan torus $T\subset K$. By a theorem of Harish-Chandra, these are precisely the simple groups that admit irreducible unitary representations whose matrix coefficients are square integrable with respect to a Haar measure — the discrete series (square-integrable representations). Their equivalence classes are parametrized by Harish-Chandra parameters $\lambda\in i\mathfrak t^\star$, i.e. imaginary-valued linear forms on the Lie algebra $\mathfrak t$ of $T$ that are regular and such that $\lambda+\rho$ integrates to a character of $T$ ($\rho$ is half the sum of the positive roots); $\pi_\lambda^G$ denotes the corresponding representation. For a closed reductive subgroup $H\subset G$, the restriction $\pi_\lambda^G|_H$ is called admissible, in the sense used throughout the source, when it is unitarily equivalent to a discrete Hilbert direct sum $\sum_j n^{G,H}(\lambda,\mu_j)\,\pi_{\mu_j}^H$ of irreducible unitary representations of $H$, each occurring with finite multiplicity — in the source's realization of this notion the $\mu_j$ are Harish-Chandra parameters of $H$ and the constituents $\pi_{\mu_j}^H$ are discrete series representations of $H$; an explicit such decomposition is called a branching law. Deciding which triples $(G,H,\pi_\lambda^G)$ have admissible (equivalently, discretely decomposable with finite multiplicity) restriction is the 'Stage A' classification problem in Kobayashi's program on branching laws for real reductive groups. For reductive symmetric pairs $(G,H)$ — where $H$ is the fixed-point group of an involution of $G$ — a complete classification of discretely decomposable restrictions is due to Kobayashi–Oshima (2012), building on Kobayashi's discrete-decomposability criteria; for subgroups that do not arise from a symmetric pair, only scattered families of examples are known.

The source paper studies subgroups locally isomorphic to $SU(2,1)$, the special unitary group of the Hermitian form of signature $(2,1)$ on $\mathbb C^3$, a simple real group of real rank one whose Lie algebra $\mathfrak{su}(2,1)$ is the noncompact real form of $\mathfrak{sl}(3,\mathbb C)$ with a compactly embedded Cartan subalgebra. Gross and Wallach constructed a distinguished such subgroup inside every quaternionic real form: a noncompact simple $G$ whose root system $\Phi(\mathfrak g,\mathfrak t)$ relative to a compactly embedded Cartan admits a small system of positive roots $\Psi$ with $n_0+n_1=2$, a positive system $\Psi$ being called small when the maximal root $\beta$ of $\Psi$ is compact and at most two noncompact simple roots $\alpha_0,\alpha_1$ occur in the expansion of $\beta$, with total coefficient $n_0+n_1\le 2$; the quaternionic real forms comprise the groups locally isomorphic to $SU(2,n)$, $SO(4,n)$, $E_{6(2)}$, $E_{7(-5)}$, $E_{8(-24)}$, $F_{4(4)}$ and $G_{2(2)}$, stated in the source to be exactly the noncompact simple groups whose associated symmetric space carries an invariant quaternionic structure. The root vectors of the two roots $\{\pm\alpha,\pm\beta\}$ (with $\alpha$ noncompact simple) span a copy of $\mathfrak{sl}(3,\mathbb C)$ whose real part is $\mathfrak{su}(2,1)$, giving the subgroup $H$. A discrete series representation is called quaternionic when its Harish-Chandra parameter is dominant for $\Psi$. Márquez–Simondi–Vargas prove (Theorem 1 and Proposition 1) that for this $H$ the restriction $\pi_\lambda^G|_H$ is admissible if and only if $\pi_\lambda^G$ is a quaternionic discrete series representation, and give an explicit multiplicity formula for the branching law in terms of weight multiplicity data of a compact subgroup, in the spirit of the Duflo–Vargas and Heckman convolution formulae. They also settle a few neighboring cases: no discrete series of a group locally isomorphic to $SO(3,2n)$ restricts admissibly to the standard $SO(3)$ factor; for $G$ locally isomorphic to $Sp(1,q)$, the quaternionic discrete series restrict admissibly to a subgroup $H_0\simeq Sp(1,1)$, again with multiplicities; and for the rank-one real form of $F_4$ no square-integrable representation restricts admissibly to the analytic subgroup with Lie algebra $\mathfrak{su}(2,1)$ (a result of Vargas from 2010). However, the authors emphasize that $SU(2,1)$ maps into a simple Lie group $G$ 'in perhaps several ways', i.e. through several conjugacy classes of embeddings $\phi$, of which the Gross–Wallach construction treats only one, and they pose as an open question the classification of all admissible triples over every embedding.

Problem Statement

For each connected noncompact simple matrix Lie group $G$ admitting discrete series (i.e. with a maximal compact subgroup $K$ of the same rank as $G$ and a compact Cartan torus $T$), and for each conjugacy class of embeddings $\phi:SU(2,1)\to G$, conjugacy being by elements of $G$ — noting that $G$ may contain several non-conjugate such subgroups, and that an analytic subgroup of $G$ with Lie algebra $\mathfrak{su}(2,1)$ need only be locally isomorphic to $SU(2,1)$ (as in the settled cases) and is not thereby excluded from the classification — determine all triples $(G,\pi_\lambda^G,\phi)$, with $\pi_\lambda^G$ an irreducible square-integrable representation of Harish-Chandra parameter $\lambda$, such that the restriction $\pi_\lambda^G\big|_{\phi(SU(2,1))}$ is an admissible representation of the image subgroup, i.e. decomposes as a discrete Hilbert sum of irreducible unitary representations of $\phi(SU(2,1))$ with finite multiplicities, the constituents being discrete series of the image subgroup parametrized by its Harish-Chandra parameters, as in the source. The answer must enumerate the embedding classes (including any classification of $\mathfrak{su}(2,1)$-subalgebras of $\mathfrak g$ this requires) and, for each, give the exact set of Harish-Chandra parameters $\lambda$ yielding admissible restriction. It must in particular reproduce the two settled $SU(2,1)$ cases: the Gross–Wallach subgroup of every quaternionic real form (admissible exactly for the quaternionic discrete series) and the analytic $SU(2,1)$-subgroup of the rank-one real form of $F_4$ (never admissible); and it must not conflict with the source's neighboring settled results, which concern other subgroups and are not part of the $SU(2,1)$-classification (no admissible restriction of any discrete series to the $SO(3)$ factor for $G$ locally isomorphic to $SO(3,2n)$, and admissible restriction of the quaternionic discrete series of $Sp(1,q)$ to $H_0\simeq Sp(1,1)$).

References

1. O. Márquez, S. Simondi, J. A. Vargas, Branching laws: some results and new examples, Revista de la Unión Matemática Argentina 60 (2019), no. 1, 45–59, DOI 10.33044/revuma.v60n1a04, https://doi.org/10.33044/revuma.v60n1a04; preprint arXiv:1806.01705, https://arxiv.org/abs/1806.01705
2. B. Gross, N. Wallach, On quaternionic discrete series representations, and their continuations, Journal für die reine und angewandte Mathematik 481 (1996), 73–124, DOI 10.1515/crll.1996.481.73, https://doi.org/10.1515/crll.1996.481.73
3. M. Duflo, E. Galina, J. A. Vargas, Square Integrable Representations of Reductive Lie Groups with Admissible Restriction to $SL_2(\mathbb R)$, Journal of Lie Theory 27 (2017), no. 4, 1033–1056, DOI 10.5802/jolt.981, https://doi.org/10.5802/jolt.981; preprint (by E. Galina and J. A. Vargas, under the title Discrete series of simple Lie groups with admissible restriction to a $SL_2(\mathbb R)$): arXiv:1506.00609, https://arxiv.org/abs/1506.00609
4. T. Kobayashi, Y. Oshima, Classification of discretely decomposable $A_{\mathfrak q}(\lambda)$ with respect to reductive symmetric pairs, Advances in Mathematics 231 (2012), no. 3–4, 2013–2047, DOI 10.1016/j.aim.2012.07.006, https://doi.org/10.1016/j.aim.2012.07.006
5. J. A. Vargas, Harish Chandra Modules of Rank One Lie Groups with Admissible Restriction to Some Reductive Subgroup, Journal of Lie Theory 20 (2010), no. 4, 643–663, DOI 10.5802/jolt.612, https://doi.org/10.5802/jolt.612
6. J. A. Vargas, Associated symmetric pair and multiplicities of admissible restriction of discrete series, International Journal of Mathematics 27 (2016), no. 12, 1650100, DOI 10.1142/S0129167X16501007, https://doi.org/10.1142/S0129167X16501007; preprint arXiv:1311.3851, https://arxiv.org/abs/1311.3851
7. T. Kobayashi, Recent advances in branching problems of representations, Sugaku Expositions 37 (2024), no. 2, 129–177, DOI 10.1090/suga/485, https://arxiv.org/abs/2112.00642
8. M. Duflo, J. Vargas, Branching laws for square integrable representations, Proceedings of the Japan Academy, Series A, Mathematical Sciences 86 (2010), no. 3, 49–54, DOI 10.3792/pjaa.86.49, https://doi.org/10.3792/pjaa.86.49
9. B. Ørsted, J. A. Vargas, Pseudo-dual pairs and branching of Discrete Series, preprint arXiv:2302.14190 (2023), https://arxiv.org/abs/2302.14190
10. B. Ørsted, J. A. Vargas, Discrete Series Representations Decomposing Discretely with Finite Multiplicity under Restriction to Symmetric Subgroups, Journal of Lie Theory 35 (2025), no. 4, 909–955, DOI 10.5802/jolt.1413, https://doi.org/10.5802/jolt.1413
11. B. Ørsted, J. A. Vargas, Branching laws and a duality principle, Part I, preprint arXiv:2412.08351 (2024), https://arxiv.org/abs/2412.08351
12. T. Kobayashi, Harish-Chandra's admissibility theorem and beyond, preprint arXiv:2412.20387 (2024), https://arxiv.org/abs/2412.20387