For the pair (G,H)=(O(p,q),O(p−1,q))(G,H)=(O(p,q),O(p-1,q)) with p≥3p\ge 3, q≥2q\ge 2, Kobayashi and Speh proved a hidden-symmetry, packet-level branching law: for regular parameters λ≥12(p+q−2)\lambda\ge\tfrac12(p+q-2) and μ≥12(p+q−3)\mu\ge\tfrac12(p+q-3), the reducible representations U(λ)=Π+,λ⊕Π−,λU(\lambda)=\Pi_{+,\lambda}\oplus\Pi_{-,\lambda} of GG and V(μ)=π+,μ⊕π−,μV(\mu)=\pi_{+,\mu}\oplus\pi_{-,\mu} of HH satisfy dim⁡Hom⁡H(U(λ)∣H,V(μ))=1\dim\operatorname{Hom}_H(U(\lambda)|_H,V(\mu))=1 for every pair of regular parameters: exactly one member of each two-element packet pairs with the other packet, with multiplicity one, and an interlacing condition on infinitesimal characters selects which members pair. The theorem is stated only under these regularity and dimensional hypotheses, and the authors observe that it suggests a generalization of the Gross–Prasad conjectures from tempered to these non-tempered packets. The problem is to complete this theory: determine the packet-level branching behavior (including any multiplicity-one statement) for non-regular parameters, treat the degenerate signatures p=1,2p=1,2 or q=1q=1, and make the suggested Gross–Prasad-type conjecture for non-tempered (Arthur-type) packets precise and prove it.

Background

Let G=O(p,q)G=O(p,q) be the indefinite orthogonal group, the group of linear transformations of Rp+q\mathbb{R}^{p+q} preserving a symmetric form of signature (p,q)(p,q); it is a real reductive Lie group. A branching law describes how an irreducible unitary representation Π\Pi of GG decomposes when restricted to a subgroup HH; an irreducible unitary representation π\pi of HH lies in the discrete spectrum of Π∣H\Pi|_H when there is a nonzero continuous HH-equivariant map π→Π∣H\pi\to\Pi|_H, and the dimension of the space Hom⁡H(Π∣H,π)\operatorname{Hom}_H(\Pi|_H,\pi) of such symmetry breaking operators is the multiplicity of π\pi.

For H=O(p−1,q)H=O(p-1,q) the relevant representations arise as discrete series of indefinite Stiefel manifolds: Kobayashi constructed two families of irreducible unitary representations of O(p,q)O(p,q), denoted Π+,λp,q\Pi_{+,\lambda}^{p,q} and Π−,λp,q\Pi_{-,\lambda}^{p,q}, the first consisting of subrepresentations of L2(O(p,q)/O(p−1,q))L^2(O(p,q)/O(p-1,q)) and the second of subrepresentations of L2(O(p,q)/O(p,q−1))L^2(O(p,q)/O(p,q-1)), parametrized by λ∈Z+12(p+q)\lambda\in\mathbb{Z}+\tfrac12(p+q) with λ>0\lambda>0; geometrically they are the quantization of minimal elliptic coadjoint orbits, and analytically they unitarize Zuckerman derived-functor modules (cohomological induction from a θ\theta-stable parabolic). Each carries an infinitesimal character, the representation-theoretic analogue of the highest weight of a finite-dimensional representation, given for the packet below by (λ,p+q−42,…,κ(p+q))(\lambda,\tfrac{p+q-4}{2},\dots,\kappa(p+q)), where κ(n)=0\kappa(n)=0 if nn is even and κ(n)=12\kappa(n)=\tfrac12 if nn is odd.

A parameter λ\lambda is called regular if λ≥12(p+q−2)\lambda\ge\tfrac12(p+q-2), and μ\mu regular if μ≥12(p+q−3)\mu\ge\tfrac12(p+q-3); regularity holds exactly when the infinitesimal character equals that of a finite-dimensional representation. For regular λ\lambda the two inequivalent representations Π+,λ\Pi_{+,\lambda} and Π−,λ\Pi_{-,\lambda} share the same infinitesimal character, and following Arthur's packet philosophy one calls {Π+,λ,Π−,λ}\{\Pi_{+,\lambda},\Pi_{-,\lambda}\} a packet; similarly H=O(p−1,q)H=O(p-1,q) has packets {π+,μ,π−,μ}\{\pi_{+,\mu},\pi_{-,\mu}\}. Write U(λ)=Π+,λ⊕Π−,λU(\lambda)=\Pi_{+,\lambda}\oplus\Pi_{-,\lambda} and V(μ)=π+,μ⊕π−,μV(\mu)=\pi_{+,\mu}\oplus\pi_{-,\mu}. These representations are non-tempered except in one boundary case: Π+,λ\Pi_{+,\lambda} is non-tempered for p≥3p\ge3 and Π−,λ\Pi_{-,\lambda} is non-tempered for q≥3q\ge3, while for q=2q=2 a regular Π−,λ\Pi_{-,\lambda} is a holomorphic discrete series representation on the identity component; non-tempered representations have matrix coefficients that decay more slowly than those of discrete series, unlike the tempered representations to which the classical branching conjectures apply.

The Gross–Prasad conjectures (1992) predict, for a pair of special orthogonal groups over a local field and for tempered representations inside LL-packets, which single member of the packet of SOn−1SO_{n-1} occurs in the restriction of a given member of the packet of SOnSO_n; over the reals the prediction is expressed by interlacing of parameters and is related to an ε\varepsilon-factor. Ørsted and Speh (2019) conjectured packet-level multiplicity-one statements of this type for the corresponding packets of the connected special orthogonal groups SO0(p,q)SO_0(p,q) and SO0(p−1,q)SO_0(p-1,q), formulated both in terms of Arthur packets of cohomologically induced modules (in the sense of Adams–Johnson) and, over pure inner forms and relevant pairs, in Vogan's language of Arthur–Vogan packets.

Kobayashi and Speh (2020) proved, under the standing Assumption O\mathcal{O} (p≥3p\ge 3, q≥2q\ge 2, imposed to avoid special cases) and for regular λ\lambda and μ\mu: (i) each Πλ\Pi_\lambda in the packet {Π+,λ,Π−,λ}\{\Pi_{+,\lambda},\Pi_{-,\lambda}\} pairs with exactly one πμ\pi_\mu in the packet {π+,μ,π−,μ}\{\pi_{+,\mu},\pi_{-,\mu}\} and vice versa, with multiplicity one; equivalently dim⁡Hom⁡H(U(λ)∣H,V(μ))=1\dim\operatorname{Hom}_H(U(\lambda)|_H,V(\mu))=1; and (ii) in interlacing form, dim⁡Hom⁡H(Π−,λ∣H,π−,μ)=1\dim\operatorname{Hom}_H(\Pi_{-,\lambda}|_H,\pi_{-,\mu})=1 when μ>λ>p+q−42>⋯>12>0\mu>\lambda>\tfrac{p+q-4}{2}>\dots>\tfrac12>0 and dim⁡Hom⁡H(Π+,λ∣H,π+,μ)=1\dim\operatorname{Hom}_H(\Pi_{+,\lambda}|_H,\pi_{+,\mu})=1 when λ>μ>p+q−42>⋯>12>0\lambda>\mu>\tfrac{p+q-4}{2}>\dots>\tfrac12>0. This hidden symmetry — the packet-level law is far simpler than the branching laws of the individual representations, which they combined from Kobayashi's complete multiplicity-free description of the discrete spectra — proves the conjectures of Ørsted–Speh in this regular, split-form range and, as the authors state, suggests a generalization of the Gross–Prasad conjecture, which was formulated only for tempered representations. Two aspects of the source work remain restricted: all statements assume regularity of λ\lambda and μ\mu, and the packet theorem excludes the signatures p=1,2p=1,2 or q=1q=1, where the packets degenerate (for p=1p=1 there are no subrepresentations of L2(O(p,q)/O(p−1,q))L^2(O(p,q)/O(p-1,q)) and Π+,λ\Pi_{+,\lambda} is regarded as zero; for p=2p=2 the representation splits into holomorphic and anti-holomorphic discrete series on the identity component; for q=1q=1 the parameter set of Π−,λ\Pi_{-,\lambda} is empty, so that packet member vanishes).

Problem Statement

The problem is to complete the hidden-symmetry packet-level branching theory for the pair (G,H)=(O(p,q),O(p−1,q))(G,H)=(O(p,q),O(p-1,q)) so that it covers the full parameter range, all signatures, and the Gross–Prasad-type generalization it suggests. Concretely, for parts (1) and (3) under the standing Assumption O\mathcal{O} (p≥3p\ge3, q≥2q\ge2), with packets {Π+,λ,Π−,λ}\{\Pi_{+,\lambda},\Pi_{-,\lambda}\} of GG and {π+,μ,π−,μ}\{\pi_{+,\mu},\pi_{-,\mu}\} of HH, U(λ)=Π+,λ⊕Π−,λU(\lambda)=\Pi_{+,\lambda}\oplus\Pi_{-,\lambda} and V(μ)=π+,μ⊕π−,μV(\mu)=\pi_{+,\mu}\oplus\pi_{-,\mu} as in the source paper:

(1) Non-regular parameters. For λ∈Z+12(p+q)\lambda\in\mathbb{Z}+\tfrac12(p+q) with λ>0\lambda>0 and μ∈Z+12(p+q−1)\mu\in\mathbb{Z}+\tfrac12(p+q-1) with μ>0\mu>0 (the natural parameter ranges of the representations) but violating the regularity thresholds (λ<12(p+q−2)\lambda<\tfrac12(p+q-2) or μ<12(p+q−3)\mu<\tfrac12(p+q-3)), determine the spaces Hom⁡H(U(λ)∣H,V(μ))\operatorname{Hom}_H(U(\lambda)|_H,V(\mu)): decide whether dim⁡Hom⁡H(U(λ)∣H,V(μ))≤1\dim\operatorname{Hom}_H(U(\lambda)|_H,V(\mu))\le 1 still holds and is nonzero exactly under an explicit interlacing-type condition on the parameters; if packet-level multiplicity one fails at or beyond the regularity walls, identify the correct packet-level branching law.

(2) Small-rank signatures. Establish the corresponding packet-level description in the signatures excluded by Assumption O\mathcal O, namely p=1p=1, p=2p=2, or q=1q=1, where the packets degenerate (Π+,λ\Pi_{+,\lambda} vanishes for p=1p=1 and splits into holomorphic and anti-holomorphic discrete series on the identity component for p=2p=2, while Π−,λ\Pi_{-,\lambda} vanishes for q=1q=1, and on the HH-side π+,μ\pi_{+,\mu} vanishes for p=2p=2 while π−,μ\pi_{-,\mu} vanishes for q=1q=1, whereas for p=1p=1 the subgroup H=O(0,q)=O(q)H=O(0,q)=O(q) is compact and both packet members vanish, so the HH-packet is empty and the packet-level statement is vacuous there, the corresponding requirement being the restriction of the surviving member Π−,λ\Pi_{-,\lambda} of the GG-packet to the compact subgroup O(q)O(q)), formulating the statement so that it specializes to the known law when p≥3p\ge3, q≥2q\ge2.

(3) Gross–Prasad-type generalization. Make precise, and prove, the suggested generalization of the Gross–Prasad conjecture from tempered to these non-tempered packets: a multiplicity-one prediction for the restriction O(p,q)↓O(p−1,q)O(p,q)\downarrow O(p-1,q) covering the full parameter range, formulated — as in the Ørsted–Speh conjectures — for Arthur packets of the cohomologically induced representations and, over pure inner forms forming relevant pairs, for Arthur–Vogan packets.

Parts (1)–(3) are one coherent objective: a complete Gross–Prasad-type multiplicity-one theory for the restriction of these packets, removing the regularity, signature, and split-form caveats under which the hidden-symmetry law is currently proved.

References

  1. B. H. Gross and D. Prasad, On the decomposition of a representation of SO_n when restricted to SO_{n-1}, Canadian Journal of Mathematics 44 (1992), no. 5, 974–1002. DOI: 10.4153/CJM-1992-060-8, https://doi.org/10.4153/CJM-1992-060-8
  2. Toshiyuki Kobayashi and Birgit Speh, A hidden symmetry of a branching law, in: Lie Theory and Its Applications in Physics, Springer Proceedings in Mathematics & Statistics 335, Springer, 2020, pp. 15–28. DOI: 10.1007/978-981-15-7775-8_2; arXiv:2006.16667, https://arxiv.org/abs/2006.16667
  3. Toshiyuki Kobayashi, Branching laws of unitary representations associated to minimal elliptic orbits for indefinite orthogonal group O(p,q), Advances in Mathematics 388 (2021), 107862. DOI: 10.1016/j.aim.2021.107862; arXiv:1907.07994, https://arxiv.org/abs/1907.07994
  4. Bent Ørsted and Birgit Speh, Branching laws for discrete series of some affine symmetric spaces, arXiv:1907.07544 (2019), https://arxiv.org/abs/1907.07544
  5. Bent Ørsted and Birgit Speh, Some branching laws for symmetric spaces, arXiv:2209.10755 (2022), https://arxiv.org/abs/2209.10755
  6. Toshiyuki Kobayashi and Birgit Speh, Symmetry breaking for representations of rank one orthogonal groups, Memoirs of the American Mathematical Society 238 (2015), no. 1126. DOI: 10.1090/memo/1126; arXiv:1310.3213, https://arxiv.org/abs/1310.3213
  7. Toshiyuki Kobayashi and Birgit Speh, Symmetry breaking for representations of rank one orthogonal groups II, Lecture Notes in Mathematics 2234, Springer, 2018. DOI: 10.1007/978-981-13-2901-2; arXiv:1801.00158, https://arxiv.org/abs/1801.00158
  8. Toshiyuki Kobayashi and Birgit Speh, Symmetry breaking for orthogonal groups and a conjecture by B. Gross and D. Prasad, in: Geometric Aspects of the Trace Formula, Simons Symposia, Springer, 2018, pp. 245–266. DOI: 10.1007/978-3-319-94833-1_8; arXiv:1702.00263, https://arxiv.org/abs/1702.00263
  9. Clemens Weiske, Branching of unitary O(1,n+1)-representations with non-trivial (g,K)-cohomology, arXiv:2106.13562 (2021), https://arxiv.org/abs/2106.13562
  10. Toshiyuki Kobayashi, Recent advances in branching problems of representations, Sugaku Expositions 37 (2024), 129–177. DOI: 10.1090/suga/485; arXiv:2112.00642, https://arxiv.org/abs/2112.00642
  11. Toshiyuki Kobayashi, Stability of branching multiplicities for orthogonal Gelfand pairs, arXiv:2604.22262 (2026), https://arxiv.org/abs/2604.22262
  12. Toshiyuki Kobayashi and Birgit Speh, How does the restriction of representations change under translations? A story for the general linear groups and the unitary groups, arXiv:2502.08479 (2025), https://arxiv.org/abs/2502.08479
  13. Michael Harris, Toshiyuki Kobayashi, and Birgit Speh, Translation functors, branching problems, and applications to the restriction of coherent cohomology of Shimura varieties, arXiv:2509.17007 (2025), https://arxiv.org/abs/2509.17007