Sufficiency of the characteristic-cubic obstruction for P-convexity with respect to a covector in the indefinite-curvature case (converse of Hörmander's Theorem 6.6)

By a classical theorem of Malgrange, a differential operator $P(D)$ with constant coefficients is surjective on $C^\infty(X)$ exactly when the open set $X\subset\mathbb{R}^n$ is P-convex for supports. Hörmander's paper 'Local P-convexity' (2000) localized this property to single characteristic covectors $\xi$ — P-convexity with respect to $\xi$ — and, for operators of principal type, translated it into the boundary geometry of $X$. When the characteristic set at $\xi$ has nonsingular indefinite transversal curvature $B$, the situation of the ultrahyperbolic operator in $\mathbb{R}^4$, Theorem 6.6 of that paper proves only a necessary condition: if a scaled, translated characteristic cubic is contained in $X$ along an interval of the bicharacteristic whose endpoints lie in $X$, then $X$ is not P-convex with respect to $\xi$. Whether the converse holds — whether the absence of such cubics forces P-convexity with respect to $\xi$ — is the paper's principal open problem. The earlier geometric line (Malgrange, Persson, Tintarev) settled the elliptic, planar, wave-operator and definite-curvature cases, and the modern line (Kalmes; Ciąś and Kalmes 2026) characterizes only flat, semi-elliptic, or orthogonally degenerate operators. The open problem: prove that the necessary condition of Theorem 6.6 is also sufficient, or exhibit a counterexample.

Background

For a polynomial $p$ in $n$ real variables, let $P(D)$ be the differential operator with constant coefficients obtained by substituting $D_j=(1/i)\partial/\partial x_j$; $p$ is called the symbol of $P$. For an open set $X\subset\mathbb{R}^n$, a classical problem of the linear theory is to decide when $P(D)$ is surjective on the space $C^\infty(X)$ of smooth functions on $X$. Malgrange (1962) answered this geometrically: $X$ is called P-convex for supports if for every compact $K\subset X$ there is a compact $K'\subset X$ such that $\operatorname{supp}P(-D)\varphi\subset K$ implies $\operatorname{supp}\varphi\subset K'$ for every test function $\varphi\in C_0^\infty(X)$ (where $P(-D)$ is the operator with reflected coefficients and $C_0^\infty(X)$ denotes the compactly supported smooth functions), and $P(D)$ is surjective on $C^\infty(X)$, and more generally on the local spaces of distributions of finite order over $X$, precisely when $X$ is P-convex for supports. Convex open sets are always P-convex for supports, and every open set is P-convex for supports when $P$ is elliptic. The theory of the remaining cases was developed in Hörmander's treatise The Analysis of Linear Partial Differential Operators II, which among other things gives a complete geometric characterization in the plane and a general necessary condition: the boundary distance $d(x,\complement X)$ must satisfy the minimum principle (a real function $f$ on $X$ satisfies the minimum principle in a closed affine subspace $F$ if for every compact $K\subset F\cap X$ the minimum of $f$ over $K$ is attained already on the relative boundary of $K$ in $F$) in every characteristic hyperplane, that is, every hyperplane whose normal $\xi$ is a characteristic covector, meaning $p(\xi)=0$.

Hörmander's paper 'Local P-convexity' (2000) refined the theory to a local, directional notion. If $X$ is not P-convex, the source constructs a test function $\psi\in C_0^\infty(\overline X)$ with $P(-D)\psi\in C_0^\infty(X)$ whose support touches $\partial X$ and whose exterior conormals all point into a narrow cone around a single characteristic covector $\xi$; here a covector $\xi$ is an exterior conormal of a closed set $S$ at a boundary point $x$ if $\langle y,\xi\rangle\le\langle x,\xi\rangle$ for all $y\in S$. The set $X$ is called P-convex with respect to $\xi$ when no such $\psi$ exists for that $\xi$, and the paper proves that $X$ is P-convex for supports if and only if it is P-convex with respect to every characteristic covector. This reduces a global question to one frequency direction at a time and makes the geometry of the characteristic set at $\xi$ the decisive object.

Fix a characteristic covector $\xi$ with $p'(\xi)\ne0$ (so $P$ is of principal type at $\xi$; the straight line with direction $p'(\xi)$ is the bicharacteristic direction). After a linear change of variables one may take $\xi=(0,1,0,\dots,0)$ and $p'(\xi)=(1,0,\dots,0)$, and then the symbol has the normal form $p(\eta)=\eta_1\eta_2+\eta_2^2\,r(\eta''/\eta_2),\qquad\eta''=(\eta_3,\dots,\eta_n),$ in a neighbourhood of $\xi$, with $r$ real analytic and $r(\eta'')=O(|\eta''|^2)$. The symmetric matrix $B=r''(0)$ measures the curvature of the characteristic set at $\xi$ in the directions transversal to the radial direction, and its signature determines which geometric obstruction to P-convexity with respect to $\xi$ appears. The 2000 paper gives complete 'if and only if' characterizations of P-convexity with respect to $\xi$ in three regimes: for isolated characteristics (those for which a conic neighbourhood of $\xi$ — a neighbourhood invariant under positive dilations of the covector — contains no other zeros of $p$) via the minimum principle for $d(x,\complement X)$ in hyperplanes with conormal $\xi$; for flat characteristics ($r''(0)=0$) via the absence of certain convex cylinders in $\overline X$ along bicharacteristics; and for convex characteristics ($B$ semidefinite, with the local condition $p=0\iff q=0$ for a defining function $q$ of the characteristic surface) via the absence of translates of duals of osculating quadrics — the second-order tangent quadrics of the characteristic surface — in $\overline X$ along bicharacteristic intervals. These results subsume and extend the earlier geometric characterizations of Persson (operators with principal part the wave operator; principal type in $\mathbb{R}^3$) and Tintarev (real principal type, in terms of tangent curves).

When $B$ is nonsingular and indefinite, no characterization is known. Writing $W_+\oplus W_-$ for the $B$-orthogonal decomposition of $\mathbb{R}^{n-2}$ on which $\pm B$ are positive definite, and $x''=x_+''+x_-''$ for the corresponding splitting of $x''=(x_3,\dots,x_n)$, the source defines the characteristic cubic $E=\Bigl\{x\in\mathbb{R}^n:\ |x_1|<1,\ 2x_2+\frac{\langle B^{-1}x_+'',x_+''\rangle}{1+x_1}+\frac{\langle B^{-1}x_-'',x_-''\rangle}{x_1-1}<0\Bigr\}$ (the quadric $\{x:|x_1|<1,\ 2x_2+\langle(G+x_1B)^{-1}x'',x''\rangle<0\}$ with $G$ equal to $B$ on $W_+$ and $-B$ on $W_-$, the minimal positive form with $|\langle B\eta'',\eta''\rangle|\le\langle G\eta'',\eta''\rangle$). Theorem 6.6 of the source proves: if $X\subset\mathbb{R}^n$ is open and connected, $x^0\in\partial X$, and there exist $c>0$ and a neighbourhood $Y$ of $x^0+(-c,c)\times\{0\}^{n-1}$ with $Y\cap(\{x^0\}+cE)\subset X\quad\text{and}\quad x^0\pm(c,0,\dots,0)\in X,$ then $X$ is not P-convex with respect to $\xi$. The emblematic case is the ultrahyperbolic operator $p(\eta)=\eta_1^2+\eta_2^2-\eta_3^2-\eta_4^2$ in $\mathbb{R}^4$, whose characteristic set is a cone with signature-$(1,1)$ transversal curvature at every characteristic covector. Proposition 6.7 of the source gives supporting evidence for the converse: if a null solution (a solution of $P(-D)\psi=0$) has support containing an interval of the bicharacteristic and bounded by a quadric $\{x:x_2+\tfrac12\langle\Phi(x_1)x'',x''\rangle\le0\}$, then the matrix inequality $\langle(\Phi(x_1)^{-1}-x_1B)\eta'',\eta''\rangle\ge|\langle B\eta'',\eta''\rangle|$ holds for $|x_1|<1$, which implies that such a support must contain a characteristic cubic; the source reads this as evidence that the cubic condition is the sharp obstruction. The converse of Theorem 6.6 — whether every failure of P-convexity with respect to $\xi$ is witnessed by a characteristic cubic in the configuration above — was left open by the source.

Problem Statement

Let $P$ be a differential operator with constant coefficients in $\mathbb{R}^n$, with symbol $p$, and let $\xi$ be a characteristic covector of $P$ ($p(\xi)=0$) at which $p$ is of principal type ($p'(\xi)\ne0$). Suppose that after the linear change of variables with $\xi=(0,1,0,\dots,0)$ and $p'(\xi)=(1,0,\dots,0)$ the symbol takes, in a neighbourhood of $\xi$, the normal form $p(\eta)=\eta_1\eta_2+\eta_2^2 r(\eta''/\eta_2)$ with $r$ real analytic, $r(\eta'')=O(|\eta''|^2)$ and $\eta''=(\eta_3,\dots,\eta_n)$, and that the transversal curvature $B=r''(0)$ is nonsingular and indefinite — the situation of the ultrahyperbolic operator $p(\eta)=\eta_1^2+\eta_2^2-\eta_3^2-\eta_4^2$ in $\mathbb{R}^4$ at every characteristic covector. Let $W_+\oplus W_-$ be the $B$-orthogonal decomposition of $\mathbb{R}^{n-2}$ with $\pm B$ positive definite on $W_\pm$, let $x''=x_+''+x_-''$ be the corresponding splitting, and let $E=\Bigl\{x\in\mathbb{R}^n:\ |x_1|<1,\ 2x_2+\frac{\langle B^{-1}x_+'',x_+''\rangle}{1+x_1}+\frac{\langle B^{-1}x_-'',x_-''\rangle}{x_1-1}<0\Bigr\}$ be the characteristic cubic (Definition 6.5 of the source). Theorem 6.6 of Hörmander (2000) proves the following necessary condition: if $X\subset\mathbb{R}^n$ is open and connected, $x^0\in\partial X$, and there exist $c>0$ and a neighbourhood $Y$ of $x^0+(-c,c)\times\{0\}^{n-1}$ such that $Y\cap(\{x^0\}+cE)\subset X\quad\text{and}\quad x^0\pm(c,0,\dots,0)\in X,$ then $X$ is not P-convex with respect to $\xi$.

Question (the converse, open since the source): is the absence of every such characteristic-cubic configuration also sufficient? Either prove that an open connected $X\subset\mathbb{R}^n$ which contains no translate $\{x^0\}+cE$ in the configuration of Theorem 6.6 — for any $x^0\in\partial X$, $c>0$, and neighbourhood $Y$ of $x^0+(-c,c)\times\{0\}^{n-1}$ as above — is necessarily P-convex with respect to $\xi$, which would complete, together with the source's results for isolated, flat, and convex characteristics, the geometric characterization of P-convexity with respect to a covector for principal-type operators; or exhibit a counterexample: an explicit operator $P$, a characteristic covector $\xi$ with nonsingular indefinite $B$ in the normal form, and an open connected set $X$ that is not P-convex with respect to $\xi$ although no scaled translate of the characteristic cubic $E$ lies in $X$ in the forbidden configuration. The ultrahyperbolic operator in $\mathbb{R}^4$ is the source's emblematic instance of the hypothesis.

References

1. Lars Hörmander (2000). Local P-convexity. Journal d'Analyse Mathématique 80, 101–141. DOI 10.1007/BF02791535. URL https://doi.org/10.1007/BF02791535. (Zbl 0968.35026)
2. Lars Hörmander (2001). Asgeirsson's mean value theorem and related identities. Journal of Functional Analysis 184(2), 377–401. DOI 10.1006/jfan.2001.3743. URL https://doi.org/10.1006/jfan.2001.3743.
3. Lars Hörmander (2005). The Analysis of Linear Partial Differential Operators II: Differential Operators with Constant Coefficients. Classics in Mathematics, Springer, Berlin (reprint of the 1983 edition). ISBN 978-3-540-22516-4. DOI 10.1007/b138375. URL https://doi.org/10.1007/b138375.
4. Lars Hörmander (1994). Notions of Convexity. Progress in Mathematics 127, Birkhäuser, Basel. ISBN 0-8176-3799-0. URL https://zbmath.org/723330 (zbMATH Zbl 0835.32001).
5. Lars Hörmander (2018). Unpublished Manuscripts. Springer, Cham. ISBN 978-3-319-69849-6. DOI 10.1007/978-3-319-69850-2. URL https://doi.org/10.1007/978-3-319-69850-2.
6. Bernard Malgrange (1962). Sur les ouverts convexes par rapport à un opérateur différentiel. Comptes Rendus Hebdomadaires des Séances de l'Académie des Sciences, Paris 254, 614–615. Zbl 0117.06301. URL https://zbmath.org/?q=an%3A0117.06301.
7. Jan Persson (1979). The Cauchy problem at simply characteristic points and P-convexity. Annali di Matematica Pura ed Applicata (4) 122, 117–140. DOI 10.1007/BF02411691. Zbl 0424.35007. URL https://doi.org/10.1007/BF02411691.
8. Jan Persson (1981). The wave operator and P-convexity. Bollettino della Unione Matematica Italiana, Serie V, B 18, 591–604. Zbl 0479.35024. URL https://zbmath.org/?q=an%3A0479.35024.
9. Jan Persson (1993). The geometry of P-convex sets. Bollettino della Unione Matematica Italiana, Serie VII, B 7(3), 549–573. Zbl 0831.35033. URL https://zbmath.org/?q=an%3A0831.35033.
10. Kyril Tintarev (1988). On the geometry of P-convex sets for operators of real principal type. Israel Journal of Mathematics 64(2), 195–206. DOI 10.1007/BF02787223. URL https://doi.org/10.1007/BF02787223.
11. Kyril Tintarev (1992). Characterization of P-convexity for supports in terms of tangent curves. Journal of Mathematical Analysis and Applications 164(2), 590–596. DOI 10.1016/0022-247X(92)90137-3. URL https://doi.org/10.1016/0022-247X(92)90137-3.
12. Thomas Kalmes (2011). Every P-convex subset of R² is already strongly P-convex. Mathematische Zeitschrift 269(3–4), 721–731. DOI 10.1007/s00209-010-0765-7. URL https://doi.org/10.1007/s00209-010-0765-7.
13. Thomas Kalmes (2019). Surjectivity of differential operators and linear topological invariants for spaces of zero solutions. Revista Matemática Complutense 32(1), 37–55. DOI 10.1007/s13163-018-0266-5. URL https://doi.org/10.1007/s13163-018-0266-5.
14. Tomasz Ciąś and Thomas Kalmes (2026). From zero solutions with partially bounded supports to solvability and Runge approximation for partial differential equations. Preprint arXiv:2607.19272. URL https://arxiv.org/abs/2607.19272.
15. Maxim N. Demchenko (2024). Existence of a solution to the scattering problem for the ultrahyperbolic equation. Preprint arXiv:2410.20093. URL https://arxiv.org/abs/2410.20093.