A diffeomorphism-invariant, effectively checkable class of hypoelliptic variable-coefficient operators: the open core of the limitations of Hörmander's 1958 constant-strength theorem

Hörmander (1958) proved that a linear partial differential operator with smooth coefficients is hypoelliptic — solutions with smooth right-hand side are smooth — whenever it is of constant strength and formally hypoelliptic: at every point the operator with coefficients frozen at that point is hypoelliptic in the constant-coefficient sense, and the frozen operators are mutually comparable in strength. The paper flagged three limitations: it gave no quantitative derivative-growth (Gevrey-scale) estimates; the defining conditions depend on the choice of coordinates, so the class is not invariant under diffeomorphisms; and only scalar operators were treated. The Gevrey-scale question was resolved affirmatively: Shafii-Mousavi and Zieleźny (1979) proved that constant-strength operators with Gevrey-class coefficients and a d-hypoelliptic frozen operator map Gevrey data to solutions of quantified Gevrey order, and the result has since been extended to ultradifferentiable (Roumieu) classes. The systems extension was substantially carried out by Volevich and continued in the Newton-polyhedron school. What remains open is the problem posed by Malgrange (1957) in the parallel announcement of the same theorem: exhibit a class of hypoelliptic operators that is invariant under diffeomorphisms, admits membership conditions checkable from the operator itself, and is large enough to contain the constant-strength class together with the elliptic and subelliptic classes — in other words, an intrinsic characterization of hypoellipticity for variable-coefficient operators, including its extension to systems such as the multi-order Douglis–Nirenberg structures that Malgrange's framework missed.

Background

A linear partial differential operator with variable coefficients on an open set $Ω ⊂ ℝ^n$ is written $P(x,D) = Σ_{|α|≤m} a_α(x)D^α$, where $D^α = (-i∂/∂x)^α$ and the coefficients $a_α$ are smooth functions. Such operators act on distributions, which are continuous linear functionals on the space of compactly supported smooth test functions; smooth functions are embedded in this space by integration against test functions. The operator is called hypoelliptic if $P(x,D)u ∈ C^∞$ for a distribution $u$ forces $u ∈ C^∞$: every solution of an equation with smooth right-hand side is itself smooth, with no singularities inherited from the equation. Elliptic operators — those whose principal symbol $Σ_{|α|=m} a_α(x)ξ^α$ has no real zeros for $ξ ≠ 0$ — are the classical example.

For operators with constant coefficients, Hörmander (1955) characterized hypoellipticity completely: the polynomial symbol $P(ξ)$ must satisfy $P^{(α)}(ξ)/P(ξ) → 0$ as $|ξ| → ∞$ for every multi-index $α ≠ 0$ — a multi-index being a tuple of non-negative integers labelling a mixed derivative — where $P^{(α)}$ denotes the derivative of the symbol with respect to $ξ$. The regularity behind this condition is quantitative. A constant-coefficient operator $Q(D)$ is called $d$-hypoelliptic, for some $d ≥ 1$, when

$$|ξ|^{|β|/d}|Q^{(β)}(ξ)| ≤ C(1+|Q(ξ)|)$$

holds for all multi-indices $β$ and all $ξ ∈ ℝ^n$, with $C$ independent of both. Every constant-coefficient hypoelliptic operator is $d$-hypoelliptic for some $d$, and the smallest such $d$ governs the Gevrey scale of its solutions. Here the Gevrey class $Γ^p(Ω)$, $p ≥ 1$, is the space of smooth functions whose derivatives satisfy $|D^α u| ≤ A^{|α|+1}(α!)^p$ on each compact subset; $Γ^1$ is the class of real-analytic functions and larger $p$ means less regularity. Elliptic operators are exactly the case $d = 1$, while parabolic-type symbols such as that of the heat operator require $d = 2$. In this language, data in $Γ^p$ produce solutions in $Γ^{\max\{d,p\}}$.

For variable coefficients no characterization of hypoellipticity is known. The theorem of Hörmander (1958) and proved independently by Malgrange (1957), identifies a large sufficient class. The operator $P(x,D)$ is of constant strength and formally hypoelliptic if, first, for each fixed point $x_0$ the frozen operator $P(x_0,D)$ — the constant-coefficient operator obtained by freezing the coefficients at $x_0$ — is hypoelliptic in the constant-coefficient sense, and, second, the frozen operators are pairwise comparable in strength, that is $|P(x_1,ξ)| ≤ C(1+|P(x_2,ξ)|)$ for all points $x_1, x_2$ in a neighborhood and all frequencies $ξ$, with a constant independent of $ξ$. The theorem states that such operators are hypoelliptic. Peetre (1961) gave another proof, and Hörmander (1961) weakened the hypotheses through the condition called HE there, showing that the sufficient conditions are not optimal.

Hypoellipticity itself is a coordinate-free property: if $P$ is hypoelliptic and $χ$ is a diffeomorphism (a smooth bijective map with smooth inverse), the pulled-back operator is again hypoelliptic, because a distribution transfers under $χ$ together with its singular support, the complement of the largest open set on which the distribution coincides with a smooth function. The sufficient conditions of the 1958 class, however, are tied to coordinates. Malgrange (1957) observed explicitly that the constant-strength class is not invariant under differentiable changes of variables, and posed the problem of finding a large class that is; he also observed that his framework does not cover the multi-order elliptic systems of Douglis and Nirenberg, in which different components of a system are differentiated to different orders, and posed that question as well.

The later literature closed the quantitative axis. Newberger and Zieleźny (1973) determined the growth of hypoelliptic polynomials relative to Gevrey classes by the method of iterates, which characterizes the Gevrey class of a function through the growth of its iterates $Q^k u$ under an operator. Shafii-Mousavi and Zieleźny (1979) proved that if $P(x,D)$ has constant strength near a point, has coefficients in $Γ^p$ with $p > 1$, and the frozen operator is $d$-hypoelliptic, then solutions with data in $Γ^p$ lie in $Γ^{d p'}$ with $p' = \max\{d,p\}$. Fernández, Galbis and Jornet (2004) and Chaïli and Mahrouz (2025) extended the picture to ultradifferentiable classes of Roumieu type, where the factorial-type bound $(α!)^p$ is replaced by a general weight sequence $(M_p)$ with mild stability assumptions. The systems axis was developed by Volevich (1960, 1964) and continued in the Newton-polyhedron framework of Bouzar and Volevich (2004), in which the Newton polyhedron — the convex hull of the exponent vectors of the monomials appearing in the operator — records the relative size of terms. What the literature does not supply is the intrinsic formulation: a diffeomorphism-invariant class of hypoelliptic variable-coefficient operators with checkable membership containing the constant-strength class. Invariant sufficient conditions exist for special structures — ellipticity, and subellipticity for sums of squares of vector fields spanning by commutators (the commutator, or Lie bracket, of two vector fields is the new vector field measuring their failure to commute) — but a unified intrinsic characterization of hypoellipticity remains open, as the 2025 systematic survey of de Lessa Victor and Ragognette confirms for the neighboring characterization questions.

Problem Statement

The 1958 paper left three directions open. Later literature resolved the first — the Gevrey-scale estimates (Shafii-Mousavi and Zieleźny 1979 and successors) — and substantially developed the third, the systems extension (Volevich 1960, 1964 and successors). The surviving open core, posed by Malgrange (1957) in the parallel announcement of the same theorem, is the following.

Primary objective. Exhibit a class $𝒞$ of differential operators with variable coefficients, defined on open subsets of $ℝ^n$, such that all of the following hold:

1. Invariance: $𝒞$ is stable under diffeomorphisms — if $P ∈ 𝒞$ on an open set and $χ$ is a diffeomorphism, the pulled-back operator $χ^*P$ again belongs to $𝒞$.

2. Intrinsic checkability: membership of $P$ in $𝒞$ is effectively checkable, meaning it is decided by finitely many conditions on the coefficient data of $P$ itself — finitely many jet conditions on the coefficients together with finitely many symbol-type inequalities — and not by the hypoellipticity property of $P$ or of transformed operators.

3. Hypoellipticity: every $P ∈ 𝒞$ is hypoelliptic, and a proof of this is supplied.

4. Generality: $𝒞$ contains the elliptic operators, the operators of constant strength that are formally hypoelliptic in the sense of the 1958 theorem, and the subelliptic operators given by sums of squares of vector fields spanning by commutators.

Extension (systems). Formulate the intrinsic analogue for systems of operators, including the multi-order (Douglis-Nirenberg-type) structures that Malgrange's framework missed, and prove hypoellipticity there.

Alternatively, no class $𝒞$ satisfying requirements 1-4 exists — that is, every membership condition of the delimited form fixed in requirement 2, given by finitely many coefficient-jet conditions together with finitely many symbol-type inequalities, is incompatible with requirements 1, 3 and 4.

References

1. L. Hörmander, On the theory of general partial differential operators, Acta Mathematica 94 (1955) 161-248. DOI: 10.1007/BF02392492, https://doi.org/10.1007/BF02392492
2. L. Hörmander, On interior regularity of the solutions of partial differential equations, Communications on Pure and Applied Mathematics 11(2) (1958) 197-218. DOI: 10.1002/cpa.3160110205, https://doi.org/10.1002/cpa.3160110205
3. B. Malgrange, Sur une classe d'opérateurs différentiels hypoelliptiques, Bulletin de la Société Mathématique de France 85 (1957) 283-306. DOI: 10.24033/bsmf.1490, https://doi.org/10.24033/bsmf.1490
4. L. Hörmander, Hypoelliptic differential operators, Annales de l'Institut Fourier 11 (1961) 477-492. DOI: 10.5802/aif.117, https://doi.org/10.5802/aif.117
5. J. Peetre, A proof of the hypoellipticity of formally hypoelliptic differential operators, Communications on Pure and Applied Mathematics 14 (1961) 737-744. DOI: 10.1002/cpa.3160140406, https://doi.org/10.1002/cpa.3160140406
6. E. Newberger and Z. Zieleźny, The growth of hypoelliptic polynomials and Gevrey classes, Proceedings of the American Mathematical Society 39 (1973) 547-552. DOI: 10.1090/S0002-9939-1973-0318660-6, https://doi.org/10.1090/S0002-9939-1973-0318660-6
7. M. Shafii-Mousavi and Z. Zieleźny, On hypoelliptic differential operators of constant strength, Pacific Journal of Mathematics 82(1) (1979) 249-256. DOI: 10.2140/pjm.1979.82.249, https://doi.org/10.2140/pjm.1979.82.249
8. C. Fernández, A. Galbis and D. Jornet, ω-hypoelliptic differential operators of constant strength, Journal of Mathematical Analysis and Applications 297 (2004) 561-576. DOI: 10.1016/j.jmaa.2004.03.028, https://doi.org/10.1016/j.jmaa.2004.03.028
9. C. Bouzar and L. R. Volevich, Hypoelliptic systems connected with Newton's polyhedron, Mathematische Nachrichten 273 (2004) 14-27. DOI: 10.1002/mana.200310193, https://doi.org/10.1002/mana.200310193
10. A. Boiti, R. Chaïli and T. Mahrouz, Iterates of systems of operators in spaces of ω-ultradifferentiable functions, Annales Polonici Mathematici 118 (2016) 95-111 (preprint arXiv:1606.00222). DOI: 10.4064/ap4024-12-2016, https://doi.org/10.4064/ap4024-12-2016
11. H. Ghazaryan and V. Margaryan, On smooth solutions of a class of almost hypoelliptic equations of constant strength, Eurasian Mathematical Journal 10(4) (2019) 92-95. DOI: 10.32523/2077-9879-2019-10-4-92-95, https://doi.org/10.32523/2077-9879-2019-10-4-92-95
12. R. Chaïli and T. Mahrouz, Smoothness of solutions of differential equations of constant strength in Roumieu spaces, Georgian Mathematical Journal 32(3) (2025) 381-389. DOI: 10.1515/gmj-2024-2069, https://doi.org/10.1515/gmj-2024-2069 (preprint arXiv:2305.01258, https://arxiv.org/abs/2305.01258)
13. L. R. Volevich, A class of hypoelliptic systems, Soviet Mathematics - Doklady 1 (1960) 1190-1193; translation from Doklady Akademii Nauk SSSR 134 (1960) 1275-1278. zbMATH: 0107.30601, https://zbmath.org/3175472
14. L. R. Volevich, Hypoelliptic systems with variable coefficients, Soviet Mathematics - Doklady 5 (1964) 797-800; translation from Doklady Akademii Nauk SSSR 156 (1964) 1262-1265. zbMATH: 0173.36803, https://zbmath.org/3276773
15. B. de Lessa Victor and L. F. Ragognette, Relations between different types of Hypoellipticity: A systematic approach, preprint arXiv:2505.11801 (2025), https://arxiv.org/abs/2505.11801