Completing the relative-efficiency theory of kernel versus local log-polynomial density estimators: full-support asymptotics, sharp worst-case constants, and structural classification

Hall and Tao (2002, JRSS-B) compared the global $L^2$ performance of kernel density estimators and local likelihood (local log-polynomial) density estimators away from boundaries: the two have virtually identical integrated variances, but the kernel estimator's integrated squared bias (ISB) is smaller than that of the degree-1 local log-linear estimator by as much as a factor 4, a bound obtained from the sharp inequality $\int (f')^4/f^2 < 9\int (f'')^2$; for degree $p\ge 2$ the worst-case ISB ratio over multimodal smooth density classes is unbounded, while over unimodal classes it is finite but its exact value $S_p$ was not determined (only explicit lower bounds $T_p$ were given). Their theory was established only on regions where $f$ exceeds a threshold $\varepsilon$, using truncated or ridged (penalized) estimator variants, and the sharp constants are attained only along degenerating sequences of densities. The open problem is to complete this quantitative efficiency theory: (a) a unified full-support asymptotic theory for the mean integrated squared error (MISE) of local log-polynomial estimators covering points where the density vanishes, without ad hoc estimator modifications; (b) exact determination of the worst-case constants — the value of $S_p$ for unimodal density classes and the extremal/stability analysis of the sharp inequalities underlying the factor-4 bound; and (c) a structural characterization of the features of $f$ that make local log-polynomial estimation superior versus catastrophically inferior to kernel estimation, beyond the special class of exponentiated polynomials.

Background

Nonparametric density estimation asks for an estimate of an unknown probability density $f$ from an independent identically distributed sample. The classical kernel density estimator (KDE) is $\hat f_{\mathrm{ker}}(x)=(nh)^{-1}\sum_{i=1}^n K((X_i-x)/h)$, where $K$ is a fixed symmetric probability kernel (a bounded, symmetric, compactly supported density) and the bandwidth $h\to 0$ controls the smoothing. Its performance is measured by the mean integrated squared error, $\mathrm{MISE}=\mathrm{E}\int(\hat f-f)^2$, which decomposes into integrated squared bias (ISB) plus integrated variance; standard theory gives $\mathrm{MISE}(h)=h^4\int b(x)^2\,dx+(nh)^{-1}\kappa+o\{h^4+(nh)^{-1}\}$, where $b$ is the pointwise bias coefficient, $\kappa=\int K^2$, and the kernel moments are $\kappa_j=\int u^jK(u)\,du$.

Local likelihood density estimation (Tibshirani and Hastie, 1987; Loader, 1996; Hjort and Jones, 1996) replaces the fixed kernel weights with a local parametric fit: at each point $x$ one fits an exponentiated polynomial $\psi(u)=\exp\{\sum_{j=0}^p\theta_j(u-x)^j\}$ by maximizing a kernel-weighted log-likelihood in $\theta$, and takes $\hat f(x)=\exp(\hat\theta_0)$. Such estimators (local log-polynomial estimators, of which the degree-1 version is the local log-linear estimator) are automatically nonnegative, have good behavior near support boundaries and in tails, and nest parametric fitting at large bandwidths. Their drawback is that the estimator is defined implicitly through a nonlinear maximization.

Hall and Tao (2002) made the first sharp global $L^2$ comparison of the two families in interior regions. Pointwise, the local log-linear estimator has bias coefficient $b_{\mathrm{lin}}(x)=\tfrac12\kappa_2\{f''(x)-[f'(x)]^2/f(x)\}$ against the kernel's $b_{\mathrm{ker}}(x)=\tfrac12\kappa_2 f''(x)$, with identical variance terms, so all global differences come from bias. Integrating, they proved the exact identity $\int b_{\mathrm{lin}}^2=\int b_{\mathrm{ker}}^2+\tfrac{1}{12}\kappa_2^2\int (f')^4/f^2$, and then the sharp inequality $\int (f')^4/f^2<9\int (f'')^2$ over the class $\mathcal{C}_2$ of compactly supported densities with one continuous derivative on the line, two piecewise monotone derivatives interior to the support, and $\int (f'')^2<\infty$. Since $\int b_{\mathrm{ker}}^2=\tfrac14\kappa_2^2\int (f'')^2$, this yields $\int b_{\mathrm{lin}}^2<4\int b_{\mathrm{ker}}^2$, with the constant 4 sharp: it is approached only along sequences of densities with endpoint behavior $(1-|x|)^{3/2+\varepsilon}$ as $\varepsilon\downarrow 0$, never attained by a single density. In sample-size terms, the worst case costs a factor $4^{1/4}$ of effective sample size.

For degree $p\ge 2$ compared with $k$th-order kernels (a higher-order kernel $L_k$ satisfies $\int L_k=1$, $\int u^jL_k=0$ for $1\le j\le k-1$, and is used to kill lower-order bias; here $k=p+1$ for odd $p$, $k=p+2$ for even $p$), the local log-polynomial bias coefficient involves derivatives of $\log f$, namely $\beta_{\mathrm{pol},p}=(\log f)^{(k)}f$ for odd $p$ and $\beta_{\mathrm{pol},p}=(\log f)^{(k)}f+k(\log f)^{(k-1)}f'$ for even $p$. Over the analogous class $\mathcal{C}_k$, Hall and Tao showed the ISB ratio of local log-polynomial to kernel estimators is unbounded, even for symmetric multimodal densities on $[-1,1]$ (constructed bimodal sequences whose between-mode density is $O(C^{-3})$ while $\int(f^{(k)})^2\to 0$). For densities that are exponentiated polynomials — $\log f$ itself a polynomial of degree $\le p$ — the local model is exactly correct and the gains are unbounded in the opposite direction. Restricted to unimodal densities in $\mathcal{C}_k$ (a single local maximum), the worst-case ratio $S_p=\sup \int b_{\mathrm{pol}}^2/\int b_{\mathrm{ker}}^2$ is finite, proved via $L^2$-boundedness of the one-sided Hardy–Littlewood maximal function (an averaging operator $Mg(x)=\sup_{r>0}r^{-1}\int_{(x-r,x)}|g|$ whose $L^2$ bound controls log-derivative functionals of monotone densities), and explicit lower bounds $T_p$ were computed from unimodal families $(1-x^2)^{k-1/2+\varepsilon}$; but the exact value of $S_p$ was not determined.

The theory has documented gaps. First, the integrated-MISE statements were established by integrating only over regions $\mathcal{S}(\varepsilon)=\{x: f(x)>\varepsilon\}$ and by modifying the estimator (e.g. truncation $\hat f_{\mathrm{lin}}=\min(\tilde f_{\mathrm{lin}},C)$, or ridging, meaning a ridge-type penalty added to the local log-likelihood so the fit remains stable where data are sparse); behavior at and near points where $f$ vanishes — exactly where the near-extremal families live — was not covered, and no unified asymptotic theory uniform down to zeros of $f$ was developed. Second, the sharp constants (9 and 4) are attained only as limits along degenerating sequences; the densities that come closest to extremality are uncharacterized. Third, $S_p$ is known only to be finite with lower bounds. Fourth, no classification of the structural features of $f$ that separate good from catastrophic performance of local log-polynomial estimators (beyond the exponentiated-polynomial class) is known. These gaps are the present problem.

Problem Statement

Complete the quantitative relative-efficiency theory of kernel versus local log-polynomial density estimators that Hall and Tao (2002) initiated. The three requested components serve one objective — knowing exactly when and by how much local likelihood density estimation can lose (or win) against kernel density estimation in global $L^2$ performance — and are linked by the same bias functionals:

(1) Full-support asymptotic theory. Establish (or refute the possibility of) asymptotic MISE expansions, of the same form and with the same leading terms as the interior theory, for local log-linear and local log-polynomial density estimators that hold uniformly on the entire support of $f$, including points where $f$ vanishes or is arbitrarily small, without restricting integration to sets $\mathcal{S}(\varepsilon)=\{x: f(x)>\varepsilon\}$ and without ad hoc truncation or ridging modifications of the estimator; or prove that no such unified theory exists by exhibiting, within the smoothness classes $\mathcal{C}_2$ and $\mathcal{C}_k$ defined below, densities for which the interior leading terms fail near zeros of $f$.

(2) Sharp worst-case constants and extremals. Two linked parts. (2a) Determine the exact value of the worst-case integrated-squared-bias ratio $S_p=\sup_{f\in\mathcal{C}_{\mathrm{uni},k}}\int b_{\mathrm{pol}}^2/\int b_{\mathrm{ker}}^2$ over the class of unimodal densities in $\mathcal{C}_k$ (the source established finiteness and explicit lower bounds $T_p$, but not the value). (2b) Characterize the extremal and near-extremal densities of the sharp inequality $\int (f')^4/f^2<9\int (f'')^2$ and of the resulting factor-4 bound, beyond the known extremizing family $f_\varepsilon\propto(1-|x|)^{3/2+\varepsilon}$ ($\varepsilon\downarrow 0$) and its non-attainment, including a quantitative description of how the ratio degrades as densities depart from extremality.

(3) Structural classification. Delineate the structural features of the target density — such as, but not restricted to, modality, the depth of inter-modal minima, and the tail exponent at support endpoints — that guarantee superiority of local log-polynomial estimators over kernel estimators, that guarantee bounded relative inefficiency, and that permit unbounded inefficiency, where each of these three regimes is measured by the same integrated-squared-bias ratio $\int b_{\mathrm{pol}}^2/\int b_{\mathrm{ker}}^2$ of the degree-$p$ local log-polynomial estimator to its comparable $k$th-order kernel estimator that underlies components (1) and (2), beyond the known special class of exponentiated polynomials (densities whose logarithm is a polynomial of degree at most $p$, for which the local model is exactly correct) and the known symmetric multimodal constructions exhibiting unbounded inefficiency for degrees $p\ge 2$. For each of the three regimes, determine the structural conditions that are sufficient to guarantee it, and determine which of those conditions are sharp (cannot be relaxed).

Here $b_{\mathrm{lin}}(x)=\tfrac12\kappa_2\{f''(x)-[f'(x)]^2/f(x)\}$ and $b_{\mathrm{ker}}(x)=\tfrac12\kappa_2 f''(x)$ for degree 1, and for degree $p\ge 2$ with $k=p+1$ (odd $p$) or $k=p+2$ (even $p$), $b_{\mathrm{pol}}(x)=(1/k!)\lambda_k\beta_{\mathrm{pol},p}(x)$ with $\beta_{\mathrm{pol},p}=(\log f)^{(k)}f$ (odd $p$) or $(\log f)^{(k)}f+k(\log f)^{(k-1)}f'$ (even $p$) against $b_{\mathrm{ker}}(x)=(1/k!)\lambda_k f^{(k)}(x)$ for the comparable $k$th-order kernel estimator, where $\lambda_k=\int u^kL_k(u)\,du$ is the $k$th moment of that higher-order kernel; $\mathcal{C}_k$ is the class of compactly supported densities whose first $k-1$ derivatives are continuous on the real line, whose first $k$ derivatives are each piecewise monotone on the interior of the support, and for which $\int(f^{(k)})^2<\infty$; and $\mathcal{C}_{\mathrm{uni},k}$ its unimodal subclass. Answers must preserve the generality of these classes; no restriction to specific parametric families, finite samples, or particular kernels beyond the standard moment conditions is part of the question.

References

1. Peter Hall and Terence Tao (2002), 'Relative Efficiencies of Kernel and Local Likelihood Density Estimators', Journal of the Royal Statistical Society Series B (Statistical Methodology) 64(3):537-547. DOI: 10.1111/1467-9868.00349, https://doi.org/10.1111/1467-9868.00349
2. Catherine R. Loader (1996), 'Local likelihood density estimation', The Annals of Statistics 24(4):1602-1618. DOI: 10.1214/aos/1032298287, https://doi.org/10.1214/aos/1032298287
3. N. L. Hjort and M. C. Jones (1996), 'Locally parametric nonparametric density estimation', The Annals of Statistics 24(4):1619-1647. DOI: 10.1214/aos/1032298288, https://doi.org/10.1214/aos/1032298288
4. Robert Tibshirani and Trevor Hastie (1987), 'Local Likelihood Estimation', Journal of the American Statistical Association 82(398):559-567. DOI: 10.1080/01621459.1987.10478466, https://doi.org/10.1080/01621459.1987.10478466
5. Pedro Delicado (2006), 'Local likelihood density estimation based on smooth truncation', Biometrika 93(2):472-480. DOI: 10.1093/biomet/93.2.472, https://doi.org/10.1093/biomet/93.2.472
6. Håkon Otneim, Hans Arnfinn Karlsen and Dag Tjøstheim (2013), 'Bias and bandwidth for local likelihood density estimation', Statistics & Probability Letters 83(5):1382-1387. DOI: 10.1016/j.spl.2013.02.003, https://doi.org/10.1016/j.spl.2013.02.003
7. Simon J. Sheather (2004), 'Density Estimation', Statistical Science 19(4). DOI: 10.1214/088342304000000297, https://doi.org/10.1214/088342304000000297
8. Gery Geenens (2014), 'Probit Transformation for Kernel Density Estimation on the Unit Interval', Journal of the American Statistical Association 109(505):346-358. DOI: 10.1080/01621459.2013.842173, arXiv:1303.4121, https://doi.org/10.1080/01621459.2013.842173
9. Gery Geenens and Craig Wang (2018), 'Local-Likelihood Transformation Kernel Density Estimation for Positive Random Variables', Journal of Computational and Graphical Statistics 27(4):822-835. DOI: 10.1080/10618600.2018.1424636, arXiv:1602.04862, https://doi.org/10.1080/10618600.2018.1424636
10. D. P. Amali Dassanayake, Igor Volobouev and A. Alexandre Trindade (2017), 'Local orthogonal polynomial expansion for density estimation', Journal of Nonparametric Statistics 29(4):806-830. DOI: 10.1080/10485252.2017.1371715, arXiv:1505.00275, https://doi.org/10.1080/10485252.2017.1371715
11. Byeong U. Park, W. C. Kim and M. C. Jones (2002), 'On local likelihood density estimation', The Annals of Statistics 30(5):1480-1495. DOI: 10.1214/aos/1035844984, https://doi.org/10.1214/aos/1035844984
12. Byeong U. Park, Young Kyung Lee, Tae Yoon Kim, Cheolyong Park and Shinto Eguchi (2006), 'On local likelihood density estimation when the bandwidth is large', Journal of Statistical Planning and Inference 136(3):839-859. DOI: 10.1016/j.jspi.2004.08.019, https://doi.org/10.1016/j.jspi.2004.08.019
13. Sergey Dovgal (2016), 'Towards Model Selection for Local Log-Density Estimation. Fisher and Wilks-type theorems', arXiv preprint. arXiv:1607.00806, https://arxiv.org/abs/1607.00806
14. Panagiotis Avramidis (2016), 'Adaptive likelihood estimator of conditional variance function', Journal of Nonparametric Statistics 28(1):132-151. DOI: 10.1080/10485252.2015.1122189, https://doi.org/10.1080/10485252.2015.1122189
15. Kee-Hoon Kang and Jung-Hoon Han (2003), 'On Practical Efficiency of Locally Parametric Nonparametric Density Estimation Based on Local Likelihood Function', Communications of the Korean Statistical Society 10(2):607-617 (journal now titled Communications for Statistical Applications and Methods). DOI: 10.5351/CKSS.2003.10.2.607, https://doi.org/10.5351/CKSS.2003.10.2.607