Feature selection for functional data via blinding beyond the fixed-$p$ regime: consistency with a diverging number of candidate functions and principled algorithm tuning

The blinding framework of Fraiman, Gimenez and Svarc (Journal of Multivariate Analysis, 2016) selects a small subset of user-provided features for functional data by replacing the random trajectory with its conditional expectation given the chosen features and asking when the outputs of a statistical procedure (classification, principal components, or functional regression) are preserved. Its consistency theorems are proved only for a fixed number $p$ of candidate features, under an assumed sparse informative subset, and with hand-tuned parameters of the stochastic subset-search heuristic. The authors explicitly leave open: extending the theory to $p\to\infty$ (requiring asymptotics that keep the nonparametric estimation hypothesis H1 valid and Besicovitch-type regularity of the regression function), a principled data-driven choice of the search parameters $(\epsilon, d_1, N_0, N_1, r)$, and a formal theory linking functional selection to discrete high-dimensional selection including the number of observation points needed. These issues are not resolved in the subsequent literature: the closest follow-ups either extend the idea to parametric multivariate models, or replace the framework with different regularized formulations having no selection-consistency theory for the blinding procedure.

Background

In functional data analysis, each observation is a random curve $X=\{X(t):t\in[a,b]\}$ viewed as an element of $L^2[a,b]$, observed in practice on a finite grid. A recurring task is feature selection: given a statistical procedure already carried out on such data, identify a small set of interpretable features that carry the information the procedure uses. Candidate features are defined by a user-chosen family of functions $\mathcal A=\{f_1,\dots,f_p\}$ applied to the trajectory — for example pointwise evaluations $X(t_j)$, local averages over subintervals, occupation measures (the time the trajectory spends in a given interval), or counts of up-crossings of a level.

Fraiman, Justel and Svarc (2008) introduced the 'blinding' idea for multivariate data: blind (average out) unnecessary variables by replacing them with conditional expectations, and keep the subsets of variables that best reproduce the output of the statistical procedure. Fraiman, Gimenez and Svarc (2016) extended this to functional data. For a subset $I\subseteq\{1,\dots,p\}$, the blinded process is $Z(I)(t)=E(X(t)\mid f(I))$, where $f(I)$ is the random vector of selected feature values. A discrepancy $h(I)$ measures how much the output of the procedure changes when $X$ is replaced by $Z(I)$ — for classification, a matching error between the labels assigned to the original and blinded trajectories; for principal components analysis (PCA, a standard dimension-reduction technique), the distance between the eigen-elements computed from the original and blinded processes; for functional regression, the mean squared distance between predictions. The target is the family $I_0$ of subsets of size at most $d$ minimizing $h(I)$. Given a sample, one estimates $Z(I)$ nonparametrically — for instance by $r$-nearest-neighbor ($r$-NN) regression, averaging the $r$ sample curves whose feature vectors $f(I)$ are closest — and minimizes the empirical counterpart $h_n(I)$. The key hypothesis H1 asserts that these estimates are strongly consistent for $Z(I)$; under H1 and additional problem-specific conditions, the papers prove that the empirical minimizers converge to the population-optimal subsets for classification, PCA, and functional regression. All of these results assume sparsity: a small informative subset of $\mathcal A$ exists. Because exhaustive search over subsets is infeasible for moderate $p$, a stochastic forward-selection heuristic is used, depending on user-chosen parameters: a tolerance $\epsilon$, an exhaustive-step depth $d_1$, search counts $N_0$ and $N_1$, and the number of nearest neighbors $r$.

Section 7 of the source acknowledges the following unresolved issues: (1) no principled or data-driven strategy for choosing $(\epsilon,d_1,N_0,N_1,r)$ — the authors state that an optimal way to choose them is beyond the scope of the paper; (2) the theory assumes $p$ fixed, and extending it to $p\to\infty$ would require new asymptotics ensuring H1 remains valid in growing feature dimension, plus Besicovitch-type regularity of the regression function — referring to the Stone–Besicovitch conditions of Forzani, Fraiman and Llop (2012), under which nonparametric functional regression stays consistent in infinite-dimensional settings; (3) the link with discrete high-dimensional variable selection — such as the LASSO (least absolute shrinkage and selection operator, a penalized regression that sets some coefficients exactly to zero) applied to discretized curves — including theoretical results on how many observation points suffice, is flagged as needing deep study, echoing Cuevas (2014) and Kneip and Sarda (2011). Sparsity of an informative subset, by contrast, is noted there as a standing assumption of all the statistical procedures studied, not formulated as an open problem.

Problem Statement

Establish the statistical theory of the functional blinding feature-selection framework of Fraiman, Gimenez and Svarc (2016) beyond the fixed-$p$, hand-tuned regime in which its consistency is currently proved, while holding the source's sparse-informative-subset assumption fixed. The common objective is to make the selection consistent — and the search procedure trustworthy — precisely in the high-dimensional situations the method is meant for. Concretely, with $p=p_n$ candidate features and sample size $n$: (a) prove consistency of the empirical optimal subsets (for the classification, PCA, and functional-regression objectives) when $p_n\to\infty$, including verifiable conditions that keep the nonparametric estimation hypothesis H1 valid, together with the required Besicovitch/Stone-type regularity of the regression function; (b) provide a principled, data-driven rule for selecting the stochastic search parameters $(\epsilon,d_1,N_0,N_1,r)$, together with a theoretical guarantee that the selection made under the tuned rule is consistent in the sense of (a); and (c) develop the formal comparison between the functional blinding approach and discrete high-dimensional selection methods applied to discretized curves — for each of the objectives in (a) — including how many observation points (grid size) are needed as a function of the sample size $n$ and the corresponding selection-consistency rates. The three sub-questions (a)–(c) are jointly posed: the $p\to\infty$ extension (a) is the central issue identified by the authors, while (b)–(c) are the complementary acknowledged open problems of the same framework.

References

1. Ricardo Fraiman, Yanina Gimenez, Marcela Svarc. Feature selection for functional data. Journal of Multivariate Analysis 146 (2016) 191–208. DOI: 10.1016/j.jmva.2015.09.006. https://doi.org/10.1016/j.jmva.2015.09.006 (preprint: arXiv:1502.02123, https://arxiv.org/abs/1502.02123)
2. Fraiman, Justel, Svarc. Selection of variables for cluster analysis and classification rules. Journal of the American Statistical Association 103(483) (2008) 1294–1303. DOI: 10.1198/016214508000000544. https://doi.org/10.1198/016214508000000544
3. Forzani, Fraiman, Llop. Consistent nonparametric regression for functional data under the Stone–Besicovitch conditions. IEEE Transactions on Information Theory 58(11) (2012) 6697–6708. DOI: 10.1109/tit.2012.2209628. https://doi.org/10.1109/TIT.2012.2209628
4. Ricardo Fraiman, Yanina Gimenez, Marcela Svarc. Seeking relevant information from a statistical model. ESAIM: Probability and Statistics 20 (2016) 463–479. DOI: 10.1051/ps/2016022. https://doi.org/10.1051/ps/2016022
5. Tobia Boschi, Francesca Bonin, Rodrigo Ordonez-Hurtado, Alessandra Pascale, Jonathan Epperlein. A new computationally efficient algorithm to solve Feature Selection for Functional Data Classification in high-dimensional spaces. ICML 2024; arXiv:2401.05765. https://arxiv.org/abs/2401.05765
6. José L. Torrecilla, Alberto Suárez. Feature selection in functional data classification with recursive maxima hunting. NeurIPS 2016; arXiv:1806.02922. https://arxiv.org/abs/1806.02922
7. Berrendero, Bueno-Larraz, Cuevas. An RKHS model for variable selection in functional linear regression. Journal of Multivariate Analysis 170 (2019) 25–45. DOI: 10.1016/j.jmva.2018.04.008. https://doi.org/10.1016/j.jmva.2018.04.008
8. Kneip, Poß, Sarda. Functional linear regression with points of impact. The Annals of Statistics 44(1) (2016) 1–30. DOI: 10.1214/15-aos1323. https://doi.org/10.1214/15-AOS1323
9. Kneip, Sarda. Factor models and variable selection in high-dimensional regression analysis. The Annals of Statistics 39(5) (2011) 2410–2447. DOI: 10.1214/11-aos905. https://doi.org/10.1214/11-AOS905
10. Cuevas. A partial overview of the theory of statistics with functional data. Journal of Statistical Planning and Inference 147 (2014) 1–23. DOI: 10.1016/j.jspi.2013.04.002. https://doi.org/10.1016/j.jspi.2013.04.002