Estimator selection in the gaussian setting

Baraud, Yannick; Giraud, Christophe; Huet, Sylvie

Annales de l'I.H.P. Probabilités et statistiques, Tome 50 (2014) no. 3, pp. 1092-1119.

Nous présentons une nouvelle procédure de sélection d’estimateurs pour estimer l’espérance $f$ d’un vecteur $Y$ de $n$ variables gaussiennes indépendantes dont la variance est inconnue. Nous proposons de choisir un estimateur de $f$, dont l’objectif est de minimiser le risque $l_2$, dans une collection arbitraire et éventuellement infinie $𝔽$ d’estimateurs. La procédure de choix ainsi que la collection $𝔽$ ne dépendent que des seules observations $Y$. Nous calculons une borne de risque, non asymptotique, ne nécessitant aucune hypothèse sur les estimateurs dans $𝔽$, ni la connaissance de leur dépendance en $Y$. Nous calculons des inégalités de type “oracle” quand $𝔽$ est une collection d’estimateurs linéaires. Nous considérons plusieurs cas particuliers : estimation par aggrégation, estimation par sélection de modèles, choix d’une fenêtre et du paramètre de lissage en régression fonctionnelle, choix du paramètre de régularisation dans un critère pénalisé. Pour tous ces cas particuliers, sauf pour les méthodes d’aggrégation, la méthode est très facile à programmer. A titre d’illustration nous montrons des résultats de simulations avec deux objectifs : comparer notre méthode à la procédure de cross-validation, montrer comment la mettre en œuvre dans le cadre de la sélection de variables.

We consider the problem of estimating the mean $f$ of a Gaussian vector $Y$ with independent components of common unknown variance ${\sigma }^2$. Our estimation procedure is based on estimator selection. More precisely, we start with an arbitrary and possibly infinite collection $𝔽$ of estimators of $f$ based on $Y$ and, with the same data $Y$, aim at selecting an estimator among $𝔽$ with the smallest Euclidean risk. No assumptions on the estimators are made and their dependencies with respect to $Y$ may be unknown. We establish a non-asymptotic risk bound for the selected estimator and derive oracle-type inequalities when $𝔽$ consists of linear estimators. As particular cases, our approach allows to handle the problems of aggregation, model selection as well as those of choosing a window and a kernel for estimating a regression function, or tuning the parameter involved in a penalized criterion. In all theses cases but aggregation, the method can be easily implemented. For illustration, we carry out two simulation studies. One aims at comparing our procedure to cross-validation for choosing a tuning parameter. The other shows how to implement our approach to solve the problem of variable selection in practice.

Classification : 62J05, 62J07, 62G08
Mots clés : estimator selection, model selection, variable selection, linear estimator, kernel estimator, ridge regression, Lasso, elastic net, random forest, PLS1 regression

[1] S. Arlot. Rééchantillonnage et Sélection de modèles. Ph.D. thesis, Univ. Paris XI, 2007.

[2] S. Arlot. Model selection by resampling penalization. Electron. J. Stat. 3 (2009) 557-624. | MR

[3] S. Arlot and F. Bach. Data-driven calibration of linear estimators with minimal penalties, 2011. Available at arXiv:0909.1884v2.

[4] S. Arlot and A. Celisse. A survey of cross-validation procedures for model selection. Stat. Surv. 4 (2010) 40-79. | MR | Zbl

[5] Y. Baraud. Model selection for regression on a fixed design. Probab. Theory Related Fields 117 (2000) 467-493. | MR | Zbl

[6] Y. Baraud. Estimator selection with respect to Hellinger-type risks. Probab. Theory Related Fields 151 (2011) 353-401. | MR

[7] Y. Baraud, C. Giraud and S. Huet. Gaussian model selection with an unknown variance. Ann. Statist. 37 (2009) 630-672. | MR | Zbl

[8] Y. Baraud, C. Giraud and S. Huet. Estimator selection in the Gaussian setting, 2010. Available at arXiv:1007.2096v1.

[9] L. Birgé. Model selection via testing: An alternative to (penalized) maximum likelihood estimators. Ann. Inst. Henri Poincaré Probab. Stat. 42 (2006) 273-325. | Numdam | MR

[10] L. Birgé and P. Massart. Gaussian model selection. J. Eur. Math. Soc. (JEMS) 3 (2001) 203-268. | MR | Zbl

[11] A. Boulesteix and K. Strimmer. Partial least squares: a versatile tool for the analysis of high-dimensional genomic data. Briefings in Bioinformatics 8 (2006) 32-44.

[12] L. Breiman. Random forests. Mach. Learn. 45 (2001) 5-32. | Zbl

[13] F. Bunea, A. B. Tsybakov and M. H. Wegkamp. Aggregation for Gaussian regression. Ann. Statist. 35 (2007) 1674-1697. | MR | Zbl

[14] E. Candès and T. Tao. The Dantzig selector: Statistical estimation when $p$ is much larger than $n$. Ann. Statist. 35 (2007) 2313-2351. | MR | Zbl

[15] Y. Cao and Y. Golubev. On oracle inequalities related to smoothing splines. Math. Methods Statist. 15 (2006) 398-414. | MR

[16] O. Catoni. Mixture approach to universal model selection. Technical report, Ecole Normale Supérieure, France, 1997. | Zbl

[17] O. Catoni. Statistical learning theory and stochastic optimization. In Lecture Notes from the 31st Summer School on Probability Theory Held in Saint-Flour, July 8-25, 2001. Springer, Berlin, 2004. | MR | Zbl

[18] A. Celisse. Model selection via cross-validation in density estimation, regression, and change-points detection. Ph.D. thesis, Univ. Paris XI, 2008.

[19] S. S. Chen, D. L. Donoho and M. A. Saunders. Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20 (1998) 33-61 (electronic). | MR | Zbl

[20] R. Díaz-Uriarte and S. Alvares De Andrés. Gene selection and classification of microarray data using random forest. BMC Bioinformatics 7 (2006) 3.

[21] B. Efron, T. Hastie, I. Johnstone and R. Tibshirani. Least angle regression. Ann. Statist. 32 (2004) 407-499. With discussion, and a rejoinder by the authors. | MR | Zbl

[22] R. Genuer, J.-M. Poggi and C. Tuleau-Malot. Variable selection using random forests. Patter Recognition Lett. 31 (2010) 2225-2236.

[23] C. Giraud. Mixing least-squares estimators when the variance is unknown. Bernoulli 14 (2008) 1089-1107. | MR | Zbl

[24] C. Giraud, S. Huet and N. Verzelen. High-dimensional regression with unknown variance. Statist. Sci. 27 (2013) 500-518. | MR

[25] A. Goldenshluger. A universal procedure for aggregating estimators. Ann. Statist. 37 (2009) 542-568. | MR | Zbl

[26] A. Goldenshluger and O. Lepski. Structural adaptation via ${𝕃}_p$-norm oracle inequalities. Probab. Theory Related Fields 143 (2009) 41-71. | MR | Zbl

[27] I. Helland. Partial least squares regression. In Encyclopedia of Statistical Sciences, 2nd edition 9 5957-5962. S. Kotz, N. Balakrishnan, C. Read, B. Vidakovic and N. Johnston (Eds.). Wiley, New York, 2006.

[28] I. Helland. Some theoretical aspects of partial least squares regression. Chemometrics and Intelligent Laboratory Systems 58 (2001) 97-107.

[29] A. Hoerl and R. Kennard. Ridge regression: Bayes estimation for nonorthogonal problems. Technometrics 12 (1970) 55-67. | Zbl

[30] A. Hoerl and R. Kennard. Ridge regression. In Encyclopedia of Statistical Sciences, 2nd edition 11 7273-7280. S. Kotz, N. Balakrishnan, C. Read, B. Vidakovic and N. Johnston (Eds.). Wiley, New York, 2006. | Zbl

[31] J. Huang, S. Ma and C.-H. Zhang. Adaptive Lasso for sparse high-dimensional regression models. Statist. Sinica 4 (2008) 1603-1618. | MR | Zbl

[32] A. Juditsky and A. Nemirovski. Functional aggregation for nonparametric regression. Ann. Statist. 28 (2000) 681-712. | MR | Zbl

[33] O. V. Lepskiĭ. A problem of adaptive estimation in Gaussian white noise. Teor. Veroyatnost. i Primenen. 35 (1990) 459-470. | MR | Zbl

[34] O. V. Lepskiĭ. Asymptotically minimax adaptive estimation. I. Upper bounds. Optimally adaptive estimates. Teor. Veroyatnost. i Primenen. 36 (1991) 645-659. | MR | Zbl

[35] O. V. Lepskiĭ. Asymptotically minimax adaptive estimation. II. Schemes without optimal adaptation. Adaptive estimates. Teor. Veroyatnost. i Primenen. 37 (1992) 468-481. | MR | Zbl

[36] O. V. Lepskiĭ. On problems of adaptive estimation in white Gaussian noise. In Topics in Nonparametric Estimation 87-106. Adv. Soviet Math. 12. Amer. Math. Soc., Providence, RI, 1992. | MR | Zbl

[37] G. Leung and A. R. Barron. Information theory and mixing least-squares regressions. IEEE Trans. Inform. Theory 52 (2006) 3396-3410. | MR

[38] Y. Makovoz. Random approximants and neural networks. J. Approx. Theory 85 (1996) 98-109. | MR | Zbl

[39] E. A. Nadaraya. On estimating regression. Theory Probab. Appl. 9 (1964) 141-142. | Zbl

[40] A. Nemirovski. Topics in non-parametric statistics. In Lectures on probability theory and statistics (Saint-Flour, 1998) 85-277. Lecture Notes in Math. 1738. Springer, Berlin, 2000. | MR | Zbl

[41] P. Rigollet and A. B. Tsybakov. Linear and convex aggregation of density estimators. Math. Methods Statist. 16 (2007) 260-280. | MR | Zbl

[42] J. Salmon and A. Dalalyan. Optimal aggregation of affine estimators. J. Mach. Learn. Res. 19 (2011) 635-660.

[43] C. Strobl, A.-L. Boulesteix, T. Kneib, T. Augustin and A. Zeileis. Conditional variable importance for random forests. BMC Bioinformatics 9 (2008) 307.

[44] C. Strobl, A.-L. Boulesteix, A. Zeileis and T. Hothorn. Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC Bioinformatics 8 (2007) 25.

[45] M. Tenenhaus. La régression PLS. Éditions Technip, Paris. Théorie et pratique, 1998. [Theory and application]. | MR | Zbl

[46] R. Tibshirani. Regression shrinkage and selection via the Lasso. J. Roy. Statist. Soc. Ser. B 58 (1996) 267-288. | MR | Zbl

[47] A. B. Tsybakov. Optimal rates of aggregation. In Proceedings of the 16th Annual Conference on Learning Theory (COLT) and 7th Annual Workshop on Kernel Machines 303-313. Lecture Notes in Artificial Intelligence 2777. Springer, Berlin, 2003. | Zbl

[48] G. S. Watson. Smooth regression analysis. Sankhyā Ser. A 26 (1964) 359-372. | MR | Zbl

[49] M. Wegkamp. Model selection in nonparametric regression. Ann. Statist. 31 (2003) 252-273. | MR | Zbl

[50] Y. Yang. Model selection for nonparametric regression. Statist. Sinica 9 (1999) 475-499. | MR | Zbl

[51] Y. Yang. Combining different procedures for adaptive regression. J. Multivariate Anal. 74 (2000) 135-161. | MR | Zbl

[52] Y. Yang. Mixing strategies for density estimation. Ann. Statist. 28 (2000) 75-87. | MR | Zbl

[53] Y. Yang. Adaptive regression by mixing. J. Amer. Statist. Assoc. 96 (2001) 574-588. | MR | Zbl

[54] T. Zhang. Learning bounds for kernel regression using effective data dimensionality. Neural Comput. 17 (2005) 2077-2098. | MR | Zbl

[55] T. Zhang. Adaptive forward-backward greedy algorithm for learning sparse representations. Technical report, Rutgers Univ., NJ, 2008.

[56] P. Zhao and B. Yu. On model selection consistency of Lasso. J. Mach. Learn. Res. 7 (2006) 2541-2563. | MR | Zbl

[57] H. Zou. The adaptive Lasso and its oracle properties. J. Amer. Statist. Assoc. 101 (2006) 1418-1429. | MR | Zbl

[58] H. Zou and T. Hastie. Regularization and variable selection via the elastic net. J. R. Stat. Soc. Ser. B Stat. Methodol. 67 (2005) 301-320. | MR | Zbl

[59] H. Zou, T. Hastie and R. Tibshirani On the “degrees of freedom” of the Lasso. Ann. Statist. 35 (2007) 2173-2192. | MR | Zbl