Estimating composite functions by model selection

Baraud, Yannick; Birgé, Lucien

doi:10.1214/12-AIHP516

Baraud, Yannick ; Birgé, Lucien

Annales de l'I.H.P. Probabilités et statistiques, Tome 50 (2014) no. 1, pp. 285-314.

Résumé
Abstract

Cet article traite du problème de l’estimation d’une fonction $s$ définie sur ${[- 1, 1]}^{k}$ lorsque $k$ est grand en utilisant des approximations de $s$ par des fonctions composées de la forme $g \circ u$ . Notre solution est fondée sur la sélection de modèle et conduit, pour résoudre ce problème, à une approche très générale tant sur les possibilités de choix des fonctions $g$ et $u$ que sur les cadres statistiques d’application. En particulier, et entre autres exemples, nous considérons l’approximation de $s$ par des fonctions additives, des modèles de type “single” ou “multiple index”, des réseaux de neurones, ou des mélanges de densités gaussiennes lorsque $s$ est elle-même une densité. Nous étudions également le cas où $s$ est exactement de la forme $g \circ u$ pour des fonctions $g$ et $u$ appartenant à des classes de régularités qui peuvent être anisotropes. Dans ce cas, notre approche conduit à un estimateur complètement adaptatif par rapport aux régularités de $g$ et $u$ .

We consider the problem of estimating a function $s$ on ${[- 1, 1]}^{k}$ for large values of $k$ by looking for some best approximation of $s$ by composite functions of the form $g \circ u$ . Our solution is based on model selection and leads to a very general approach to solve this problem with respect to many different types of functions $g, u$ and statistical frameworks. In particular, we handle the problems of approximating $s$ by additive functions, single and multiple index models, artificial neural networks, mixtures of Gaussian densities (when $s$ is a density) among other examples. We also investigate the situation where $s = g \circ u$ for functions $g$ and $u$ belonging to possibly anisotropic smoothness classes. In this case, our approach leads to a completely adaptive estimator with respect to the regularities of $g$ and $u$ .

MR Zbl | 2 citations dans Numdam

DOI : 10.1214/12-AIHP516

Classification : 62G05
Mots-clés : curve estimation, model selection, composite functions, adaptation, single index model, artificial neural networks, gaussian mixtures

@article{AIHPB_2014__50_1_285_0,
     author = {Baraud, Yannick and Birg\'e, Lucien},
     title = {Estimating composite functions by model selection},
     journal = {Annales de l'I.H.P. Probabilit\'es et statistiques},
     pages = {285--314},
     publisher = {Gauthier-Villars},
     volume = {50},
     number = {1},
     year = {2014},
     doi = {10.1214/12-AIHP516},
     mrnumber = {3161532},
     zbl = {1281.62093},
     language = {en},
     url = {https://www.numdam.org/articles/10.1214/12-AIHP516/}
}

TY  - JOUR
AU  - Baraud, Yannick
AU  - Birgé, Lucien
TI  - Estimating composite functions by model selection
JO  - Annales de l'I.H.P. Probabilités et statistiques
PY  - 2014
SP  - 285
EP  - 314
VL  - 50
IS  - 1
PB  - Gauthier-Villars
UR  - https://www.numdam.org/articles/10.1214/12-AIHP516/
DO  - 10.1214/12-AIHP516
LA  - en
ID  - AIHPB_2014__50_1_285_0
ER  -

%0 Journal Article
%A Baraud, Yannick
%A Birgé, Lucien
%T Estimating composite functions by model selection
%J Annales de l'I.H.P. Probabilités et statistiques
%D 2014
%P 285-314
%V 50
%N 1
%I Gauthier-Villars
%U https://www.numdam.org/articles/10.1214/12-AIHP516/
%R 10.1214/12-AIHP516
%G en
%F AIHPB_2014__50_1_285_0

Baraud, Yannick; Birgé, Lucien. Estimating composite functions by model selection. Annales de l'I.H.P. Probabilités et statistiques, Tome 50 (2014) no. 1, pp. 285-314. doi : 10.1214/12-AIHP516. https://www.numdam.org/articles/10.1214/12-AIHP516/

Bibliographie
Cité par

[1] N. Akakpo. Adaptation to anisotropy and inhomogeneity via dyadic piecewise polynomial selection. Math. Methods Statist. 21 (2012) 1-28. | MR

[2] Y. Baraud. Estimator selection with respect to Hellinger-type risks. Probab. Theory Related Fields 151 (2011) 353-401. | MR

[3] Y. Baraud, F. Comte and G. Viennet. Model selection for (auto-)regression with dependent data. ESAIM Probab. Stat. 5 (2001) 33-49. | Numdam | MR | Zbl

[4] Y. Baraud, C. Giraud and S. Huet. Gaussian model selection with an unknown variance. Ann. Statist. 37 (2009) 630-672. | MR | Zbl

[5] A. R. Barron, L. Birgé and P. Massart. Risk bounds for model selection via penalization. Probab. Theory Related Fields 113 (1999) 301-413. | MR | Zbl

[6] A. R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inform. Theory 39 (1993) 930-945. | MR | Zbl

[7] A. R. Barron. Approximation and estimation bounds for artificial neural networks. Machine Learning 14 (1994) 115-133. | Zbl

[8] L. Birgé. Model selection via testing: An alternative to (penalized) maximum likelihood estimators. Ann. Inst. Henri Poincaré Probab. Stat. 42 (2006) 273-325. | Numdam | MR

[9] L. Birgé. Model selection for Poisson processes. In Asymptotics: Particles, Processes and Inverse Problems, Festschrift for Piet Groeneboom 32-64. E. Cator, G. Jongbloed, C. Kraaikamp, R. Lopuhaä and J. Wellner (Eds). IMS Lecture Notes - Monograph Series 55. Inst. Math. Statist., Beachwood, OH, 2007. | MR | Zbl

[10] L. Birgé. Model selection for density estimation with $𝕃_{2}$ -loss. Probab. Theory Related Fields. To appear. Available at http://arxiv.org/abs/1102.2818. | Zbl

[11] L. Birgé and P. Massart. Gaussian model selection. J. Eur. Math. Soc. (JEMS) 3 (2001) 203-268. | MR | Zbl

[12] W. Dahmen, R. Devore and K. Scherer. Multidimensional spline approximation. SIAM J. Numer. Anal. 17 (1980) 380-402. | MR | Zbl

[13] R. Devore and G. Lorentz. Constructive Approximation. Springer, Berlin, 1993. | MR | Zbl

[14] J. Friedman and J. Tukey. A projection pursuit algorithm for exploratory data analysis. IEEE Trans. Comput. C-23 (1974) 881-890. | Zbl

[15] R. Hochmuth. Wavelet characterizations for anisotropic Besov spaces. Appl. Comput. Harmon. Anal. 12 (2002) 179-208. | MR | Zbl

[16] J. L. Horowitz and E. Mammen. Rate-optimal estimation for a general class of nonparametric regression models with unknown link functions. Ann. Statist. 35 (2007) 2589-2619. | MR | Zbl

[17] P. J. Huber. Projection pursuit (with discussion). Ann. Statist. 13 (1985) 435-525. | MR | Zbl

[18] A. B. Juditsky, O. V. Lepski and A. B. Tsybakov. Nonparametric estimation of composite functions. Ann. Statist. 37 (2009) 1360-1404. | MR | Zbl

[19] C. Maugis and B. Michel. A non asymptotic penalized criterion for Gaussian mixture model selection. ESAIM Probab. Stat. 15 (2011) 41-68. | Numdam | MR

[20] C. J. Stone. Optimal global rates of convergence for nonparametric regression. Ann. Statist. 10 (1982) 1040-1053. | MR | Zbl

Chen, Juntong Estimating a regression function in exponential families by model selection, Bernoulli, Volume 30 (2024) no. 2 | DOI:10.3150/23-bej1649
Chen, Juntong Robust nonparametric regression based on deep ReLU neural networks, Journal of Statistical Planning and Inference, Volume 233 (2024), p. 106182 | DOI:10.1016/j.jspi.2024.106182
Schmidt-Hieber, Johannes Nonparametric regression using deep neural networks with ReLU activation function, The Annals of Statistics, Volume 48 (2020) no. 4 | DOI:10.1214/19-aos1875
Sart, Mathieu Estimating the conditional density by histogram type estimators and model selection, ESAIM: Probability and Statistics, Volume 21 (2017), p. 34 | DOI:10.1051/ps/2016026
Akakpo, Nathalie Multivariate intensity estimation via hyperbolic wavelet selection, Journal of Multivariate Analysis, Volume 161 (2017), p. 32 | DOI:10.1016/j.jmva.2017.07.005
Lee, Young K.; Mammen, Enno; Nielsen, Jens P.; Park, Byeong U. Operational time and in-sample density forecasting, The Annals of Statistics, Volume 45 (2017) no. 3 | DOI:10.1214/16-aos1486
Rebelles, G. Structural adaptive deconvolution under $L_{p}$ -losses, Mathematical Methods of Statistics, Volume 25 (2016) no. 1, p. 26 | DOI:10.3103/s1066530716010026
Sart, Mathieu Model selection for Poisson processes with covariates, ESAIM: Probability and Statistics, Volume 19 (2015), p. 204 | DOI:10.1051/ps/2014022
Rebelles, Gilles $L_{p}$ adaptive estimation of an anisotropic density under independence hypothesis, Electronic Journal of Statistics, Volume 9 (2015) no. 1 | DOI:10.1214/15-ejs986
Lepski, Oleg Adaptive estimation over anisotropic functional classes via oracle approach, The Annals of Statistics, Volume 43 (2015) no. 3 | DOI:10.1214/14-aos1306
Baraud, Yannick; Birgé, Lucien Estimating composite functions by model selection, Annales de l'Institut Henri Poincaré, Probabilités et Statistiques, Volume 50 (2014) no. 1, pp. 285-314 | DOI:10.1214/12-aihp516
Sart, Mathieu Estimation of the transition density of a Markov chain, Annales de l'Institut Henri Poincaré, Probabilités et Statistiques, Volume 50 (2014) no. 3 | DOI:10.1214/13-aihp551
Birgé, Lucien Model selection for density estimation with L2-loss, arXiv (2008) | DOI:10.48550/arxiv.0808.1416 | arXiv:0808.1416

Cité par 13 documents. Sources : Crossref, NASA ADS