Qualitative Robustness in Bayesian Inference

Owhadi, Houman; Scovel, Clint

ESAIM: Probability and Statistics, Tome 21 (2017), pp. 251-274.

The practical implementation of Bayesian inference requires numerical approximation when closed-form expressions are not available. What types of accuracy (convergence) of the numerical approximations guarantee robustness and what types do not? In particular, is the recursive application of Bayes’ rule robust when subsequent data or posteriors are approximated? When the prior is the push forward of a distribution by the map induced by the solution of a PDE, in which norm should that solution be approximated? Motivated by such questions, we investigate the sensitivity of the distribution of posterior distributions (i.e. of posterior distribution-valued random variables, randomized through the data) with respect to perturbations of the prior and data-generating distributions in the limit when the number of data points grows towards infinity.

C.D. Aliprantis and K.C. Border, Infinite Dimensional Analysis: A Hitchhiker’s Guide. Springer, Berlin, 3rd edition (2006). | MR | Zbl

E. Atkins, M. Morzfeld and A. J. Chorin, Implicit particle methods and their connection with variational data assimilation. Monthly Weather Rev. 141 (2013) 1786–1803. | DOI

A. Barron, M.J. Schervish and L. Wasserman. The consistency of posterior distributions in nonparametric problems. Ann. Statist. 27 (1999) 536–561. | DOI | MR | Zbl

S. Basu, S.R. Jammalamadaka and W. Liu. Stability and infinitesimal robustness of posterior distributions and posterior quantities. J. Stat. Plann. Inference 71 (1998) 151–162. | DOI | MR | Zbl

R.H. Berk, Limiting behavior of posterior distributions when the model is incorrect. Ann. Math. Statist. 37 (1966) 51–58; correction, ibid 37 (1966) 745–746. | DOI | MR | Zbl

B. Betrò, Numerical treatment of Bayesian robustness problems. Internat. J. Approx. Reason. 50 (2009) 279–288. | DOI | MR | Zbl

B. Betrò and A. Guglielmi. Numerical robust Bayesian analysis under generalized moment conditions. In Bayesian robustness (Rimini, 1995), volume 29 of IMS Lecture Notes Monogr. Ser. 3–20. With a discussion by Elías Moreno and a rejoinder by the authors. Inst. Math. Statist., Hayward, CA (1996). | MR | Zbl

B. Betrò and A. Guglielmi, Methods for global prior robustness under generalized moment conditions. In Robust Bayesian analysis, Vol. 152 of Lecture Notes in Statist. Springer, New York (2000) 273–293. | MR | Zbl

B. Betrò, F. Ruggeri and M. Mȩczarski, Robust Bayesian analysis under generalized moments conditions. J. Stat. Plann. Inference 41 (1994) 257–266. | DOI | MR | Zbl

P. Billingsley, Convergence of Probability Measures. Wiley, New York, 2nd edition (1999). | MR | Zbl

G. Boente, R. Fraiman and V.J. Yohai, Qualitative robustness for stochastic processes. Ann. Statist. 46 (1987) 1293–1312. | MR | Zbl

T. Bui-Thanh and O. Ghattas, An analysis of infinite dimensional Bayesian inverse shape acoustic scattering and its numerical approximation. SIAM/ASA J. Uncertain. Quantif. 2 (2014) 203–222. | DOI | MR | Zbl

I. Castillo and R. Nickl, Nonparametric Bernstein–von Mises theorems in Gaussian white noise. Ann. Statist. 41 (2013) 1999–2028. | DOI | MR | Zbl

I. Castillo and R. Nickl, On the Bernstein–von Mises phenomenon for nonparametric Bayes procedures. Ann. Statist. 42 (2014) 1941–1969. | DOI | MR | Zbl

A.J. Chorin, M. Morzfeld and X. Tu, Implicit particle filters for data assimilation. Commun. Appl. Math. Comput. Sci. 5 (2010) 221–240. | DOI | MR | Zbl

A. J. Chorin and X. Tu, Implicit sampling for particle filters. Proc. National Acad. Sci. 106 (2009) 17249–17254. | DOI

K. Csilléry, M.G.B. Blum, O.E. Gaggiotti and O. François, Approximate Bayesian computation (abc) in practice. Trends Ecol. Evol. 25 (2010) 410–418. | DOI

I. Csiszár, I-divergence geometry of probability distributions and minimization problems. Ann. Probab. (1975) 146–158. | MR | Zbl

A. Cuevas. Qualitative robustness in abstract inference. J. Statist. Plan. Inference 18 (1988) 277–289. | DOI | MR | Zbl

A. Cuevas, Comment on ‘Bounds on posterior expectations for density bounded class with constant bandwidth’ by Sivaganesan. J. Statist. Plan. Inference 40 (1994) 340–343. | MR | Zbl

A. Cuevas González, Una definición de robustez cualitativa en inferencia Bayesiana. Trabajos de Estadística y de Investigación Operativa 35 (1984) 170–186. | DOI | Zbl

M. Dashti and A.M. Stuart, Uncertainty quantification and weak approximation of an elliptic inverse problem. SIAM J. Numer. Anal. 49 (2011) 2524–2542. | DOI | MR | Zbl

C. Dellacherie and P.-A. Meyer, Probabilities and Potential. B. Vol. 72 of North-Holland Mathematics Studies. North-Holland Publishing Co., Amsterdam (1982). | MR | Zbl

A. Doucet, N. De Freitas and N. Gordon, An introduction to sequential Monte Carlo methods. In Sequential Monte Carlo Methods in Practice. Springer (2001) 3–14. | MR | Zbl

R.M. Dudley, Real Analysis and Probability. Vol. 74 of Cambridge Studies in Advanced Mathematics. Revised reprint of the 1989 original. Cambridge University Press, Cambridge (2002). | MR | Zbl

P. Dupuis and R.S. Ellis, A Weak Convergence Approach to the Theory of Large Deviations. Vol. 902. John Wiley and Sons (2011). | MR | Zbl

D. Freedman, On the Bernstein-von Mises theorem with infinite-dimensional parameters. Ann. Statist. 27 (1999) 1119–1140. | DOI | MR | Zbl

Y. Gal and Z. Ghahramani, Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. Inter. Conf. Machine Learning PLMR 48 (2016) 1050–1059.

A. Gelman, Inference and monitoring convergence. In Markov chain Monte Carlo in practice. Springer (1996) 131–143. | Zbl

S. Ghosal, J.K. Ghosh and R.V. Ramamoorthi, Consistency issues in Bayesian nonparametrics. Stat. Textbooks Monogr. 158 (1999) 639–668. | MR | Zbl

A.L. Gibbs, Convergence in the Wasserstein metric for Markov chain Monte Carlo algorithms with applications to image restoration. Stoch. Models 20 (2004) 473–492. | DOI | MR | Zbl

A.L. Gibbs and F.E. Su, On choosing and bounding probability metrics. Inter. Statist. Rev. 70 (2002) 419–435. | DOI | Zbl

I.J. Goodfellow, J. Shlens and C. Szegedy, Explaining and harnessing adversarial examples. Inter. Confer. Learning Representations. Preprint (2015). | arXiv

N.J. Gordon, D.J. Salmond and A.F.M. Smith, Novel approach to nonlinear/non-Gaussian Bayesian state estimation. Radar and Signal Processing. IEE Proc. F 140 (1993) 107–113. IET.

P. Gustafson and L. Wasserman, Local sensitivity diagnostics for Bayesian inference. Ann. Statist. 23 (1995) 2153–2167. | DOI | MR | Zbl

R. Hable and A. Christmann, On qualitative robustness of support vector machines. J. Multivariate Anal. 102 (2011) 993–1007. | DOI | MR | Zbl

R. Hable and A. Christmann, Robustness versus consistency in ill-posed classification and regression problems. In Classification and Data Mining. Springer (2013) 27–35. | MR

F.R. Hampel, A general qualitative definition of robustness. The Annals of Mathematical Statistics, pages 1887–1896, 1971. | MR | Zbl

P.J. Huber and E.M. Ronchetti, Robust Statistics. Wiley Series in Probability and Statistics. John Wiley and Sons Inc., Hoboken, NJ, 2nd edition (2009). | MR | Zbl

A.N. Kolmogorov and V.M. Tikhomirov, $\epsilon$-entropy and $\epsilon$-capacity of sets in function spaces. Trans. Amer. Math. Soc. 17 (1961) 277–364. | MR | Zbl

T.A. Le, A.G. Baydin, R. Zinkov and F. Wood, Using synthetic data to train neural networks is model-based reasoning. arXiv preprint arXiv:1703.00868 (2017).

N. Madras and D. Sezer, Quantitative bounds for Markov chain convergence: Wasserstein and total variation distances. Bernoulli 16 (2010) 882–908. | MR | Zbl

J.W. Miller and D.B. Dunson, Robust bayesian inference via coarsening. arXiv preprint arXiv:1506.06101 (2015). | MR

I. Mizera, Qualitative robustness and weak continuity: the extreme unction. Nonparametrics and Robustness in Modern Statistical Inference and Time Series Analysis: A Festschrift in honor of Professor Jana Jurecková 1 (2010) 169. | MR

M. Morzfeld and A.J. Chorin, Implicit particle filtering for models with partial noise, and an application to geomagnetic data assimilation. Nonlinear Processes in Geophysics 19 (2012). | DOI

M. Nasser, N.A. Hamzah and Md.A. Alam, Qualitative robustness in estimation. Pakistan J. Statist. Oper. Res. 8 (2012) 619–634. | DOI | MR

H. Owhadi and C. Scovel, Brittleness of Bayesian inference and new Selberg formulas. Commun. Math. Sci. 13 (2013) 75. | MR

H. Owhadi, C. Scovel and T.J. Sullivan, Brittleness of Bayesian inference under finite information in a continuous world. Electron. J. Statist. 9 (2015) 1–79. | DOI | MR | Zbl

H. Owhadi, C. Scovel and T.J. Sullivan, On the brittleness of Bayesian inference. SIAM Rev. 57 (2015) 566–582. | DOI | MR | Zbl

A.B. Patel, M.T. Nguyen and R. Baraniuk, A probabilistic framework for deep learning. In Advances in Neural Information Processing Systems (2016) 2558–2566.

S.T. Rachev, L.B. Klebakov, S.V. Stoyanov and F.J. Fabozzi, The Methods of Distances in the Theory of Probability and Statistics. Springer, New York (2013). | MR | Zbl

P. Rebeschini and R. Van Handel, Can local particle filters beat the curse of dimensionality? The Annals of Applied Probability 25 (2015) 2809–2866. | DOI | MR | Zbl

G.O. Roberts and J.S. Rosenthal, Markov-chain Monte Carlo: Some practical implications of theoretical results. Canadian J. Statist. 26 (1998) 5–20. | DOI | MR | Zbl

G.O. Roberts and J.S. Rosenthal, General state space Markov chains and MCMC algorithms. Probab. Surveys 1 (2004) 20–71. | DOI | MR | Zbl

M.J. Schervish, Theory of Statistics. Springer (1995). | MR | Zbl

L. Schwartz, On Bayes procedures. Z. Wahrscheinlichkeitstheorie und Verw. Gebiete 4 (1965) 10–26. | DOI | MR | Zbl

A. Smith, A. Doucet, N. de Freitas and N. Gordon, Sequential Monte Carlo Methods in Practice. Springer Science & Business Media (2013).

A.M. Stuart, Inverse problems: a Bayesian perspective. Acta Numer. 19 (2010) 451–559. | DOI | MR | Zbl

C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow and R. Fergus, Intriguing properties of neural networks. In International Conference on Learning Representations (2014). | arXiv

M. Uličnỳ, J. Lundström and S. Byttner, Robustness of deep convolutional neural networks for image recognition. In Inter. Symp. Intelligent Comput. Syst. Springer (2016) 16–30.

P.J. Van Leeuwen, Particle filtering in geophysical systems. Monthly Weather Rev. 137 (2009) 4089–4114. | DOI

A. Wald, Statistical Decision Functions. John Wiley and Sons Inc., New York, NY (1950). | MR | Zbl

L. Wasserman, Asymptotic properties of nonparametric Bayesian procedures. In Practical nonparametric and semiparametric Bayesian statistics. Springer (1998) 293–304. | MR | Zbl

A.D. Woodbury and T.J. Ulrych, A full-bayesian approach to the groundwater inverse problem for steady state flow. Water Resources Res. 36 (2000) 2081–2093. | DOI