We consider an infinite horizon optimal control problem for a pure jump Markov process X, taking values in a complete and separable metric space I, with noise-free partial observation. The observation process is defined as Y$$ = h(X$$), t ≥ 0, where h is a given map defined on I. The observation is noise-free in the sense that the only source of randomness is the process X itself. The aim is to minimize a discounted cost functional. In the first part of the paper we write down an explicit filtering equation and characterize the filtering process as a Piecewise Deterministic Process. In the second part, after transforming the original control problem with partial observation into one with complete observation (the separated problem) using filtering equations, we prove the equivalence of the original and separated problems through an explicit formula linking their respective value functions. The value function of the separated problem is also characterized as the unique fixed point of a suitably defined contraction mapping.
Accepté le :
Première publication :
Publié le :
DOI : 10.1051/cocv/2019020
Mots-clés : Stochastic filtering, partial observation control problem, pure jump processes, piecewise-deterministic Markov processes, Markov decision processes
@article{COCV_2020__26_1_A25_0,
author = {Calvia, Alessandro},
title = {Stochastic filtering and optimal control of pure jump {Markov} processes with noise-free partial observation},
journal = {ESAIM: Control, Optimisation and Calculus of Variations},
publisher = {EDP-Sciences},
volume = {26},
year = {2020},
doi = {10.1051/cocv/2019020},
mrnumber = {4071313},
zbl = {1441.93309},
language = {en},
url = {http://www.numdam.org/articles/10.1051/cocv/2019020/}
}
TY - JOUR AU - Calvia, Alessandro TI - Stochastic filtering and optimal control of pure jump Markov processes with noise-free partial observation JO - ESAIM: Control, Optimisation and Calculus of Variations PY - 2020 VL - 26 PB - EDP-Sciences UR - http://www.numdam.org/articles/10.1051/cocv/2019020/ DO - 10.1051/cocv/2019020 LA - en ID - COCV_2020__26_1_A25_0 ER -
%0 Journal Article %A Calvia, Alessandro %T Stochastic filtering and optimal control of pure jump Markov processes with noise-free partial observation %J ESAIM: Control, Optimisation and Calculus of Variations %D 2020 %V 26 %I EDP-Sciences %U http://www.numdam.org/articles/10.1051/cocv/2019020/ %R 10.1051/cocv/2019020 %G en %F COCV_2020__26_1_A25_0
Calvia, Alessandro. Stochastic filtering and optimal control of pure jump Markov processes with noise-free partial observation. ESAIM: Control, Optimisation and Calculus of Variations, Tome 26 (2020), article no. 25. doi : 10.1051/cocv/2019020. http://www.numdam.org/articles/10.1051/cocv/2019020/
[1] , A dynamic programming algorithm for the optimal control of piecewise deterministic Markov processes. SIAM J. Control Opti. 40 (2001) 525–539. | DOI | MR | Zbl
[2] , and , Portfolio optimization for a large investor controlling market sentiment under partial information. SIAM J. Financ. Mat. 10 (2019) 512–546. | DOI | MR | Zbl
[3] , Applied Probability and Queues (Stochastic Modelling and Applied Probability). Vol. 51 of Applications of Mathematics, 2nd edn. Springer-Verlag, New York (2003). | MR | Zbl
[4] and , Fundamentals of Stochastic Filtering. Springer, New York (2009). | DOI | Zbl
[5] , Constrained BSDEs driven by a non quasi-left-continuous random measure and optimal control of PDMPs on bounded domains. Preprint (2017). | arXiv | MR
[6] , Optimal control of piecewise deterministic Markov processes: a BSDE representation of the value function. ESAIM: COCV 24 (2018) 311–354. | Numdam | MR | Zbl
[7] and , Constrained BSDEs representation of the value function in optimal control of pure jump Markov processes. Stoch. Process. Appl. 127 (2017) 1441–1474. | DOI | MR | Zbl
[8] , , and , Randomized filtering and Bellman equation in Wasserstein space for partial observation control problem. Stoch. Process. Appl. 129 (2019) 674–711. | DOI | MR | Zbl
[9] , and , BSDE representation and randomized dynamic programming principle for stochastic control problems of infinite-dimensional jump-diffusions. Preprint (2018). | arXiv | MR
[10] , , and , Backward SDEs for optimal control of partially observed path-dependent stochastic systems: a control randomization approach. Ann. Appl. Probab. 28 (2018) 1634–1678. | DOI | MR | Zbl
[11] , Solutions de viscosité des équations de Hamilton-Jacobi. Vol. 17 of Mathematiques & Applications. Springer-Verlag, Paris (1994). | MR | Zbl
[12] , and , On the optimal control of partially observed inventory systems. C. R. Math. Acad. Sci. Paris 341 (2005) 419–426. | DOI | MR | Zbl
[13] , and , Mean Field Games and Mean Field Type Control Theory. Springer Briefs in Mathematics. Springer, New York (2013). | DOI | MR | Zbl
[14] and , Stochastic Optimal Control: The Discrete Time Case. Vol. 139 of Mathematics in Science and Engineering. Academic Press, Inc., New York, London (1978). | MR | Zbl
[15] , Measure Theory, Vol. I, II. Springer-Verlag, Berlin (2007). | DOI | MR | Zbl
[16] , Point Processes and Queues. Springer Series in Statistics. Springer-Verlag, New York (1981). | MR | Zbl
[17] and , Linear filtering for time-varying systems using measurements containing colored noise. IEEE Trans. Automat. Contr. AC-10 (1965) 4–10. | DOI | MR
[18] and , An exact stochastic hybrid model of excitable membranes including spatio-temporal evolution. J. Math. Biol. 63 (2011) 1051–1093. | DOI | MR | Zbl
[19] , Optimal control of continuous-time Markov chains with noise-free observation. SIAM J. Control Optim. 56 (2018) 2000–2035. | DOI | MR | Zbl
[20] and , Filtering of a Markov jump process with counting observations. Appl. Math. Optim. 42 (2000) 1–18. | DOI | MR | Zbl
[21] and , Nonlinear filtering equation of a jump process with counting observations. Acta Appl. Math. 66 (2001) 139–154. | DOI | MR | Zbl
[22] and , Controlled partially observed jump processes: dynamics dependent on the observed history. In Vol 47 of Proceedings of the Third World Congress of Nonlinear Analysts, Part 4 Catania, 2000 (2001) 2449–2460. | MR | Zbl
[23] , and , Existence of optimal controls for partially observed jump processes. Acta Appl. Math. 74 (2002) 155–175. | DOI | MR | Zbl
[24] , , and , Optimal liquidation under partial information with price impact. Preprint (2019). | arXiv | MR
[25] and , Filtering of continuous-time Markov chains with noise-free observation and applications. Stochastics 85 (2013) 216–251. | DOI | MR | Zbl
[26] and , Continuous Average Control of Piecewise Deterministic Markov Processes. Springer Briefs in Mathematics. Springer, New York (2013). | MR | Zbl
[27] , and , Constrained and unconstrained optimal discounted control of piecewise deterministic Markov processes. SIAM J. Control Optim. 54 (2016) 1444–1474. | DOI | MR | Zbl
[28] , and , User’s guide to viscosity solutions of second order partial differential equations. Bull. Am. Math. Soc. (N.S.) 27 (1992) 1–67. | DOI | MR | Zbl
[29] , and , Nonlinear filtering with signal dependent observation noise. Electron. J. Probab. 14 (2009) 1863–1883. | DOI | MR | Zbl
[30] , Control of piecewise-deterministic processes via discrete-time dynamic programming, in Stochastic Differential Systems (Bad Honnef, 1985). Vol. 78 of Lecture Notes in Control and Information Sciences. Springer, Berlin (1986) 140–150. | DOI | MR | Zbl
[31] and , Piecewise-deterministic processes and viscosity solutions, in Stochastic Analysis, Control, Optimization and Applications. Systems Control Foundations and Applications. Birkhäuser Boston, Boston, MA (1999) 249–268. | MR | Zbl
[32] , Markov Models and Optimization. Vol. 49 of Monographs on Statistics and Applied Probability. Chapman and Hall, London (1993). | MR | Zbl
[33] , Optimal control of piecewise deterministic Markov processes, in Applied Stochastic Analysis (London, 1989). Vol. 5 of Stochastics Monographs, Gordon and Breach, New York (1991) 303–325. | MR | Zbl
[34] , and , Hidden Markov Models: Estimation and Control. Vol. 29 of Applications of Mathematics (New York). Springer-Verlag, New York (1995). | MR | Zbl
[35] , and , Stochastic Optimal Control in Infinite Dimension: Dynamic Programming and HJB Equations, With a Contribution by Marco Fuhrman and Gianmario Tessitore. Vol. 82 of Probability Theory and Stochastic Modelling. Springer, Cham (2017). | DOI | MR | Zbl
[36] and , Controlled Markov Processes and Viscosity Solutions. Vol. 25 of Stochastic Modelling and Applied Probability, 2nd edn. Springer, New York (2006). | MR | Zbl
[37] , and , Piecewise deterministic Markov control processes with feedback controls and unbounded costs. Acta Appl. Math. 82 (2004) 239–267. | DOI | MR | Zbl
[38] , Point Process Theory and Applications: Marked Point and Piecewise Deterministic Processes. Probability and Its Applications. Birkhäuser Boston, Inc., Boston, MA (2006). | MR | Zbl
[39] , Multivariate point processes: predictable projection, Radon-Nikodým derivatives, representation of martingales. Z. Wahrscheinlichkeitstheorie und Verw. Gebiete 31 (1974) 235–253. | DOI | MR | Zbl
[40] and , Nonlinear filtering with continuous time perfect observations and noninformative quadratic variation, in Proceeding of the 36th IEEE Conference on Decision and Control (1997) 1645–1650. | DOI
[41] and , Feynman-Kac representation for Hamilton-Jacobi-Bellman IPDE. Ann. Probab. 43 (2015) 1823–1865. | DOI | MR | Zbl
[42] and , Filtering for nonlinear systems driven by nonwhite noises: an approximation scheme. Stoch. Stoch. Rep. 44 (1993) 65–102. | DOI | MR | Zbl
[43] and , Marked Point Processes on the Real Line: The Dynamic Approach. Probability and Its Applications (New York). Springer-Verlag, New York (1995). | MR | Zbl
[44] Differential equations on closed subsets of a Banach space. Trans. Am. Math. Soc. 179 (1973) 399–414. | DOI | MR | Zbl
[45] , Markov Chains. Vol. 2 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge (1998). | MR | Zbl
[46] , and , Optimal control of infinite-dimensional piecewise deterministic Markov processes and application to the control of neuronal dynamics via Optogenetics. Netw. Heterog. Media 12 (2017) 417–459. | DOI | MR | Zbl
[47] and , Diffusions, Markov processes, and Martingales (Foundations). Vol. 1 of Wiley Series in Probability and Mathematical Statistics: Probability and Mathematical Statistics, 2nd edn. John Wiley & Sons, Ltd., Chichester (1994). | MR | Zbl
[48] and , Least-squares state estimation of systems with state-dependent observation noise. Automatica J. IFAC 21 (1985) 303–313. | DOI | MR | Zbl
[49] , Optimal control of piecewise deterministic Markov process. Stochastics 14 (1985) 165–207. | DOI | MR | Zbl
[50] , Optimal Control of Markovian Jump Processes with Different Information Structures. Ph.D. thesis, Universität Ulm (2008).
[51] , An Introduction to Stochastic Filtering Theory. Oxford University Press, New York (2008). | DOI | MR | Zbl
[52] , On reducing a jump controllable Markov model to a model with discrete time. Theory Probab. Appl. 25 (1980) 58–69. | DOI | Zbl
Cité par Sources :
This research was partially supported by three GNAMPA-INdAM projects in 2015, 2016 and 2017 and by MIUR-PRIN 2015 project Deterministic and stochastic evolution equations.
