Sensitivity results in stochastic optimal control: A Lagrangian perspective

Backhoff, J.; Silva, F. J.

ESAIM: Control, Optimisation and Calculus of Variations, Tome 23 (2017) no. 1, pp. 39-70.

In this work we provide a first order sensitivity analysis of some parameterized stochastic optimal control problems. The parameters and their perturbations can be given by random processes and affect the state dynamics. We begin by proving a one-to-one correspondence between the adjoint states appearing in a weak form of the stochastic Pontryagin principle and the Lagrange multipliers associated to the state equation when the stochastic optimal control problem is seen as an abstract optimization problem on a suitable Hilbert space. In a first place, we use this result and classical arguments in convex analysis, to study the differentiability of the value function for convex problems submitted to linear perturbations of the dynamics. Then, for the linear quadratic and the mean variance problems, our analysis provides the stability of the optimizers and the $C^1$-differentiability of the value function, as well as explicit expressions for the derivatives, even when the data perturbation is not convex in the sense of [R.T. Rockafellar, Conjugate duality and optimization. Society for Industrial and Applied Mathematics, Philadelphia, Pa. (1974).

A. Bensoussan, Lectures on stochastic control. In Nonlinear filtering and stochastic control (Cortona, 1981). Vol. 972 of Lect. Notes Math. Springer, Berlin, New York (1982) 1–62. | MR

A. Bensoussan, Stochastic maximum principle for distributed parameter system. J. Franklin Inst. 315 (1983) 387–406. | DOI | MR | Zbl

J.-M Bismut, Conjugate convex functions in optimal stochastic control. J. Math. Anal. Appl. 44 (1973) 384–404. | DOI | MR | Zbl

J.-M. Bismut, Linear quadratic optimal stochastic control with random coefficients. SIAM J. Control Optim. 14 (1976) 419–444. | DOI | MR | Zbl

J.-M Bismut, Théorie probabiliste du contrôle des diffusions. Mem. Amer. Math. Soc. 4 (167):xiii+130 (1976). | MR | Zbl

J.F. Bonnans, Optimisation Continue. Dunod, Paris (2006).

J.F. Bonnans and A. Shapiro, Optimization problems with perturbations: a guided tour. SIAM Rev. 40 (1998) 228–264. | DOI | MR | Zbl

J.F. Bonnans and A. Shapiro, Perturbation analysis of optimization problems. Springer Ser. Oper. Research. Springer-Verlag, New York (2000). | MR | Zbl

J.F. Bonnans and F.J. Silva, First and second order necessary conditions for stochastic optimal control problems. Appl. Math. Optim. 65 (2012) 403–439. | DOI | MR | Zbl

A. Cadenillas and I. Karatzas, The stochastic maximum principle for linear convex optimal control with random coefficients. SIAM J. Control Optim. 33 (1995) 590–624. | DOI | MR | Zbl

S. Chen, X. Li, and X.Y. Zhou, Stochastic linear quadratic regulators with indefinite control weight costs. SIAM J. Control Optim. 36 (1998) 1685–1702 (electronic). | DOI | MR | Zbl

S. Chen and J. Yong, Stochastic linear quadratic optimal control problems. Appl. Math. Optim. 43 (2001) 21–45. | DOI | MR | Zbl

X. Cheng and J. Yan, A new look at the Lagrange method for continuous-time stochastic optimization. Sci. China Math. 55 (2012) 2247–2258. | DOI | MR | Zbl

D. Duffie and H. R. Richardson, Mean-variance hedging in continuous time. Ann. Appl. Probab. 1 (1991) 1–15. | DOI | MR | Zbl

U.G. Haussmann, A stochastic maximum principle for optimal control of diffusions. Pitman Research Notes Math. Series. Longman, Scientific & Technical (1986). | MR | Zbl

N. Ikeda and S. Watanabe, Stochastic differential equations and diffusion processes. Vol. 24 of North-Holland Mathematical Library. North-Holland Publishing Co., Amsterdam; Kodansha, Ltd., Tokyo, 2nd edition (1989). | MR | Zbl

I. Karatzas and S.E. Shreve, Brownian motion and stochastic calculus. Vol. 113 of Grad. Texts Math. 2nd edition. Springer-Verlag, New York (1991). | MR | Zbl

I. Karatzas and S.E. Shreve, Methods of mathematical finance. Vol. 39 of Appl. Math. Springer-Verlag, New York (1998). | MR | Zbl

P. Kosmol and M. Pavon, Solving optimal control problems by means of general Lagrange functionals. Automatica J. IFAC 37 (2001) 907–913. | DOI | MR | Zbl

H.J. Kushner, On the stochastic maximum principle: Fixed time of control. J. Math. Anal. Appl. 11 (1965) 78–92. | DOI | MR | Zbl

P.D. Loewen, Parameter sensitivity in stochastic optimal control. Stochastics 22 (1987) 1–40. | DOI | MR | Zbl

H. Maurer and J. Zowe, First and second order necessary and sufficient optimality conditions for infinite-dimensional programming problems. Math. Program. 16 (1979) 98–110,. | DOI | MR | Zbl

B. Øksendal N.C. Framstad and A. Sulem, Sufficient stochastic maximum principle for the optimal control of jump diffusions and applications to finance. J. Optim. Theory Appl. 121 (2004) 77–98. | DOI | MR | Zbl

S. Peng, A general stochastic maximum principle for optimal control problems. SIAM J. Control Optim. 28 (1990) 966–979. | DOI | MR | Zbl

P.E. Protter, Stochastic integration and differential equations. Vol. 21 of Stoch. Model. Appl. Probab. 2nd edition. Version 2.1, Corrected third printing. Springer-Verlag, Berlin (2005). | MR

D. Revuz and M. Yor, Continuous martingales and Brownian motion. Vol. 293 of Grundlehren der Math. Wissens. [Fundamental Principles of Mathematical Sciences], 3rd edition. Springer-Verlag, Berlin (1999). | MR | Zbl

S.M. Robinson, First order conditions for general nonlinear optimization. SIAM J. Appl. Math. 30 (1976) 597–607. | DOI | MR | Zbl

R. T. Rockafellar, Conjugate convex functions in optimal control and the calculus of variations. J. Math. Anal. Appl. 32 (1970) 174–222. | DOI | MR | Zbl

R.T. Rockafellar, Conjugate duality and optimization. Lectures given at the Johns Hopkins University, Baltimore, Md., June (1973). Conference Board of the Mathematical Sciences Regional Conference Series in Applied Mathematics, No. 16. Society for Industrial and Applied Mathematics, Philadelphia, Pa. (1974). | MR | Zbl

S. Tang, General linear quadratic optimal stochastic control problems with random coefficients: linear stochastic Hamilton systems and backward stochastic Riccati equations. SIAM J. Control Optim. 42 (2003) 53–75 (electronic). | DOI | MR | Zbl

J. Yong and X. Y. Zhou, Stochastic controls. Hamiltonian systems and HJB equations. Vol. 43 of Appl. Math. Springer-Verlag, New York (1999). | MR | Zbl

X.Y. Zhou, Maximum principle, dynamic programming, and their connection in deterministic control. J. Optim. Theory Appl. 65 (1990) 363–373. | DOI | MR | Zbl

X.Y. Zhou, A unified treatment of maximum principle and dynamic programming in stochastic controls. Stochastics Stochastics Rep. 36 (1991) 137–161. | DOI | MR | Zbl

X.Y. Zhou and D. Li, Continuous-time mean-variance portfolio selection: a stochastic LQ framework. Appl. Math. Optim. 42 (2000) 19–33. | DOI | MR | Zbl

J. Zowe and S. Kurcyusz, Regularity and stability for the mathematical programming problem in banach spaces. Appl. Math. Optim. 5 (1979) 49–62. | DOI | MR | Zbl