Randomized dynamic programming principle and Feynman-Kac representation for optimal control of McKean-Vlasov dynamics

By Erhan Bayraktar, Andrea Cosso, Huyen Pham. 2016

100 citations recorded by the source corpus.

We analyze a stochastic optimal control problem, where the state process follows a McKean-Vlasov dynamics and the diffusion coefficient can be degenerate. We prove that its value function V admits a nonlinear Feynman-Kac representation in terms of a class of forward-backward stochastic differential equations, with an autonomous forward process. We exploit this probabilistic representation to rigorously prove the dynamic programming principle (DPP) for V. The Feynman-Kac representation we obtain has an important role beyond its intermediary role in obtaining our main result: in fact it would be useful in developing probabilistic numerical schemes for V. The DPP is important in obtaining a characterization of the value function as a solution of a non-linear partial differential equation (the so-called Hamilton-Jacobi-Belman equation), in this case on the Wasserstein space of measures. We should note that the usual way of solving these equations is through the Pontryagin maximum principle, which requires some convexity assumptions. There were attempts in using the dynamic programming approach before, but these works assumed a priori that the controls were of Markovian feedback type, which helps write the problem only in terms of the distribution of the state process (and the control problem becomes a deterministic problem). In this paper, we will consider open-loop controls and derive the dynamic programming principle in this most general case. In order to obtain the Feynman-Kac representation and the randomized dynamic programming principle, we implement the so-called randomization method, which consists in formulating a new McKean-Vlasov control problem, expressed in weak form taking the supremum over a family of equivalent probability measures. One of the main results of the paper is the proof that this latter control problem has the same value function V of the original control problem.

Subfields: Applied Mathematics, Finance

View Paper: https://www.semanticscholar.org/paper/a5f0de0135aee2c9e5cbe4f059c4357f0ea87eca

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Mathematics

In Finance

Citations in the corpus

Zero-sum stochastic differential games of generalized McKean–Vlasov type (2018, background): “Let us mention however the paper [4] which states a randomized DPP for the control case with open-loop controls (but without dependence on the control distribution).”
Mean-field optimal control as Gamma-limit of finite agent controls (2018, methodology): “The most popular methods are based on extending Pontryagin’s maximum principle [2, 6, 9, 14, 17] or deriving a dynamic programming principle, and with it a form of a Hamilton-Jacobi-Bellman equation on a space of probability measures [8, 47, 54].”
Bellman equation and viscosity solutions for mean-field stochastic control problem (2015, unknown): “erministic control problem as in (6.3), and instead one has to consider the pair (Xt,P Xt ) as state variable in order to state a dynamic programming principle. This will be investigated in detail in [4]. From Itˆo’s formula (3.14), the inﬁnitesimal version of the above DPP leads to the dynamic prog”
Viscosity solutions to parabolic master equations and McKean–Vlasov SDEs with closed-loop controls (2018, result): “−Bt) ∧(T−t)+Bt +T−t = EP0 h [|Bt|−[T−t]]+ 2i +T−t<EP0 h |Bt|2 i +T−t= t+T−t= T. This is a desired contradiction. Technically, the choice of A2 t would work, see e.g. Bayraktar, Cosso, & Pham [3]. The following results can be proved easily, in particular, the viscosity property in (iv) follow”
Master Bellman equation in the Wasserstein space: Uniqueness of viscosity solutions (2021, background): “Concerning second-order equations, papers [41, 42, 4, 17, 16] focus on the existence of viscosity solutions, proving that the value function solves in the viscosity sense the Master Bellman equation.”
McKean–Vlasov optimal control: The dynamic programming principle (2019, background): “Specialisations to linear–quadratic settings were also explored by Pham [65], Li, Sun, and Yong [57], Li, Sun, and Xiong [58], Huang, Li, and Yong [43], Yong [80], and Basei and Pham [8].”
Mean–field moral hazard for optimal energy demand response management (2019, background): “…in the literature onMcKean–Vlasov problems, see, for example„ Bensoussan, Frehse, and Yam (2013), Bensoussan, Frehse, and Yam (2015), Bensoussan, Frehse, and Yam (2017), Pham andWei (2018), Pham andWei (2017), Bayraktar, Cosso, and Pham (2018), or Djete et al. (2019) for the most recent results.”