On the maximum principle for relaxed control problems of nonlinear stochastic systems

Abstract

We consider optimal control problems for a system governed by a stochastic differential equation driven by a d-dimensional Brownian motion where both the drift and the diffusion coefficient are controlled. It is well known that without additional convexity conditions the strict control problem does not admit an optimal control. To overcome this difficulty, we consider the relaxed model, in which admissible controls are measure-valued processes and the relaxed state process is governed by a stochastic differential equation driven by a continuous orthogonal martingale measure. This relaxed model admits an optimal control that can be approximated by a sequence of strict controls by the so-called chattering lemma. We establish optimality necessary conditions, in terms of two adjoint processes, extending Peng’s maximum principle to relaxed control problems. We show that relaxing the drift and diffusion martingale parts directly as in deterministic control does not lead to a true relaxed model as the obtained controlled dynamics is not continuous in the control variable.

1 Introduction

Our main goal in this paper is to prove a stochastic maximum principle for relaxed controls in the case where both the drift and the diffusion coefficient are controlled.

It is well known that the two main approaches to handling optimal control problems are the dynamic programming by Bellman and the maximum principle by Pontryagin [36]. The maximum principle provides a set of necessary conditions for optimality that an optimal control must satisfy, as detailed in [36]. These conditions include a forward equation for the state process, a backward equation for the adjoint variable, and minimization of the Hamiltonian function.

Within stochastic control, two main approaches for a stochastic maximum principle emerge, based on the solution concept (weak/strong) and control type (open-loop/feedback). The first approach, for strong solutions with open-loop controls, was established by [23] using spike variations. In [17] the author employed martingale methods and the Girsanov theorem to derive a maximum principle for weak solutions with feedback controls.

In [10] the author addressed systems where the diffusion coefficient depends on the control variable, utilizing convex perturbations and the first-order adjoint variable. His result constitutes a weak maximum principle with the variational inequality applied to the Gâteaux derivative of the Hamiltonian. In [30], a global maximum principle was established for a nonconvex domain and controlled diffusion coefficient, involving the introduction of a second-order adjoint process. This extension was further developed for jump-diffusion processes in [33].

Pontryagin’s maximum principle has proven widespread application in solving problems related to mathematical finance and portfolio optimization, as shown in [31]. We recommend [36] for a comprehensive overview and detailed references on the subject.

The main motivation behind relaxed controls lies in their property to guarantee the existence of optimal solutions in this class. This concept originated with Young’s work [37] on generalized solutions in the calculus of variations, leading to the notion of Young measure. Subsequently, this framework was extended to deterministic control theory, giving rise to the concept of relaxed control. A key challenge in nonconvex control problems arises from the lack of closure of the set of strict controls under simple convergence of measurable functions. Relaxed controls elegantly address this challenge by replacing strict controls with random Dirac probability measures. This effectively transforms the set of strict controls into a compact subset of probability measures, ensuring closure under the topology of weak convergence. This “relaxation” of the convergence requirement enables us to formulate the optimal control problem as a continuous function optimization over a compact metric space, guaranteeing the existence of an optimal solution. The authors in [9, 16, 24] established the first existence results of relaxed controls for stochastic differential equations with uncontrolled diffusion coefficients. Subsequently, more complex systems with controlled diffusion coefficients were tackled by [14, 18, 19]. Using Krylov’s method of Markovian selection, they proved that the optimal relaxed control can be expressed in a feedback form. Furthermore, in [22] the authors used an abstract approach based on the concept of occupation measure to prove the existence of optimal relaxed controls.

1.1 The relaxed stochastic maximum principle and contributions of the paper

Optimality necessary conditions for stochastic systems in the form of Pontryagin’s maximum principle have been developed for relaxed controls in [7, 8, 28] in the case of continuous diffusions. These results have been extended to mean-field systems in [3–5]. See also [2, 12, 32] for versions of the relaxed stochastic maximum principle including doubly forward-backward stochastic differential equations and stochastic equations driven by G-Brownian motion.

Our main goal in this paper is to prove a stochastic maximum principle for relaxed controls in the case where both the drift and the diffusion coefficient are controlled. We show that the natural pathwise representation of the relaxed state process satisfies a stochastic differential equation driven by an orthogonal continuous martingale measure [15].

Note that another type of relaxation has been considered in the literature [1, 6, 35], where the authors replace the drift and the diffusion coefficients in the controlled stochastic equation by their integrals with respect to the relaxed control as in the deterministic control problems. They obtain a linear convex relaxed control problem. We prove that the main drawback of this type of relaxation is that the dynamics obtained is not continuous with respect to the control variable by providing a counterexample. As a byproduct, the relaxed and strict control problems have different value functions and the control problem obtained cannot be considered as a true relaxation. This is the first main contribution of the present paper.

Our second main result is to derive necessary conditions for optimality satisfied by an optimal relaxed control in the form of a Peng stochastic maximum principle. This is achieved through first- and second-order adjoint processes. By using the so-called Chattering lemma, the optimal relaxed control is approximated by a sequence of nearly optimal strict controls. Under pathwise uniqueness of the stochastic equation associated with the relaxed control, we prove a strong approximation result for the controlled processes. Ekeland’s variational principle then allows us to derive necessary conditions for near-optimality satisfied by the sequence of strict controls. The final step involves proving the convergence of the corresponding adjoint processes and Hamiltonian functions, completing the proof.

Our work extends the existing maximum principles in several ways. It generalizes Peng’s principle [30] to relaxed controls and [28] to include controlled diffusion coefficient. Furthermore, assuming that a strict optimal control exists, we recover Peng’s original principle [30]. The key advantage of our result is that it applies to a natural class of controls, which is the closure of the class of strict controls, and for which the existence of an optimal solution is guaranteed. Another advantage of our method is that it is based on an approximation procedure, which could be helpful to solve numerically problems arising in practical situations. Our method relies on an approximation scheme, making it a valuable tool for addressing various numerical problems encountered in real-world control applications.

The rest of the paper is organized as follows. In the second section, we formulate the control problem and introduce the assumptions of the model. The third section is devoted to the relaxed model. In the last section, we prove rigorously the second-order maximum principle for the relaxed control problem, representing the main contribution of this paper.

2 Formulation of the problem and notations

We consider in this paper stochastic control problems of the following type.

Let $(\Omega ,\mathcal{F},(\mathcal{F}_{t})_{t\geq 0},P)$ be a probability space equipped with a complete filtration $(\mathcal{F}_{t})_{t\geq 0}$ satisfying the usual conditions. Let $( B_{t} )$ be a standard d-dimensional Brownian motion.

Consider a compact set $\mathbb{A}$ in $\mathbb{R}^{k}$, and let $\mathcal{U}_{\mathrm{ad}}$ be the class of strict controls, which are measurable, $\mathcal{F}_{t}$-adapted processes $u: [ 0,T ] \times \Omega \longrightarrow \mathbb{A}$. For any u $\in \mathcal{U}_{\mathrm{ad}}$, we consider the control problem where the controlled process is a solution of the following stochastic differential equation (SDE):

$$ \textstyle\begin{cases} dX_{t}=b(t,X_{t},u_{t})\,dt+\sigma (t,X_{t},u_{t})\,dB_{{t}} \\ X_{0}=x. \end{cases} $$

(2.1)

We assume that

$$\begin{aligned}& b : [ 0;T ] \times \mathbb{R} ^{n}\times \mathbb{A} \longrightarrow \mathbb{R} ^{n} \\& \sigma : [ 0;T ] \times \mathbb{R} ^{n}\times \mathbb{A} \longrightarrow \mathcal{M}_{n\times d}(\mathbb{R} ) \end{aligned}$$

are bounded and Borel measurable functions.

The expected cost corresponding to a strict control u is given by

$$ J(u)=E \biggl[ g(X_{T})+{ \int _{0}^{T}} h(t,X_{t},u_{t})\,dt \biggr] , $$

(2.2)

where

$$\begin{aligned} &g :\mathbb{R} ^{n}\longrightarrow \mathbb{R}\\ &h : [ 0,T ] \times \mathbb{R} ^{n}\times \mathbb{A} \longrightarrow \mathbb{R} \end{aligned}$$

are Borel measurable functions.

The solution X of the above SDE is called the response of the control $u\in \mathcal{U}_{\mathrm{ad}}$. The objective of the strict control problem is to minimize the cost functional $J(\cdot)$ over the set $\mathcal{U}_{\mathrm{ad}}$, subject to equation (2.1). A control that solves this problem is called optimal. A strict control $u^{\ast}\in \mathcal{U}_{\mathrm{ad}}$ is called optimal if it achieves the infimum of $J(u)$ over $\mathcal{U}_{\mathrm{ad}}$.

Notations

Throughout this paper, we will use the following notations.

$x\cdot y$: the inner product of the vectors x and y.

$\vert x \vert = \vert x_{1} \vert + \vert x_{2} \vert +\cdots+ \vert x_{n} \vert$ for a n-dimensional vector $x=(x_{1},x_{2},\ldots,x_{n})$.

$A^{\ast}$: the transpose of a matrix A.

$f_{x}$: the gradient of the function f with respect to x.

$f_{xx}$: the Hessian of a scalar function f.

$\mathcal{M}_{n\times d}(\mathbb{R} )$: the space of $n\times d$ matrices.

$\mathbb{A}$: a compact subset of $\mathbb{R}^{k}$ called the action space.

$\mathcal{U}_{\mathrm{ad}}$: the space of strict controls.

$\mathcal{P}( [ 0,T ] \times \mathbb{A})$: the space of probability measure on the compact set $[ 0,T ] \times \mathbb{A}$.

$\mathbb{V}$: the subset of $\mathcal{P}( [ 0,T ] \times \mathbb{A})$ consisting of probability measures whose projection on $[ 0,T ]$ is the Lebesgue measure.

$\mathcal{R}$: the space of relaxed controls.

$C_{b}^{2}(\mathbb{R}^{d};\mathbb{R})$: the space of bounded continuous functions having bounded continuous first- and second-order derivatives.

$\mathbb{D}( [ 0,T ] ,\mathbb{R}^{n})$: is the Skorokhod space of functions that are continuous from the right and have limits from the left.

Assumptions

Let us assume the following conditions on the coefficients.

$(\mathbf{A}_{1})$ The maps b, σ, h, g, and f are continuous and bounded.

$(\mathbf{A}_{2})$ b, σ, h, g admit derivatives up to the second order with respect to x, which are bounded and continuous in $( x,a )$.

Under the above hypothesis, (2.1) has a unique strong solution and the cost functional (2.2) is well defined from $\mathcal{U}_{\mathrm{ad}}$ into $\mathbb{R}$.

Note that for questions of existence of optimal controls, the probability space, Brownian motion may change with the control u. Indeed, the existence of optimal controls uses heavily the concept of weak solution of stochastic differential equations. It is worth noting that for weak solutions of the stochastic differential equations, the probability space and Brownian motion are parts of the solution. Another way to deal with weak solutions is to use martingale problems [21].

The infinitesimal generator L, associated with our controlled SDE, is the second-order differential operator acting on functions f in $C_{b}^{2}(\mathbb{R}^{n};\mathbb{R})$, defined by

$$ Lf(t,x,a)= \biggl( {\sum} _{i,j}a_{ij} \frac {\partial ^{2}f}{\partial x_{i}x_{j}}+ {\sum} _{i}b_{i} \frac {\partial f}{\partial x_{i}} \biggr) (t,x,a), $$

(2.3)

where $a_{ij}(t,x,u)$ denotes the generic term of the symmetric matrix $\sigma \sigma ^{\ast}(t,x,u)$ [21].

As it is well known, weak solutions for Itô SDEs are equivalent to the existence of solutions of the corresponding martingale problems [21]. The approach by martingale problems simplifies limit analysis and avoids the relaxation complications associated with the stochastic integral part [14]. Let us define a strict control using martingale problems.

Definition 2.1

A strict control is a term $\alpha =(\Omega ,\mathcal{F},\mathcal{F}_{t},P,u_{t},X_{t})$ such that

(1) $(\Omega ,\mathcal{F},\mathcal{F}_{t},P)$ is a probability space equipped with a filtration $(\mathcal{F}_{t})_{t\geq 0}$ satisfying the usual conditions.

(2) $( u_{t} )$ is an $\mathbb{A}$-valued process, progressively measurable with respect to $(\mathcal{F}_{t})$.

(3) $(X_{t})$ is $\mathbb{R}^{n}$-valued $\mathcal{F}_{t}-$ adapted, with continuous paths, such that

$$ f(X_{t})-f(x)- \int _{0}^{t}Lf(s,X_{s},u_{s})\,ds \quad \text{is a }P\text{-martingale}. $$

Remark 2.2

1) Condition 3) in the above definition is equivalent to saying that SDE (2.1) has a weak solution.

2) Under assumptions A$_{\mathbf{1}}$ and A$_{ \mathbf{2}}$ the controlled equation (2.1) has a unique strong solution for every fixed probability space and Brownian motion. So we fix the probability reference, and a strict control $( u_{t} )$ will be just an $\mathbb{A}$-valued process progressively measurable with respect to $(\mathcal{F}_{t})$. There is no need to specify the probability space.

3 The relaxed control problem

3.1 A typical example

As we are going to see in a simple example, most control problems have no optimal solutions within the space of strict controls [14]. Let us consider the following well-known example from deterministic control [11].

Minimize

$$ J(u)= \int _{0}^{1} \bigl( X(t) \bigr) ^{2}\,dt $$

(3.1)

over the set $\mathbb{U}$ of measurable functions $u:[0,1]\rightarrow \{-1,1\}$, where $X(t)$ is the solution of

$$ \textstyle\begin{cases} dX(t)=u(t)\,dt \\ X(0)=0. \end{cases} $$

(3.2)

We have $\inf_{u\in \mathbb{U}}J(u)=0$.

Indeed let us consider the sequence of Rademacher functions:

$$ u_{n}(t)=(-1)^{k}\quad \text{if } \frac{k}{n}\leq t\leq \frac{(k+1)}{n},0 \leq k\leq n-1. $$

It is not difficult to show that $|X^{u_{n}}(t)|\leq 1/n$ and $|J(u_{n})|\leq 1/n^{2}$, which implies that $\inf_{u\in \mathbb{U}}J(u)=0$. There is, however, no control û such that $J(\widehat{u})=0$ because this would imply that for every t, $X^{\widehat{u}}(t)=0$; and as a consequence we obtain $\widehat{u}_{t}=0$, which is impossible. This limit, if it exists, would be the natural candidate for optimality.

The classical way to overcome this difficulty is to introduce relaxed controls, which are measure-valued functions that describe the introduction of a stochastic parameter. Let $dt\delta _{u(t)}(da)$ be the product measure on $[0,1]\times \{-1,1\}$ such that its projection on $[0,1]$ is the Lebesgue measure and is defined as follows:

$$\iint _{[0,1]\times \{-1,1\}} f(t,a)\,dt\delta _{u(t)}(da)=f(t,u_{t}). $$

$\delta _{u(t)}(da)$ denotes the Dirac measure concentrated at the point $u(t)$.

The following lemma is known in deterministic control. We give its proof for the sake of completeness.

Lemma 3.1

Let $dt\delta _{u_{{n}}(t)}(da)$ be the relaxed control associated with the Rademacher function $u_{n}(t)$, then the sequence $( dt\delta _{u_{{n}}(t)}(da) )$ converges weakly to $dt\frac{1}{2}(\delta _{-1}+\delta _{1})(da)$.

Proof

It is sufficient to show that for every bounded continuous function $f:[0,1]\times \{-1,1\}\longrightarrow \mathbb{R}$

$$\begin{aligned}& \iint _{[0,1]\times \{-1,1\}} f(t,a)\mu _{n}(dt,du)\quad \text{converges to} \iint _{[0,1]\times \{-1,1\}} f(t,a)\mu (dt,du)= \frac{1}{2} \\& \quad \biggl( { \int _{[0,1]}} f(t,-1)\,dt+{ \int _{[0,1]}} f(t,1)\,dt \biggr) , \end{aligned}$$

as $n\longrightarrow +\infty$.

Assume $n=2m$.

$$\begin{aligned} \iint _{[0,1]\times \{-1,1\}} f(t,a)\mu _{n}(dt,du)&= \sum _{k=0}^{n-1} { \int _{k/n}^{ ( k+1 ) /n}} f\bigl(t, ( -1 ) ^{k}\bigr)\,dt\\ &= \sum_{k=0}^{m-1} \int _{2j/2m}^{ ( 2j+1 ) /2m} f(t,1)\,dt+ \sum _{k=0}^{m-1} \int _{ ( 2j+1 ) /2m}^{ ( 2j+2 ) /2m} f(t,-1)\,dt. \end{aligned}$$

$f(t,-1)$ and $f(t,1)$ are continuous on $[0,1]$, which is bounded and closed, then they are uniformly continuous. Then, for some $\varepsilon >0$, there exists $N\in \mathbb{N}^{\ast}$ such that for every $m\geq N$ such that $\vert t-s \vert <\frac{1}{m}$ we have $\vert f(t,a)-f(s,a) \vert <\varepsilon$ for $a=1$ or $a=-1$.

This implies in particular that

$$\begin{aligned}& \biggl\vert { \int _{2j/2m}^{ ( 2j+1 ) /2m}} f(t,a)\,dt-{ \int _{ ( 2j+1 ) /2m}^{ ( 2j+2 ) /2m}} f(t,a)\,dt \biggr\vert < \frac {\varepsilon}{2m}\quad \text{for } j=0,1,\ldots,m-1, \end{aligned}$$

and therefore

$$\begin{aligned}& \Biggl\vert {\sum_{j=0}^{m-1}} { \int _{2j/2m}^{ ( 2j+1 ) /2m}} f(t,a)\,dt-{\sum _{j=0}^{m-1}} { \int _{ ( 2j+1 ) /2m}^{ ( 2j+2 ) /2m}} f(t,a)\,dt \Biggr\vert < \frac {\varepsilon}{2}. \end{aligned}$$

But we know that

$$\begin{aligned}& \sum_{j=0}^{m-1} { \int _{2j/2m}^{ ( 2j+1 ) /2m}} f(t,a)\,dt+\sum _{j=0}^{m-1} \int _{ ( 2j+1 ) /2m}^{ ( 2j+2 ) /2m} f(t,a)\,dt= \int _{[0,1]} f(t,a)\,dt. \end{aligned}$$

Therefore

$$\begin{aligned}& \lim_{m\rightarrow +\infty}{\sum_{j=0}^{m-1}} { \int _{2j/2m}^{ ( 2j+1 ) /2m}} f(t,a)\,dt= \lim _{m\rightarrow +\infty}{\sum_{j=0}^{m-1}} { \int _{ ( 2j+1 ) /2m}^{ ( 2j+2 ) /2m}} f(t,a)\,dt=1/2{ \int _{[0,1]}} f(t,a)\,dt,\\& \quad a=1\text{ or }-1, \end{aligned}$$

and

$$\begin{aligned} \lim_{n\rightarrow +\infty}{\sum_{k=0}^{n-1}} { \int _{k/n}^{ ( k+1 ) /n}} f\bigl(t, ( -1 ) ^{k}\bigr)\,dt&=\frac{1}{2} \biggl( { \int _{[0,1]}} f(t,1)\,dt+{ \int _{[0,1]}} f(t,-1)\,dt \biggr) \\ & ={ \int _{0}^{1}} { \int _{\{-1,1 \}}} f(t,a)\frac{1}{2} ( \delta _{-1}+\delta _{1} ) (da)\,dt, \end{aligned}$$

which achieves the proof.

The case where n is odd can be proved by using the same arguments. □

Remark 3.2

The sequence of Rademacher functions is a typical example of a minimizing sequence with no limit in the set of strict controls. However, its weak limit is $dt(1/2)(\delta _{-1}+\delta _{1})(da)$.

Now we can define the relaxed control as any probability measure on $[ 0,1 ] \times \{ -1,1 \}$ defined by $\mu =dt.\mu _{t}(da)$, and the relaxed dynamics will be

$$X^{\mu}(t)={ \int _{0}^{t}} { \int _{ \{ -1,1 \} }} a.\mu _{s}(da)\,ds. $$

The corresponding relaxed cost functional is given by

$$\mathcal{J}(\mu )= \int _{0}^{1} \bigl( X^{\mu}(t) \bigr) ^{2}\,dt. $$

Let us point out that in the case where the relaxed control μ is associated with a strict control u, in other words $\mu =dt.\delta _{u(t)}(da)$, then $\mathcal{J}(\mu )=J(u)$.

It is clear that if $\mu ^{\ast}=dt(1/2)(\delta _{-1}+\delta _{1})(da)$, $X^{\mu}(t)={\int _{0}^{t}} {\int _{ \mathbb{A}}} a.(1/2)(\delta _{-1}+\delta _{1})(da)\,ds=0$, therefore $\mathcal{J}(\mu ^{\ast })=0$. This means that $dt(1/2)(\delta _{-1}+\delta _{1})(da)$ is an optimal control in the space of relaxed controls.

3.2 The set of relaxed controls

The idea of relaxed control is to replace the $\mathbb{A}$-valued process $u_{t}$ with a $\mathcal{P(}\mathbb{A})$-valued process $\mu _{t}$, where $\mathcal{P(}\mathbb{A})$ is the space of probability measures equipped with the topology of weak convergence. Then μ may be identified as a random product measure on $[0,T]\times \mathbb{A}$, whose projection on $[0,T]$ coincides with the Lebesgue measure.

Let $\mathbb{V}$ be the set of product measures on $[0,T]\times \mathbb{A}$ whose projection on $[0,T]$ coincides with the Lebesgue measure dt. It is clear that every μ in $\mathbb{V}$ may be disintegrated as $\mu =dt.\mu _{t}(da)$, where $\mu _{t}(da)$ is a transition probability [14].

$\mathbb{V}$ as a closed subspace of the compact space of probability measures $\mathcal{P}([0,T]\times \mathbb{A})$ is compact for the topology of weak convergence. In fact it can be proved that it is compact also for the topology of stable convergence, where test functions are measurable, bounded functions $f(t,a)$ continuous in a. See [14] for further details.

Definition 3.3

A relaxed control on the filtered probability space $(\Omega ,\mathcal{F},(\mathcal{F}_{t})_{t\geq 0},P)$ is a random variable $\mu =dt.\mu _{t}(da)$ with values in $\mathbb{V}$ such that $\mu _{t}(da)$ is progressively measurable with respect to $(\mathcal{F}_{t}\mathcal{)}_{t\geq 0}$ and such that for each t, $1_{(0;t]}$ μ is $\mathcal{F}_{t}$-measurable.

The problem now is to define rigourously the dynamics associated with a relaxed control. More precisely, since the diffusion term is controlled, one has to define the concept of martingale measure.

Let us denote by $\mathcal{R}$ the collection of all relaxed controls.

3.3 The relaxed dynamics

When dealing with existence results, it is important to point out that the probability space and Brownian motion are parts of the relaxed control. The following definition gives a precise meaning of the notion of control.

Definition 3.4

A relaxed control is a term $\alpha =(\Omega ,\mathcal{F},\mathcal{F}_{t},P,\mu ,X_{t})$ such that

(1) $(\Omega ,\mathcal{F},\mathcal{F}_{t},P)$ is a probability space equipped with a filtration $(\mathcal{F}_{t})_{t\geq 0}$ satisfying the

usual conditions.

(2) μ is a $\mathbb{V}$-valued process, $\mu (\omega ,dt,du)=dt.\mu (\omega ,t,du)$, and $\mu (\omega ,t,du)$ is progressively measurable with respect to $(\mathcal{F}_{t})$ and such that for each t, $1_{(0,t]}.\mu$ is $\mathcal{F}_{t}-$ adapted.

(3) $(X_{t})$ is $\mathbb{R}^{n}$-valued $\mathcal{F}_{t}$-adapted, with continuous paths, such that

$$ f(X_{t})-f(x)- \int _{0}^{t} \int _{\mathbb{A}}Lf(s,X_{s},a)\mu (s,da)\,ds \quad \text{is a }P\text{-martingale}. $$

(3.3)

Let us define the corresponding relaxed cost functional by

$$ \mathcal{J}(\mu )=E \biggl[ g(X_{T})+{ \int _{0}^{T}} \int _{\mathbb{A}}h(t,X_{t},a) \mu (t,da)\,dt \biggr] . $$

(3.4)

In case the relaxed control is defined by $dt\delta _{u_{t}}(da)$, we recover the cost functional corresponding to the strict control u. More precisely, $\mathcal{J}(dt\delta _{u_{t}}(da))=J(u)$.

It is proved in [14] that the relaxed control problem admits an optimal solution.

Theorem 3.5

Under assumption (A$_{\mathbf{1}}$), the relaxed optimal control problem defined by the martingale problem (3.3) and the relaxed cost functional (3.4) admits an optimal solution.

In what follows we give a pathwise representation of the solution of the relaxed martingale problem in terms of an Itô stochastic differential equation driven by an orthogonal martingale measure. Martingale measures were introduced by Walsh [34], see also [15, 25] for more details.

Definition 3.6

Let $(\Omega ,\mathcal{F},\mathcal{F}_{t},P)$ be a filtered probability space and $M(t,B)$ be a random process, where $B\in \mathcal{B} ( \mathbb{A} )$ the Borel σ-field of $\mathbb{A}$. M is an $( \mathcal{F}_{t},P)$-martingale measure if:

1)For every $B\in \mathcal{B} ( \mathbb{A} ) , ( M(t,B) ) _{t \geq 0}$ is a square integrable martingale, $M(0,B)=0$.

2)For every $t>0$, $M(t,.)$ is a σ-finite $L^{2}$-valued measure.

It is called continuous if for each $B\in \mathcal{B} ( \mathbb{A} )$, $M(t,B)$ is continuous and orthogonal if $M(t,B).M(t,C)$ is a martingale whenever $B\cap C=\varnothing$.

Remark 3.7

When the martingale measure M is orthogonal, it is proved in [34] the existence of a random positive σ-finite measure $\mu ( dt,da ) \$on $[ 0,T ] \times \mathbb{A}$ such that $\langle M(.,B),M(.,B) \rangle _{t}=\mu ( [ 0,t ] \times B )$ for all $t>0$, and $B\in \mathcal{B} ( \mathbb{A} )$. $\mu ( dt,da )$ is called the covariance measure of M.

Theorem 3.8

1) Let P be a solution of the martingale problem (3.3). Then P is the law of a d-dimensional adapted and continuous process X defined on an extension of the space $( \Omega ,\mathcal{F}, \mathcal{F}_{t},P )$ and which is a solution of the following SDE starting at x:

$$ \textstyle\begin{cases} dX_{t}=\int _{\mathbb{A}}b(t,X_{t},a)\,\mu _{t}(da)\,dt+\int _{\mathbb{A}}\sigma (t,X_{t},a)\,M(da,dt), \\ \mathit{X}_{0}=x, \end{cases} $$

(3.5)

where $M=(M^{k})_{k=1}^{d}$ is a family of d-strongly orthogonal continuous martingale measures, each of them having intensity $dt\mu _{t}(da)$.

2) Under assumptions ($\mathbf{A}_{1}$) and ($\mathbf{A}_{2}$), SDE (3.5) has a unique strong solution.

Proof

Let us give an outline of the proof.

1) Suppose that X is a solution of SDE (3.5) on some probability space $( \Omega , \mathcal{F},\mathcal{F}_{t},P )$, and let $f\in C_{b}^{2}(\mathbb{R}^{n},\mathbb{R})$. An application of Itô’s formula gives

$$\begin{aligned} f(X_{t})= {}& f(X_{0})+ \int _{0}^{t} \int _{\mathbb{A}}f_{x}(X_{s})b(s,X_{s},a) \mu _{s}(da)\,ds+ \int _{0}^{t} \int _{ \mathbb{A}}f_{x}(X_{s})\sigma (s,X_{s},a)M(ds,da) \\ &{} +\frac {1}{2} \int _{0}^{t} \int _{\mathbb{A}}f_{xx}(X_{s}) \sigma \sigma ^{\ast}(s,X_{s},a)\mu _{s}(da)\,ds. \end{aligned}$$

It is clear that $f(X_{t})-f(X_{0})-\int _{0}^{t}\int _{\mathbb{A}}Lf(s,X_{s},a).\mu (s,da)\,ds=\int _{0}^{t}\int _{ \mathbb{A}}f_{x}(X_{s})\sigma (s,X_{s},a)M(ds,da)$, which is a martingale.

Conversely suppose that P is a solution of the relaxed martingale problem (3.3). This implies that

$$\begin{aligned}& f(X_{t})-f(x)- \int _{0}^{t} \int _{\mathbb{A}}Lf(s,X_{s},a).\mu (s,da)\,ds \\& \quad \text{is a } ( P,\mathcal{F}_{t} ) \text{-martingale for any } f \in C_{b}^{2}\bigl(\mathbb{R}^{d},\mathbb{R} \bigr). \end{aligned}$$

Choose $f(x)=x_{i}$ the ith coordinate of x, where $x=(x_{1},x_{2},\ldots,x_{n})\in B_{R}$ the ball of center 0 and radius R in $\mathbb{R}^{d}$, $B_{R}= \{ x\in \mathbb{R}^{d}/ \vert x \vert < R \}$. Define the first exist time of the process $X_{t}$ from the ball $B_{R}$, $\tau _{R}=\inf \{ t:X_{t}\notin B_{R} \}$.

f being $C_{b}^{2}$, it follows that $\Gamma _{i}^{R}=X_{i}(t\wedge \tau _{R})-X_{i}(0)-\int _{0}^{t \wedge \tau _{R}}\int _{\mathbb{A}}b_{i}(s,X_{s},a)\mu _{s}(da)\,ds$ is a $( P,\mathcal{F}_{t} )$ continuous square integrable $( P,\mathcal{F}_{t} )$-martingale. Therefore $\Gamma _{i}(t)=X_{i}(t)-X_{i}(0)-\int _{0}^{t}\int _{ \mathbb{A}}b_{i}(s,X_{s},a)\mu _{s}(da)\,ds$ is a $( P,\mathcal{F}_{t} )$ continuous $( P,\mathcal{F}_{t} ) -$ local martingale for any $i=1,2,\ldots,n$.

Now, choosing $f\in C_{b}^{2}(\mathbb{R}^{d})$ such that $f(x)=x_{i}x_{j}$ for $x=(x_{1},x_{2},\ldots,x_{n})\in B_{R}$, we see similarly that $\langle \Gamma _{i},\Gamma _{j} \rangle (t)=\int _{0}^{t}\int _{\mathbb{A}}a_{ij}(s,X_{s},a)\mu _{s}(da)\,ds$ with $( a_{ij} )$ is the symmetric matrix $\sigma \sigma ^{\ast}$ and $\langle \Gamma _{i},\Gamma _{j} \rangle (t)$ is the bounded variation process such that $\Gamma _{i}(t)\Gamma _{j}(t)- \langle \Gamma _{i},\Gamma _{j} \rangle (t)$ is a $( P,\mathcal{F}_{t} ) -$ local martingale for all i, j. According to Theorem III-10 in [15], on an extension of the probability space $( \Omega ,\mathcal{F},\mathcal{F}_{t},P )$, there exists a family of d-strongly orthogonal continuous martingale measures $M=(M^{k})_{k=1}^{d}$, each of them having intensity $dt\mu _{t}(da)$ such that

$$\begin{aligned}& \Gamma _{i}(t)=\sum_{k=1}^{d} \int _{0}^{t} \int _{\mathbb{A}}\sigma _{ik}(s,X_{s},a)M^{k}(ds,da), \end{aligned}$$

which achieves the proof.

2) The proof is similar to the existence and uniqueness of the solution of an SDE under Lipschitz conditions [21]. □

Remark 3.9

i) Note that the family of orthogonal martingale measures $M=(M^{k})_{k=1}^{d}$ corresponding to the relaxed control $\mathit{dt}\mu _{t}(da)$ is not unique.

ii) From now on, the probability space and the Brownian motion $( B_{t} )$ are fixed. So, a relaxed control will be defined as in Definition 3.3. The Brownian motion $( B_{t} )$ remains a Brownian motion on this new probability space, but the filtration is no longer the natural filtration of $( B_{t} )$.

Now we are able to define precisely the relaxed control problem by the following.

Minimize over $\mathcal{R}$ the cost functional $\mathcal{J}(\mu )$ defined by (3.4) subject to the relaxed dynamics (3.5).

3.3.1 Approximation of the relaxed control problem

In this section we will prove that the relaxed control problem is the closure of the set of strict controls. This means that if $( u^{n} )$ is a sequence of strict controls such that $( \delta _{u_{t}^{n}}(da)\,dt )$ converges to $\mu _{t}(da)\,dt$ weakly, then the sequence of corresponding trajectories $( X^{n} )$ converges to $X^{\mu}$ where $X^{\mu}$ is the solution of relaxed SDE (3.5). This implies in particular that the map $\mu \longrightarrow X^{\mu}$ is continuous; and as a consequence, the strict and relaxed problems have the same value function.

The following lemma [11, 29], which is classical in deterministic as well as in stochastic control, shows that the closure (for the topology of weak convergence) of the set of strict controls is exactly the set of relaxed controls. We give the proof for the sake of completeness.

Lemma 3.10

(Chattering lemma))

Let μ be a relaxed control. Then there exists a sequence of strict controls $(u^{n})$ with values in $\mathbb{A}$ such that

$$ \mu _{t}^{n}(da)\,dt=\delta _{u_{t}^{n}}(da)\,dt\quad \textit{converges weakly to }\mu _{t}(da)\,dt \quad P\textit{-a.s.} $$

Proof

For any g continuous in $[0,T]\times \mathbb{A}$, suppose that $\mu (t,da)$ has continuous sample paths. Let $n\geq 1$, and let $( T_{i}= [ t_{i},s_{i} [ )$ be subintervals of the interval $[0,T]$ of length not exceeding $2^{-n}$. Cover A by finitely many disjoint sets $( A_{j} )$ such that diameter $(A_{j})\leq 2^{-n}$. Choose a point $(t_{i},a_{ij})$ in $T_{i}\times A_{j}$. We have $\sum_{j}\mu (t_{i},A_{j})=1$. Subdivide each $T_{i}$ further into disjoint left-closed, right-open intervals $T_{ij}$ such that its length is the product of $\mu (t_{i},A_{j})$ with the length of $T_{i}$. ∀ $\varepsilon >0$, for n large enough, we have

$$\begin{aligned} & \bigl\vert g(t,a)-g(t_{i},a_{ij}) \bigr\vert < \varepsilon \quad \text{for }(t,a)\in T_{i}\times A_{j}, \\ &\sup_{a} \bigl\vert g(t,a)-g(t_{i},a) \bigr\vert < \varepsilon \quad \text{for }t\in T_{i}. \end{aligned}$$

Define the sequence of predictable process $\mu ^{n}(\cdot)$ by $\mu ^{n}(t,da)=\delta _{a_{ij}}(da)$ for $t\in T_{ij}$. And by path-continuity of $u(\cdot)$, we may increase n further if necessary to obtain

$$\begin{aligned} & \biggl\vert { \int _{0}^{T}} { \int _{A}} g(t,a) \mu ^{n}(t,da)\,dt-{ \int _{0}^{T}} { \int _{A}} g(t,a) \mu (t,da)\,dt \biggr\vert \\ & \quad\leq 4\varepsilon T+ \biggl\vert \sum_{i,j} \biggl( { \int _{T_{ij}}} g(t,a_{ij})\,dt-{ \int _{T_{ij}}} { \int _{A}} g(t_{i},a_{ij})\mu (t_{i},da)\,dt \biggr) \biggr\vert \\ & \quad\leq 4\varepsilon T, \end{aligned}$$

which completes the proof. In case $\mu (t,da)$ is not continuous, we use an approximation by continuous functions. □

Proposition 3.11

1) Let $\mu =\mu _{t}(da)\,dt$ be a relaxed control. Then there exists a continuous orthogonal martingale measure $M(dt,da)$ whose covariance measure is given by $\mu _{t}(da)\,dt$.

2) If we denote $M^{n}(t,B)=\int _{0}^{t}\int _{B}\delta _{u_{s}^{n}}(da)\,dW_{s}$, where $( u^{n} )$ is defined as in the last lemma, then for every bounded predictable process $\varphi :\Omega \times [ 0,T ] \times \mathbb{A} \rightarrow \mathbb{R}$, such that $\varphi (\omega ,t,.)$ is continuous, we have

$$\begin{aligned}& E \biggl[ \biggl( \int _{0}^{t} \int _{\mathbb{A}}\varphi (\omega ,t,a)M^{n}(dt,da)- \int _{0}^{t} \int _{\mathbb{A}}\varphi (\omega ,t,a)M(dt,da) \biggr) ^{2} \biggr] \rightarrow 0\quad \textit{as } n\longrightarrow + \infty \end{aligned}$$

for a suitable Brownian motion B defined on an eventual extension of the probability space.

Proof

See [25] pages 196–197. □

The following theorem gives us the continuity of the controlled dynamics with respect to the control variable in the sense of law.

Theorem 3.12

Let μ be a relaxed control and $X^{\mu}$ be the corresponding relaxed process. Assume that the relaxed SDE (3.5) has a unique weak solution. Then there exists a sequence $(u^{n})$ of strict controls such that the sequence $( X^{u^{n}} )$ converges in law to $X^{\mu}$.

Proof

According to the Chattering lemma, there exists $(u^{n})$ of strict controls such that $dt\delta _{u_{t}^{n}}(da)$ converges weakly to $\mu _{t}(da)\,dt$, P-a.s. Let $( X^{u^{n}} )$ and $X^{\mu}$ be the solutions of (3.5) corresponding to $dt\delta _{u_{t}^{n}}(da)$ and $dt\mu _{t}(da)$.

$$\begin{aligned} E \bigl( \bigl\vert X_{t}^{u^{n}}-X_{s}^{u^{n}} \bigr\vert ^{2}\vert \mathcal{F}_{t}\bigr) \leq{}& E{ \int _{s}^{t}} { \int _{ \mathbb{A}}} \bigl\vert b\bigl(u,X_{s}^{u^{n}},a \bigr) \bigr\vert ^{2}\delta _{u_{s}^{n}}(da)\,ds \\ &{}+{ \int _{s}^{t}} { \int _{ \mathbb{A}}} \bigl\vert \sigma \bigl(u,X_{s}^{u^{n}},a \bigr) \bigr\vert ^{2} \delta _{u_{s}^{n}}(da)\,ds. \end{aligned}$$

Since b and σ are bounded, it follows that

$$\begin{aligned}& E \bigl( \bigl\vert X_{t}^{u^{n}}-X_{s}^{u^{n}} \bigr\vert ^{2}\vert \mathcal{F}_{t}\bigl) \leq C \vert t-s \vert . \end{aligned}$$

Therefore $( X^{u^{n}} )$ is tight on the space $\mathbb{D}( [ 0,T ] ,\mathbb{R}^{d})$. Since $( dt\delta _{u_{t}^{n}}(da) )$ converges weakly to $dt\mu _{s}(da)$ and $M^{n}(dt,da)=\delta _{u_{s}^{n}}(da)\,dB_{s}$ converges to $M(dt,da)$, and according to the uniqueness in law of the relaxed SDE (3.5), it holds that $( X^{u^{n}} )$ converges in law to $X^{\mu}$. □

We will prove under pathwise uniqueness that the approximation holds in quadratic mean.

Theorem 3.13

Let μ be a relaxed control, and let X be the solution of (3.5). Assume that the coefficients of stochastic differential equation (2.1) are continuous and bounded. Assume also that pathwise uniqueness holds for (3.5). Then there exists a sequence $(u^{n})$ of strict controls such that

$$\begin{aligned}& \textit{i)}\quad \lim_{n\rightarrow \infty}E \Bigl[ \sup _{0\leq t\leq T} \bigl\vert X_{t}^{n}-X_{t} \bigr\vert ^{2} \Bigr] =0. \\& \textit{ii)}\quad \textit{There exists a subsequence }\bigl(u^{n_{k}} \bigr)\textit{ such that }J\bigl(u^{n_{k}}\bigr) \textit{converges to }J(\mu ), \end{aligned}$$

(3.6)

where $X^{n}$ denotes the solution of the stochastic differential equation (3.5) associated with $(u^{n})$.

Proof

i) Let μ be a relaxed control, then by Lemma 3.10 there exists a sequence $(u^{n})$ such that $\mu _{t}^{n}(da)\,dt=\delta _{u_{t}^{n}}(da)\,dt\longrightarrow \mu _{t}(da)\,dt$ in $\mathcal{R}$, P-a.s. Let $X^{n}$ and X be the solutions of (3.5) associated with $\mu ^{n}$ and μ, respectively. Suppose that the result of Theorem 3.13 is false, then there exists $\gamma >0$ such that

$$ \inf_{n}E \bigl[ \bigl\vert X_{t}^{n}-X_{t} \bigr\vert ^{2} \bigr] \geq \gamma . $$

(3.7)

According to the compactness of $\mathbb{A}$ and the boundedness of the coefficients of SDE (3.5), it follows that the family of processes

$$ \Gamma ^{n}=\bigl(\mu ^{n},\mu ,X^{n},X,M^{n},M \bigr) $$

is tight. Then, by the Skorokhod selection theorem [21], there exist a probability space $(\widetilde{\Omega},\widetilde{\mathcal{F}},\widetilde{P})$ and a sequence $\widetilde{\Gamma}^{n}=(\widetilde{\mu}^{n},\widetilde {\upsilon}^{n}, \widetilde{X}^{n},\widetilde{Y}^{n},\widetilde{M}^{n},\widetilde{N}^{n})$ defined on it such that:

i) For each $n\in \mathbb{N}$, the laws of $\Gamma ^{n}$ and $\widetilde{\Gamma}^{n}$ coincide;

ii) There exists a subsequence $(\widetilde{\Gamma}^{n_{k}})$, still denoted by of $\widetilde{\Gamma}^{n}$, which converges to Γ̃, $\widetilde{P}-a.s$., where $\widetilde{\Gamma}=(\widetilde{\mu},\widetilde{\upsilon},\widetilde{X},\widetilde{Y},\widetilde{M}, \widetilde {N})$. By the uniform integrability, we have

$$ \gamma \leq \lim \inf_{n}E \Bigl[ \sup_{0\leq t\leq T} \bigl\vert X_{t}^{n}-X_{t} \bigr\vert ^{2} \Bigr] =\lim \inf_{n} \widetilde{E} \Bigl[ \sup_{0\leq t\leq T} \bigl\vert \widetilde{X}^{n}-\widetilde{Y}^{n} \bigr\vert ^{2} \Bigr] =\widetilde{E} \Bigl[ \sup_{0\leq t\leq T} \vert \widetilde{X}-\widetilde{Y} \vert ^{2} \Bigr], $$

where Ẽ is the expectation with respect to P̃. According to i), we see that $\widetilde{X}^{n}$ and $\widetilde{Y}^{n}$ satisfy the following equations:

$$\begin{aligned}& \textstyle\begin{cases} d\widetilde{X_{s}}^{n}={\int _{A}} b(s,\widetilde{X_{s}}^{n},a) \widetilde{\mu}^{n}(da)\,ds+{\int _{A}} \sigma (s,\widetilde{X_{s}}^{n},a)\,d \widetilde{M}^{n}(ds,da) \\ \widetilde{X_{0}}^{n}=x, \end{cases}\displaystyle \\& \textstyle\begin{cases} d\widetilde{Y_{s}}^{n}={\int _{A}} b(s,\widetilde{Y_{s}}^{n},a) \widetilde{\upsilon}^{n}(da)\,ds+{\int _{A}} \sigma (s,\widetilde{Y_{s}}^{n},a)\,d \widetilde{N}^{n}(ds,da) \\ \widetilde{Y_{0}}=x. \end{cases}\displaystyle \end{aligned}$$

Since $( \widetilde{\Gamma}^{n} )$ converges to Γ̃, $\widetilde{P}-a.s$., $(\widetilde{X}^{n})$ and $(\widetilde{Y}^{n})$ converge respectively to X̃ and Ỹ, which satisfy

$$\begin{aligned}& \textstyle\begin{cases} d\widetilde{X_{s}}={\int _{A}} b(s,\widetilde{X_{s}},a) \widetilde{\mu}(da)\,ds+{\int _{A}} \sigma (s,\widetilde{X_{s}},a)\,d \widetilde{M}(ds,da) \\ \widetilde{X_{0}}=x, \end{cases}\displaystyle \\& \textstyle\begin{cases} d\widetilde{Y_{s}}={\int _{A}} b(s,\widetilde{Y_{s}},a) \widetilde{\upsilon}(da)\,ds+{\int _{A}} \sigma (s,\widetilde{Y_{s}},a)\,d \widetilde{N}(ds,da) \\ \widetilde{Y_{0}}=x. \end{cases}\displaystyle \end{aligned}$$

According to Lemma 3.10, the sequence $(\mu ^{n},\mu )$ converges to $(\mu ,\mu )$ in $\mathcal{R}^{2}$. Moreover,

$$\begin{aligned}& \operatorname{law}\bigl(\mu ^{n},\mu \bigr)=\operatorname{law}\bigl( \widetilde{\mu}^{n},\widetilde{\upsilon}^{n} \bigr),\\& \bigl(\widetilde{\mu}^{n},\widetilde{\upsilon}^{n}\bigr) \implies ( \widetilde{\mu },\widetilde{\upsilon}),\quad \widetilde{P}\text{-a.s in } \mathcal{R}^{2}. \end{aligned}$$

Hence, $\operatorname{law}(\widetilde{\mu},\widetilde{\upsilon})=\operatorname{law}(\mu ,\mu )$, which implies that $\widetilde{\mu}=\widetilde{\upsilon}$, $\widetilde{P}-a.s$. By the same method, we can prove that $\widetilde{M}=\widetilde{N}$, $\widetilde{P}-a.s$. According to the pathwise uniqueness of equation (3.5), it follows that $\widetilde{X}=\widetilde{Y},\widetilde{P}-a.s$., which contradicts (3.7). i) is proved.

ii) This is a direct consequence of i) along with the continuity and boundedness of the functions h and g. □

Remark 3.14

1) Using the same arguments, we can replace the sequence $( \delta _{u_{t}^{n}}(da)\,dt )$ by any sequence $( \mu _{t}^{n}(da)\,dt )$ of relaxed controls converging weakly to $\mu _{t}(da)\,dt$. This means in particular that the function $\mu \longrightarrow X^{\mu}$ is continuous.

2) As a consequence of the last theorem, the value functions of the strict and relaxed control problems are equal. Therefore, by relaxing the control problem, the value function remains unchanged. Moreover, the relaxed control problem has an optimal solution.

3.3.2 Discussion of another relaxed model

Assume that both the drift and the diffusion coefficients are controlled. Let us consider another type of relaxation of the controlled stochastic differential equation, suggested in the literature by many authors [1, 6, 35]. Instead of relaxing the infinitesimal generator of the controlled process, the authors considered the direct relaxation of the stochastic differential equation as in deterministic control. This is carried out by integrating directly the drift and diffusion coefficient against the relaxed control, which gives the following equation:

$$ \textstyle\begin{cases} dX_{t}={\int _{\mathbb{A}}} b(t,X_{t},a)\mu _{t}(da)\,dt+{\int _{\mathbb{A}}} \sigma (t,X_{t},a)\mu _{t}(da)\,dB_{t} \\ X_{0}=x. \end{cases} $$

(3.8)

This “relaxed” form has the advantage to be linear with respect to the control variable, with a convex compact set of controls. However, its solution has a serious drawback in that it is not continuous with respect to the control variable. As a consequence, it follows that the value functions of the strict and relaxed control problems cannot be equal. Therefore, it cannot be considered as a true relaxed model. Moreover we have no mean to prove the existence of an optimal relaxed control as the dynamics and the cost functional are not continuous with respect to the control variable.

Indeed, consider the control problem governed by the following SDE:

$$\begin{aligned}& \textstyle\begin{cases} dX_{t}=u_{t}\,dB_{t} \\ X_{0}=x, \end{cases}\displaystyle \end{aligned}$$

where admissible controls are measurable functions $u:[0,1]\rightarrow \mathbb{A}\mathbbm{=} \{ -1,1 \}$.

The corresponding “relaxed” equation is defined by

$$ \textstyle\begin{cases} dX_{t}={\int _{\mathbb{A}}} a\mu _{t}(da)\,dB_{t} \\ X_{0}=x. \end{cases} $$

(3.9)

Proposition 3.15

The solution of the controlled SDE (3.9) is not continuous in the control variable.

Proof

Consider the sequence of Rademacher functions

$$\begin{aligned}& u_{n}(t)=(-1)^{k}\quad \text{if } \frac{k}{n}\leq t\leq \frac{(k+1)}{n}, 0\leq k\leq n-1. \end{aligned}$$

According to Lemma 3.1, the sequence of relaxed controls $( dt.\delta _{u_{{n}}(t)}(da) )$ converges weakly to $dt.\frac{1}{2}(\delta _{-1}+\delta _{1})(da)$.

Let $X_{t}^{n}$ be the solution of SDE (3.9) associated with the relaxed control $dt.\delta _{u_{{n}}(t)}(da)$. It is clear that

$$X_{t}^{n}={ \int _{0}^{t}} \biggl[ { \int _{\mathbb{A}}} a\delta _{u_{n}(s)}(da) \biggr]\,dB_{s}={ \int _{0}^{t}} u_{n}(s)\,dB_{s} $$

is a continuous martingale with quadratic variation $\langle X^{n},X^{n} \rangle _{t}={\int _{0}^{t}} u_{n}^{2}(s).ds=t$. Therefore $( X_{t}^{n} )$ is a Brownian motion constructed possibly on an augmented probability space.

Let $X^{\ast}$ be the relaxed state process corresponding to the limit $\mu ^{\ast}=dt.\frac{1}{2}(\delta _{-1}+\delta _{1})(da)$, then

$$ X^{\ast}(t)={ \int _{0}^{t}} { \int _{ \mathbb{A}}} a.(1/2) (\delta _{-1}+\delta _{1}) (da)\,dB_{t}=0. $$

It is obvious that the sequence of state processes $( X_{t}^{n} )$ does not converge in any topology to $X_{t}^{\ast}$. Indeed

$$ E \bigl[ \bigl\vert X_{t}^{n}-X_{t}^{\ast} \bigr\vert ^{2} \bigr] =E \bigl[ \bigl\vert X_{t}^{n} \bigr\vert ^{2} \bigr] =E \biggl[ \biggl\vert { \int _{0}^{t}} u_{n}(s).dB_{s} \biggr\vert ^{2} \biggr] ={ \int _{0}^{t}} u_{n}^{2}(s).ds=t. $$

 □

Remark 3.16

1) It is clear that the right limit is a Brownian motion, which could be represented as $X^{\ast}(t)={\int _{0}^{t}} {\int _{ \mathbb{A}}} a.M(ds,da)$ where $M ( dt,da ) ={\sum_{i=1}^{2}} \sqrt{\frac{1}{2}}\,dB_{{ s}}^{i}1_{ ( {\alpha}_{i}\in da ) }$, where $B^{1}$ and $B^{2}$ are independent Brownian motions constructed possibly on an augmentation of the probability space.

2) As a consequence of the last proposition, the value functions of the strict and “relaxed” control problems could be different. Moreover, even if the set $\mathbb{V}$ is compact, there is no mean to prove the existence of an optimal control for this model.

3) Unlike the model based on SDE (3.5), the controlled stochastic equation (3.8) is driven by the martingale measure $\mu _{t}(da)\,dB_{t}$, which is not orthogonal. Its intensity is given by $\mu _{t}(da)\otimes \mu _{t}(da)\otimes dt$. This is a worthy martingale measure in the sense of Walsh [34].

4 Necessary conditions for optimality

We know from the previous section that an optimal relaxed control μ exists in the set $\mathcal{R}$. This implies the existence of a filtered probability space still denoted by $(\Omega ,\mathcal{F},( \mathcal{F}_{t}) _{t\geq 0},P)$, a measure-valued control $dt\mu _{t} ( da )$, and an orthogonal martingale measure $M(da,dt)$ whose covariance measure is $dt\mu _{t} ( da )$ such that:

$$ \textstyle\begin{cases} dX_{t}=\int _{\mathbb{A}}b(t,X_{t},a)\,\mu _{t}(da)\,dt+\int _{\mathbb{A}}\sigma (t,X_{t},a)\,M(da,dt) \\ X(0)=x \end{cases} $$

(4.1)

and

$$ J(\mu )=\inf \bigl\{ J(\nu );\nu \text{ }\in \mathcal{R} \bigr\} . $$

(4.2)

Our goal in this section is to derive necessary conditions for optimality satisfied by the optimal relaxed control μ. According to the Chattering lemma, $dt\mu _{t} ( da )$ can be approximated in the sense of weak convergence by a sequence ($u^{n}$) of strict controls. We start by establishing the necessary conditions of near optimality that are satisfied by the strict controls $(u^{n})$. This important auxiliary result is based on Ekeland’s variational principle [13] and is interesting in itself. Indeed in most practical situations it is sufficient to characterize and compute nearly optimal controls.

Lemma 4.1

(Ekeland’s variational principle)

Let $(E,d)$ be a complete metric space and $f:E\rightarrow \overline{\mathbb{R} }$ be lower semicontinuous and bounded from below. Given $\varepsilon >0$, suppose that $u^{\varepsilon}\in E$ satisfies $f(u^{\varepsilon})$ $\leq \inf (f)+\varepsilon$. Then, for any $\lambda >0$, there exists $\nu \in E$ such that

• $f(\nu )$ $\leq f(u^{\varepsilon})$,

• $d(u^{\varepsilon},\nu )\leq \lambda$,

• $f(\nu )\leq f(\omega )+\frac{\varepsilon}{\lambda}\,d(\omega ,\nu )$ for all $\omega \neq \nu$.

Let us endow the set $\mathcal{U}_{\mathrm{ad}}$ of strict controls with an appropriate metric. For any u and $v\in \mathcal{U}_{\mathrm{ad}}$, we set

$$ d(u,\nu )=P\otimes dt \bigl\{ (\omega ,t)\in \Omega \times [ 0;T ] ;u(t,\omega ) \neq \nu (t,\omega ) \bigr\} , $$

where $P\otimes dt$ is the product measure of P with the Lebesgue measure dt.

Remark 4.2

It is well known that $(\mathcal{U}_{\mathrm{ad}},d)$ is a complete metric space and that the cost functional J is continuous from $\mathcal{U}_{\mathrm{ad}}$ into $\mathbb{R}$, see [26].

Now, let $\mu \in \mathcal{R}$ be an optimal relaxed control and denote by X the solution of (4.1) controlled by μ. From Lemma 3.10 and Theorem 3.13, there exists a sequence $(u^{n})$ of strict controls such that

$$ \mu _{t}^{n}(da)\,dt=\delta _{u_{t}^{n}}(da)\,dt \longrightarrow \mu _{t}(da)\,dt \quad \text{weakly }P\text{-a.s, as }n \rightarrow +\infty $$

and

$$ \lim_{n\rightarrow \infty}E \bigl[ \bigl\vert X_{t}^{n}-X_{t}^{\mu} \bigr\vert ^{2} \bigr] =0, $$

where $X^{n}$ is the solution of (3.5) corresponding to $\mu ^{n}=\delta _{u_{t}^{n}}(da)\,dt$.

Let us introduce the usual Hamiltonian of the system

$$ H ( t,x,u,p,q,r ) = \bigl\langle b ( t,x,u ) ,p \bigr\rangle +\operatorname{tr} \bigl( q^{\ast}\sigma ( t,x,u ) \bigr) -h ( t,x,u ), $$

where $A^{\ast}$ denotes the transpose of the vector or matrix A.

We define as in [36] by $( p,q )$ and $( P,Q )$ the first- and second-order adjoint processes satisfying the following backward SDEs, assuming that $( \mathcal{F}_{t} )$ is the natural filtration of the Brownian motion:

$$\begin{aligned}& \textstyle\begin{cases} dp(t)= - [ b_{x}^{\ast}(t)p(t)+\sum_{j=1}^{d}\sigma _{x}^{j}(t)q^{j}(t)-h_{x}(t) ]\,dt+q_{t}\,dB_{t} \\ p_{T}= -g_{x}(x_{T}). \end{cases}\displaystyle \end{aligned}$$

(4.3)

$$\begin{aligned}& \textstyle\begin{cases} dP_{t}= - [ b_{x}^{\ast}(t)P_{t}+P_{t}b_{x}(t)+\sum_{j=1}^{d}\sigma _{x}^{j\ast}(t)P_{t}\sigma _{x}^{j}(t)Q_{t}\\ \hphantom{dP_{t}=}{} +\sum_{j=1}^{d} [ \sigma _{x}^{j\ast}(t)Q_{t}^{j}+Q_{t}^{j}\sigma _{x}^{j}(t) ] +H_{xx} ( t ) ] +\sum Q_{t}^{j}\,dB_{t} \\ P_{T}= =-g_{xx}(x_{T}), \end{cases}\displaystyle \end{aligned}$$

(4.4)

where $b_{x}(t)=b_{x}(t,X_{t},u_{t})$ and $\sigma _{x}^{j}(t)=\sigma _{x}^{j}(t,X_{t},u_{t})$ and $h_{x}(t)=h_{x}(t,X_{t},u_{t})$.

Under conditions $(\mathbf{A}_{1})$ and $(\mathbf{A}_{2})$, BSDEs (4.3) and (4.4) have unique solutions satisfying the following estimates:

$$\begin{aligned} & E \biggl[ \sup_{0\leq t\leq T} \vert p_{t} \vert ^{2}+{ \int _{0}^{T}} \vert q_{t} \vert ^{2}\,dt \biggr] < \infty , \\ & E \biggl[ \sup_{0\leq t\leq T} \vert P_{t} \vert ^{2}+{ \int _{0}^{T}} \vert Q_{t} \vert ^{2}\,dt \biggr] < \infty . \end{aligned}$$

Remark 4.3

In case $( \mathcal{F}_{t} )$ is not necessarily the natural filtration of the Brownian motion, we must add in the backward equations (4.3) and (4.4) two cadlag martingales that are orthogonal to the Brownian motion. This comes from the Itô representation theorem for Brownian martingales.

4.1 Necessary conditions for near optimality

The generalized Hamiltonian $\mathcal{H}$ associated with a strict control u and the corresponding state process X is defined as in [36] by

$$ \mathcal{H}^{(u(\cdot),X(\cdot))} ( t,y,v ) =H \bigl( t,y,v,p_{t},q_{t}-P_{t} \sigma ( t,X_{t},u_{t} ) \bigr) - \frac{1}{2} \operatorname{Tr} \bigl[ \sigma ( t,X_{t},u_{t} ) ^{\ast}P_{t} \sigma ( t,X_{t},u_{t} ) \bigr], $$

where $( p,q )$ and $( P,Q )$ are solutions of the adjoint equations (4.3) and (4.4). The following theorem gives necessary conditions for near optimality for the strict control $u^{n}$ in terms of an approximate maximum principle. See [38] Theorem 4.1 for a complete proof of this intermediary result.

Proposition 4.4

There exists a sequence of strict controls $(u^{n})$ such that

$$ J\bigl(u^{n}\bigr)=J\bigl(\mu ^{n}\bigr)\leq J(\mu )+ \varepsilon _{n}=\inf_{\nu \in \mathcal{R}}J(\nu )+ \varepsilon _{n}, $$

and there exist unique adapted solutions $( p^{n},q^{n} )$ and $( P^{n},Q^{n} )$ of the adjoint equations (4.3) and (4.4), corresponding to the admissible pair $(u^{n},X^{n})$, such that for any $\gamma \in [ 0,1/3 )$

$$ E \biggl[ { \int _{0}^{T}} \mathcal{H} \bigl( t,X_{t}^{n},u_{t}^{n} \bigr)\,dt \biggr] \geq \sup_{a\in \mathbb{A}}E \biggl[ { \int _{0}^{T}} \mathcal{H} \bigl( t,X_{t}^{n}, \alpha \bigr)\,dt \biggr] -\varepsilon ^{\gamma }. $$

(4.5)

Proof

Let us give the outline of the proof. According to the optimality of μ and the Chattering lemma, there exist a sequence $(\varepsilon _{n})$ of positive numbers with $\lim_{n\rightarrow \infty}\varepsilon _{n}=0$ and a sequence of strict controls $(u^{n})$ such that $(u^{n})$

$$ J\bigl(u^{n}\bigr)=J\bigl(\mu ^{n}\bigr)\leq J(\mu )+ \varepsilon _{n}=\inf_{u\in U}J(u)+ \varepsilon _{n}. $$

According to a suitable version of Lemma 4.1 with $\lambda =\varepsilon ^{\frac{2}{3}}$,

$$ J\bigl(u^{n}\bigr)\leq J(u)+\varepsilon ^{\frac{1}{3}}d \bigl(u^{n},u\bigr),\quad \forall u\in \mathcal{U}_{\mathrm{ad}}. $$

(4.6)

Let us define the perturbation

$$ u^{n,h}= \textstyle\begin{cases} a&\text{if }t\in [ t_{0};t_{0}+h ] \\ u^{n}&\text{otherwise}. \end{cases} $$

(4.7)

From (4.6) we have

$$ 0\leq J\bigl(u^{n,h}\bigr)-J\bigl(u^{n}\bigr)+\varepsilon ^{\frac{1}{3}}d\bigl(u^{n,h},u^{n}\bigr). $$

Using the definition of d, it holds that

$$ 0\leq J\bigl(u^{n,h}\bigr)-J\bigl(u^{n}\bigr)+\varepsilon ^{\frac{1}{3}}h. $$

(4.8)

Let us denote by $x^{n,h}$ the solution of (2.1) corresponding to $u^{n,h}$, which is defined in (4.7). To get the desired variational inequality we differentiate the function $J(u^{n,h})$ with respect to h at $h=0$. See [38] Theorem 4.1 for details. □

4.2 The relaxed maximum principle

Let X be the corresponding optimal state process associated with the optimal relaxed control μ, and $( p,q )$ and $( P,Q )$ be the solutions of the adjoint equations (4.9) and (4.10) associated with $( \mu ,X )$. We assume that $( \mathcal{F}_{t} )$ is the natural filtration of the Brownian motion.

$$ \textstyle\begin{cases} dp(t)= - [ b_{x}^{\ast}(t)p(t)+\sum_{j=1}^{d}\sigma _{x}^{jT}(t)q^{j}(t)-h_{x}(t) ]\,dt+q_{t}\,dB_{t} \\ p_{T}= -g_{x}(x_{T}), \end{cases} $$

(4.9)

and

$$ \textstyle\begin{cases} dP_{t}= - [ \overline{b}_{x}^{\ast}(t)P_{t}+P_{t} \overline{b}_{x}(t)+\sum_{j=1}^{d}\overline{\sigma}_{x}^{j\ast}(t)P_{t}\overline{\sigma}_{x}^{j}(t)Q_{t} \\ \hphantom{dP_{t}=} +\sum_{j=1}^{d} [ \overline {\sigma}_{x}^{j\ast}(t)Q_{t}^{j}+Q_{t}^{j}\overline{\sigma}_{x}^{j}(t) ] +\overline{H}_{xx} ( t ) ]\,dt+\sum Q_{t}^{j}\\ P_{T} =-g_{xx}(X_{T}), \end{cases} $$

(4.10)

where $\overline{k}=k(t,X_{t},\mu _{t})={\int _{A}} k(t,X_{t},a)\mu _{t}(da)$, and k stands to be $b_{x}$, $\sigma _{x}$, $f_{x}$, $h_{x}$, and $H_{xx}$.

The generalized Hamiltonian function associated with the optimal pair $( \mu ,X )$ is defined by

$$ \mathcal{H}^{(\mu ,X(\cdot))} ( t,y,v ) =H \bigl( t,y,v,p_{t},q_{t}-P_{t} \overline{\sigma} ( t,X_{t},\mu ) \bigr) - \frac{1}{2} \operatorname{Tr} \bigl[ \sigma ( t,X_{t},\mu ) ^{\ast}P_{t} \sigma ( t,X_{t},\mu ) \bigr]. $$

Theorem 4.5

(Relaxed maximum principle)

Assume ($\mathbf{A}_{1}$) and (A2). Let $( \mu ,X )$ be an optimal pair, then there exist unique adapted solutions $(p,q)$ and $(P,Q)$ of the adjoint equations (4.9) and (4.10), respectively, such that

$$ E \biggl[ { \int _{0}^{T}} \mathcal{H} ( t,X_{t}, \mu )\,dt \biggr] =\sup_{\alpha \in \mathbb{A}}E \biggl[ { \int _{0}^{T}} \mathcal{H} ( t,X_{t}, \alpha )\,dt \biggr]. $$

(4.11)

The proof of this theorem is based on the following stability theorem of adjoint processes with respect to the control variable.

Theorem 4.6

(Stability theorem for BSDEs)

Let $(p^{n},q^{n})$, $(P^{n},Q^{n})$, and $( \textit{resp}.(p,q),(P,Q) )$ be the solutions of (4.3) and (4.4) associated with the pair $( u^{n},X^{n} )$ (resp the solutions of (4.9) and (4.10) associated with the pair $( \mu ,X )$. Then we have

$$ \textit{i)}\quad \lim_{n\rightarrow \infty}E \biggl[ \sup _{t\leq T} \bigl\vert p^{n}-p \bigr\vert ^{2}+{ \int _{t}^{T}} \bigl\vert q^{n}-q \bigr\vert ^{2}\,ds \biggr] =0 $$

(4.12)

and

$$ \textit{ii)}\quad \lim_{n\rightarrow \infty}E \biggl[ \sup _{t\leq T} \bigl\vert P^{n}-P \bigr\vert ^{2}+{ \int _{t}^{T}} \bigl\vert Q^{n}-Q \bigr\vert ^{2}\,ds \biggr] =0. $$

(4.13)

Proof

i) Let us write down the drivers of the first-order adjoint equations (4.3) and (4.9) corresponding to $(u^{n},X^{n})$ and $( \mu ,X)$.

$$\begin{aligned}& G^{n}\bigl(t,p_{t}^{n},q_{t}^{n} \bigr)=-b_{x}^{n}(t)p^{n}(t)+\sum _{j=1}^{d}\sigma _{x}^{j,n}(t)q^{n}(t)-h_{x}^{n}(t) \\& G(t,p_{t},q_{t})=-\overline{b}_{x}(t)p(t)+ \sum_{j=1}^{d}\overline{ \sigma}_{x}^{j}(t)q(t)-\overline{h}_{x}(t), \end{aligned}$$

where

$$\begin{aligned}& f^{n}(t)=f\bigl(t,X_{t}^{n},u_{t}^{n} \bigr)= { \int _{\mathbb{A}}} f\bigl(t,X_{t}^{n},a \bigr)\delta _{u_{t}^{n}}(da)\quad \text{for }f=b_{x}, \sigma _{x}, h_{x}, \\& \overline{f}(t)=f\bigl(t,X(t),\mu (t)\bigr)= \int _{A}f\bigl(t,X(t),a\bigr)\mu (t,da)\quad \text{where }f\text{ stands for }b_{x}, \sigma _{x}, h_{x}. \end{aligned}$$

By using the result of Hu and Peng [20], Theorem 2.1, it is sufficient to show that

$$\begin{aligned}& \lim_{n\rightarrow \infty}E \biggl[ \biggl\vert { \int _{t}^{T}} \bigl( G^{n}(t,p_{t},q_{t})-G(t,p_{t},q_{t}) \bigr)\,dt \biggr\vert ^{2} \biggr] =0. \end{aligned}$$

Indeed, we have

$$ \begin{aligned} \biggl\vert { \int _{t}^{T}} \bigl( G^{n}(t,p_{t},q_{t})-G(t,p_{t},q_{t}) \bigr)\,dt \biggr\vert \leq{}& \biggl\vert { \int _{t}^{T}} \bigl( b_{x}^{n}(t)- \overline{b}_{x}(t) \bigr) p(t)\,dt \biggr\vert \\ & {}+ \biggl\vert { \int _{t}^{T}} \bigl( \sigma _{x}^{n}(t)- \overline{\sigma}_{x}(t) \bigr) q(t)\,dt \biggr\vert \\ & {}+ \biggl\vert { \int _{t}^{T}} \bigl( h_{x}^{n}(t)- \overline{h}_{x}(t) \bigr)\,dt \biggr\vert . \end{aligned} $$

(4.14)

Let us deal with the first term on the right-hand side of (4.14).

$$ \begin{aligned} & { \int _{t}^{T}} \bigl( b_{x}^{n}(t)- \overline{b}_{x}(t) \bigr) p(t)\,dt \\ &\quad={ \int _{t}^{T}} \biggl( { \int _{\mathbb{A}}} b_{x}\bigl(t,X_{t}^{n},a \bigr)\delta _{u_{t}^{n}}(da)- \int _{A}b_{x}(t,X_{t},a) \mu _{t}(da) \biggr) p(t)\,dt \\ &\quad={ \int _{t}^{T}} \biggl( { \int _{\mathbb{A}}} b_{x}\bigl(t,X_{t}^{n},a \bigr)\delta _{u_{t}^{n}}(da)- \int _{A}b_{x}(t,X_{t},a) \delta _{u_{t}^{n}}(da) \biggr) p(t)\,dt \\ &\qquad{}+{ \int _{t}^{T}} \biggl( { \int _{\mathbb{A}}} b_{x}(t,X_{t},a)\delta _{u_{t}^{n}}(da)- \int _{A}b_{x}(t,X_{t},a) \mu _{t}(da) \biggr) p(t)\,dt. \end{aligned} $$

(4.15)

$b_{x}$ being Lipschitz in x and $( X_{t}^{n} )$ converges to $X_{t}$ uniformly in t in probability imply that the first term on the right-hand side of (4.15) converges in probability to 0.

In addition, we have $E ( \sup_{0\leq t\leq T} \vert p(t) \vert ^{2} ) <+\infty$, therefore $\sup_{0\leq t\leq T} \vert p(t) \vert <+\infty$, P-a.s, which implies the existence of a P-negligible set N such that for each $\omega \notin N$ there exist $M(\omega )<+\infty$ s.t. $\sup_{0\leq t\leq T} \vert p(t) \vert \leq M( \omega )$.

In particular, for each $\omega \notin N$, the function $b_{x}(t,X_{t},E(X_{t}),a)p(t).1_{ [ 0,t ] }$ is a measurable bounded function in $(t,a)$ and continuous in a; therefore it is a test function for the stable convergence. Hence, by using the fact that $( \delta _{u_{t}^{n}}(da)\,dt )$ converges in $\mathbb{V}$ to $\mu _{t}(da)\,dt$, P-a.s., it follows that the second term on the right-hand side tends to 0, P-a.s.

The other terms containing $p(t)$ can be handled by using the same techniques.

The terms in (4.14) containing $q(t)$ can be treated similarly. However, one should pay a little more attention as $q(t)$ is just square integrable (in $(t,\omega )$). More precisely,

$$\begin{aligned} \biggl\vert { \int _{t}^{T}} \bigl( \sigma _{x}^{j,n}(t)- \overline{\sigma}_{x}(t) \bigr) q(t)\,dt \biggr\vert \leq{}& \biggl\vert { \int _{t}^{T}} \bigl( \sigma _{x}^{j,n}(t)- \overline{\sigma}_{x}(t) \bigr) q(t)1_{ \{ \vert q(t) \vert \leq N \} }\,dt \biggr\vert \\ & {}+ \biggl\vert { \int _{t}^{T}} \bigl( \sigma _{x}^{j,n}(t)- \overline{\sigma}_{x}(t) \bigr) q(t)1_{ \{ \vert q(t) \vert \geq N \} }\,dt \biggr\vert . \end{aligned}$$

The first integral on the right-hand side may be treated by using similar arguments as previously as the function $( \sigma _{x}^{n}(t)-\overline {\sigma}_{x}(t) ) q(t)1_{ \{ \vert q(t) \vert \leq N \} }$ is measurable bounded and continuous in a. The second term tends to 0 by Chebyshev’s inequality using the square integrability of $q(t)$.

ii) is proved by using similar arguments. □

Proof

of Theorem 4.5. The main result is proved by passing to the limit in inequality (4.5) and using Theorem 4.6 to get the desired inequality (4.11). □

Corollary 4.7

Under the same conditions as in Theorem 4.5it holds that

$$ E \biggl[ { \int _{0}^{T}} \mathcal{H} ( t,X_{t}, \mu )\,dt \biggr] = \sup_{\upsilon \in \mathcal{P} ( \mathbb{A} ) }E \biggl[ { \int _{0}^{T}} \mathcal{H} ( t,X_{t}, \upsilon )\,dt \biggr], $$

(4.16)

where $\mathcal{H} ( t,X_{t},\upsilon ) =\int _{\mathbb{A}}\mathcal{H} ( t,X_{t},a ) \upsilon (da)$ and $\mathcal{P} ( \mathbb{A} )$ is the space of probability measures on $\mathbb{A}$.

Proof

Since $\{ \delta _{a}(da);\text{ }a\in \mathbb{A} \} \subset$ $\mathbb{P} ( \mathbb{A} )$, it is clear that the inequality

$$ \sup_{\upsilon \in \mathbb{P} ( \mathbb{A} ) }E \biggl[ { \int _{0}^{T}} \mathcal{H} ( t,X_{t}, \upsilon ) \biggr] \geq \sup_{a\in \mathbb{A}}E \biggl[ { \int _{0}^{T}} \mathcal{H} ( t,X_{t},a ) \biggr] $$

is obvious. Let us prove the inequality from the other sense. If $\upsilon \in \mathbb{P} ( A )$ is a probability measure on $\mathbb{A}$, then

$$ E \biggl[ { \int _{0}^{T}} \mathcal{H} ( t,X_{t}, \upsilon )\,dt \biggr] \in \operatorname{conv} \biggl\{ E \biggl[ { \int _{0}^{T}} \mathcal{H} ( t,X_{t},a )\,dt \biggr] , a\in \mathbb{A} \biggr\} , $$

where $\operatorname{conv}(B)$ is the convex hull of B.

Hence, by using Fubini’s theorem, it holds that

$$ \sup_{\upsilon \in \mathbb{P} ( \mathbb{A} ) }E \biggl[ { \int _{0}^{T}} \mathcal{H} ( t,X_{t}, \upsilon )\,dt \biggr] \leq \sup_{a\in \mathbb{A}}E \biggl[ { \int _{0}^{T}} \mathcal{H} ( t,X_{t},a )\,dt \biggr] , $$

which implies that

$$ E \biggl[ { \int _{0}^{T}} \mathcal{H} ( t,X_{t}, \upsilon )\,dt \biggr] \leq \sup_{a\in \mathbb{A}} E \biggl[ { \int _{0}^{T}} \mathcal{H} ( t,X_{t},a )\,dt \biggr] . $$

 □

Remark

Since $\mathcal{P}(\mathbb{A})$ is a subspace of $\mathbb{V}$ whose elements are constant (in $( \omega ,t )$) relaxed controls, then (4.16) may be replaced by

$$ E \biggl[ { \int _{0}^{T}} \mathcal{H} \bigl( t,X_{t}, \mu ^{\ast} \bigr)\,dt \biggr] = \sup _{\upsilon \in \mathbb{V}}E \biggl[ { \int _{0}^{T}} \mathcal{H} ( t,X_{t},\upsilon _{t} )\,dt \biggr]. $$

(4.17)

Corollary 4.8

(Pontryagin’s relaxed maximum principle). Under the same conditions as in Theorem 4.5, there exists a Lebesgue negligible subset N in the interval $[ 0,T ]$ such that for any t not in N it holds that

$$ \mathcal{H} ( t,X_{t},\mu _{t} ) = \sup _{\upsilon \in \mathbb{V}}\mathcal{H} ( t,X_{t},\upsilon ) ,\quad P\textit{-a.s.} $$

(4.18)

Proof

Let $\varepsilon \in ] 0,T [$ and $B\in \mathcal{F}_{\varepsilon}$, for small $h>0$ define the relaxed control

$$\mu _{t}^{h}=\textstyle\begin{cases} \upsilon 1_{B}&\text{for }\varepsilon < t< \varepsilon +h \\ \mu _{t}&\text{otherwise,}\end{cases} $$

where υ is a probability measure on $\mathbb{A}$. It follows from (4.16) that

$$\begin{aligned}& 1/h \int _{\varepsilon}^{\varepsilon +h}\mathit{E} \bigl[ 1_{B} \mathcal{H} ( t,X_{t},\mu _{t} ) \bigr]\,dt\geq 1/h \int _{\theta}^{\theta +h}\mathit{E} \bigl[ 1_{B}\mathcal{H} ( t,X_{t},\upsilon ) \bigr]\,dt. \end{aligned}$$

Therefore passing at the limit as h tends to zero, we obtain

$$\begin{aligned}& \mathit{E} \bigl[ 1_{B}\mathcal{H} ( \varepsilon ,X_{ \varepsilon},\mu _{\varepsilon} ) \bigr] \geq \mathit{E} \bigl[ 1_{B} \mathcal{H} ( \varepsilon ,X_{\varepsilon},\upsilon ) \bigr] \end{aligned}$$

for any ε not in some Lebesgue null set N.

The last inequality is true for all $B\in \mathcal{F}_{\varepsilon ,}$, then for any bounded $\mathcal{F}_{\varepsilon}$-measurable random variable F it holds that

$$\begin{aligned}& E \bigl[ F\mathcal{H} ( \varepsilon ,X_{\varepsilon},\mu _{ \varepsilon } ) \bigr] \geq E \bigl[ F\mathcal{H} ( \varepsilon ,X_{\varepsilon},\upsilon ) \bigr] , \end{aligned}$$

which leads to

$$\begin{aligned}& E \bigl[ \mathcal{H} ( \varepsilon ,X_{\varepsilon},\mu _{ \varepsilon } ) /\mathcal{F}_{\varepsilon} \bigr] \geq E \bigl[ \mathcal{H} ( \varepsilon ,X_{\varepsilon},\upsilon ) / \mathcal{F}_{\varepsilon } \bigr] . \end{aligned}$$

We conclude by using the measurability of the Hamiltonian with respect to $\mathcal{F}_{\varepsilon}$. □

4.3 Example

To illustrate our results, we present an example inspired from [36]. To simplify the notations, we suppose that the problem is one dimensional. Assume that the dynamics is given by

$$ \textstyle\begin{cases} dx_{t}=u(t)\,dB_{{t}},&t\in [ 0,1 ] \\ x_{0}=0 \end{cases} $$

(4.19)

and the cost functional is defined by

$$ J(u)=\frac{1}{2}E \biggl[ { \int _{0}^{1}} \biggl\vert x_{t}^{2}-\frac{1}{2}u_{t}^{2} \biggr\vert \,dt+x(1)^{2} \biggr]. $$

The strict controls are measurable functions from $[ 0,1 ]$ to the set $\{ -1,1 \}$.

By replacing $x_{t}={\int _{0}^{1}} u(s)\,dB_{{s}}$ in the cost functional, we obtain

$$ J(u)=\frac{1}{2}E \biggl[ { \int _{0}^{1}} \biggl( \frac{3}{2}-t \biggr) u(t)^{2}\,dt \biggr]. $$

Since $( \frac{3}{2}-t ) >0$ for any $t\in [ 0,1 ]$, it is clear that $J(u)$ attains its minimum for $\overline{u}(t)=0$, with the state process $\overline{x}(t)=0$. But this is impossible as the strict controls take only the values −1 and 1.

Let us define the relaxed optimal control problem. As the action space $\mathbb{A}= \{ -1,1 \}$, a relaxed control is defined explicitly by

$$ dt.\mu _{t}(da)=dt. \bigl[ \alpha (t)\delta _{1}(da)+ \bigl(1-\alpha (t)\bigr) \delta _{-1}(da) \bigr], $$

where $\alpha (t)$ is a measurable function such that $0\leq \alpha (t)\leq 1$.

The cost functional associated with a relaxed control is then defined by

$$\begin{aligned} J(\mu ) & =E \biggl[ { \int _{0}^{1}} \biggl( \frac{3}{2}-t \biggr) ( { \int _{ \{ -1,1 \} \ }} a \bigl[ \alpha (t)\delta _{1}(da)+ \bigl(1-\alpha (t) \delta _{-1}(da) \bigr] \bigr) ^{2}\,dt \biggr] \\ & =E \biggl[ { \int _{0}^{1}} \biggl( \frac{3}{2}-t \biggr) \bigl( 2\alpha (t)-1 \bigr) ^{2}\,dt \biggr]. \end{aligned}$$

The cost functional attains its minimum at $\alpha (t)=\frac{1}{2}$ and the optimal control is given by

$$\mu =dt \biggl( \frac{1}{2}\delta _{1}(da)+ \frac{1}{2}\delta _{-1}(da) \biggr) . $$

Let us verify that this optimal control satisfies the necessary conditions of Theorem 4.5.

The first- and second-order adjoint processes $(p_{t},q_{t})$ and $(P_{t},Q_{t})$ are the unique adapted solutions of first- and second-order adjoint equations. The unique solutions are $(p_{t},q_{t})=(0,0)$ and $(P_{t},Q_{t})=(2t-4,0)$.

It follows that the generalized Hamiltonian is given by

$$\begin{aligned} \mathcal{H}\bigl(t,X(t),a\bigr) & =\frac{1}{2} \bigl( P(t)+1 \bigr) u^{2}+q(t)u \\ & =\frac{1}{2}(2t-3)u^{2}. \end{aligned}$$

Therefore the generalized Hamiltonian for relaxed controls is defined by

$$\begin{aligned} \mathcal{H}\bigl(t,x^{\ast}(t),\mu \bigr) & =\frac{1}{2}(2t-3) ( { \int _{ \{ -1,1 \} \ }} a \bigl[ \alpha (t)\delta _{1}(da)+ \bigl(1-\alpha (t)\delta _{-1}(da) \bigr] \bigr) ^{2} \\ & =\frac{1}{2}(2t-3) \bigl( 2\alpha (t)-1 \bigr) ^{2}. \end{aligned}$$

$(2t-3)$ being negative for $t\in [ 0,1 ]$, it follows that the generalized Hamiltonian is concave, then attains it maximum at $\alpha (t)=\frac{1}{2}$.

Therefore the relaxed optimal control $dt\mu (da)=dt ( \frac{1}{2}\delta _{1}(da)+\frac{1}{2}\delta _{-1}(da) )$ satisfies the maximum principle.