Optimal control theory tackles finding control functions that optimize performance indices in dynamic systems. It's a powerful tool for solving complex problems in engineering, economics, and other fields where decisions must be made over time.

The Pontryagin maximum principle and Hamilton-Jacobi-Bellman equation are key methods in optimal control. They provide necessary and sufficient conditions for optimality, helping us find the best control strategies for various systems and objectives.

Optimal Control Theory

Formulation of optimal control problems

• Optimal control problems involve finding a control function that optimizes a performance index
  • Performance index is a functional that depends on the state and control variables (cost function, objective function)
• Variational problems involve finding a function that optimizes a functional
  • Functional is a mapping from a space of functions to real numbers (energy functional, action functional)
• To formulate an optimal control problem as a variational problem:
  • Define the state variables (position, velocity) and control variables (force, acceleration)
  • Specify the system dynamics that describe how the state variables evolve over time (equations of motion, state equations)
  • Define the performance index to be optimized, which is a functional of the state and control variables (minimum time, minimum energy)
  • Specify the initial and final conditions (initial state, target state), as well as any constraints on state and control variables (bounds on control inputs, state constraints)

Application of Pontryagin maximum principle

• Pontryagin maximum principle provides necessary conditions for optimality in optimal control problems
• Steps to apply the Pontryagin maximum principle:
  1. Define the Hamiltonian function: $H(x, u, \lambda, t) = L(x, u, t) + \lambda^T f(x, u, t)$
     • $L(x, u, t)$ is the integrand of the performance index (Lagrangian, running cost)
     • $f(x, u, t)$ represents the system dynamics (state equations, equations of motion)
     • $\lambda$ is the costate vector, also known as adjoint variables or Lagrange multipliers
  2. Derive the optimal control by maximizing the Hamiltonian with respect to the control variable $u$ (first-order necessary condition)
  3. Obtain the state and costate equations:
     • State equation: $\dot{x} = \frac{\partial H}{\partial \lambda}$ describes the evolution of the state variables
     • Costate equation: $\dot{\lambda} = -\frac{\partial H}{\partial x}$ describes the evolution of the costate variables
  4. Solve the state and costate equations with the optimal control, initial conditions, and transversality conditions (boundary conditions, final state constraints)

Dynamic Programming and the Hamilton-Jacobi-Bellman Equation

Hamilton-Jacobi-Bellman equation in practice

• The Hamilton-Jacobi-Bellman (HJB) equation is a partial differential equation that provides a sufficient condition for optimality in optimal control problems
• The HJB equation is based on the principle of optimality and the concept of the value function
  • Value function $V(x, t)$ represents the optimal cost-to-go from state $x$ at time $t$ (optimal value, optimal cost)
• The HJB equation is given by:
  • $-\frac{\partial V}{\partial t} = \min_{u} {L(x, u, t) + \frac{\partial V}{\partial x}^T f(x, u, t)}$
  • $L(x, u, t)$ is the running cost, which is the integrand of the performance index (Lagrangian, instantaneous cost)
  • $f(x, u, t)$ represents the system dynamics (state equations, equations of motion)
• Solving the HJB equation yields the optimal value function and the optimal control policy (feedback control law, state feedback)

Optimal control vs dynamic programming

• Dynamic programming is a solution approach for optimal control problems that breaks down the problem into smaller subproblems (principle of optimality, Bellman equation)
• The principle of optimality states that an optimal policy has the property that whatever the initial state and initial decisions are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decisions
• The value function in dynamic programming satisfies the HJB equation
  • The HJB equation can be derived using the principle of optimality and the dynamic programming approach (Bellman equation, dynamic programming recursion)
• The optimal control can be obtained from the value function by minimizing the right-hand side of the HJB equation with respect to the control variable (argmin, optimal policy)
• Solving the HJB equation using dynamic programming involves discretizing the state space and time, and then solving the resulting discrete-time optimal control problem using backward induction (value iteration, policy iteration)