LQR Optimization problem

In general, a control optimization problem consists of a cost function, \(\mathcal{J}\), and a dynamical constraint, \(f_k\), that describes how the state evolves over time,

\[\min \mathcal{J}(x, u) = \ell_f(x_T) + \sum_{t=0}^{T-1} \ell_t(x_t, u_t)\]

subject to

\[ \begin{align}\begin{aligned}x_{t+1} = f_t(x_t, u_t), \quad t = 0, \ldots, T-1\\x_0 = x_{\text{init}}\end{aligned}\end{align} \]

where \(\ell_k(\cdot, \cdot)\) is the momentary cost function at time \(t\) for a given input \(u_t \in \mathbb{R}^m\) and state \(x_t \in \mathbb{R}^n\). The goal is to find an optimal state feedback law \(u_t^*(x_t)\), that minimizes the cost function \(\mathcal{J}\) given the control sequence \(u^* = \{u_0^*, \ldots, u_{T-1}^*\}\).

Some optimal control problems can be solved using the Dynamic Programming (DP) approach. This method consists of defining a Value function, \(\mathcal{V}_t(x, t)\),

\[\mathcal{V}_t(x, t) = \min_{u_t \in \mathcal{U}} \{ \ell_t(x_t, u_t) + \mathcal{V}_{t+1}(x, t+1) \}\]

This is the optimal (minimal) cost-to-go from time \(t\) to the terminal time \(T\). In order to initialize the Value function, the terminal boundary condition is defined as \(\mathcal{V}_T(x_T) = \ell_T(x_T)\). Therefore, we can dynamically sweep back through time.

Using the Value function, we can find the optimal control law by finding the argmin of the Value function,

\[u_t(x_t) = \arg\min_{u_t \in \mathcal{U}} \{ \ell_t(x_t, u_t) + \mathcal{V}_{t+1}(x, t+1) \}\]

and successively compute the optimal trajectory \(\{x_s^*:T\}\) in a single forward pass,

\[x_{t+1}^* = f_t(x_t^*, u_t^*(x_t^*)) = f_t(x_t^*).\]

Linear quadratic program

Linear Quadratic Regulators (LQR) are a particular class of optimal control problems that are well-structured and can be solved efficiently using quadratic programming. For the LQR problem, the \(\ell_k\) term in Equation (2.1) is defined as

\[\begin{split}\ell_t(x_t, u_t) = \begin{bmatrix} x_t \\ u_t \end{bmatrix}^T \begin{bmatrix} Q_t & S_t \\ S_t^T & R_t \end{bmatrix} \begin{bmatrix} x_t \\ u_t \end{bmatrix} + \begin{bmatrix} x_t \\ u_t \end{bmatrix}^T \begin{bmatrix} q_t \\ r_t \end{bmatrix}\end{split}\]

and the terminal cost term,

\[\ell_T(x_T) = x_T^T Q_T x_T + x_T^T q_T.\]

References

[1]

Marine Schimel, Ta-Chu Kao, Kristopher T Jensen, and Guillaume Hennequin. ILQR-VAE : control-based learning of input-driven dynamics with applications to neural data. In International Conference on Learning Representations. 2022. URL: https://openreview.net/forum?id=wRODLDHaAiW.

[2]

Brandon Amos and J. Zico Kolter. Optnet: differentiable optimization as a layer in neural networks. 2017. URL: https://arxiv.org/abs/1703.00443, doi:10.48550/ARXIV.1703.00443.

[3]

Weiwei Li and Emanuel Todorov. Iterative linear quadratic regulator design for nonlinear biological movement systems. In Proceedings of the 1st International Conference on Informatics in Control, Automation and Robotics, 222–229. 2004.

[4]

Donald E. Kirk. Optimal Control Theory: An Introduction. Dover, 2004.

[5]

Thomas Soares Mullen, Marine Schimel, Guillaume Hennequin, Christian K. Machens, Michael Orger, and Adrien Jouary. Learning interpretable control inputs and dynamics underlying animal locomotion. In The Twelfth International Conference on Learning Representations. 2024. URL: https://openreview.net/forum?id=MFCjgEOLJT.

[6]

Thomas Soares Mullen. Fish behaviour simulator. June 2024. URL: https://doi.org/10.5281/zenodo.11406468, doi:10.5281/zenodo.11406468.

[7]

Simo Sarkka and Angel F. Garcia-Fernandez. Temporal parallelization of dynamic programming and linear quadratic control. IEEE Transactions on Automatic Control, 68(2):851–866, February 2023. URL: http://dx.doi.org/10.1109/TAC.2022.3147017, doi:10.1109/tac.2022.3147017.

[8]

Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, October 2012. URL: http://dx.doi.org/10.1109/IROS.2012.6386025, doi:10.1109/iros.2012.6386025.