Linear-Quadratic Regulators

• But where should I place my poles?
• Choosing Costs
• But how can we actually solve this?

But where should I place my poles?

In my previous post about controllability, I mentioned that state feedback allows us to place our poles arbitrarily in theory. However, in practice we are often limited by how powerful our control input is. Therefore, the question naturally arises as to how we can manage this tradeoff to appropriately choose where to place the closed-loop poles for a system. A somewhat natural approach could be to define and minimise a cost function that encompasses how our state deviates from the steady state (at 0 wihtout a reference) and how much the control input deviates from 0 as well. A Linear-Quadratric Regulator (LQR) is designed to do exactly this by choosing the optimal linear state feedback regulator $\mathbf{K}$ (so that $\mathbf{u = -Kx}$) which minimises the quadratic loss function

\[J(\mathbf{u}) = \int_0^T \left( \mathbf{x^T Q x + u^T R u} \right) dt + \mathbf{x^T Q_f x}\]

or for discrete systems

\[J(\mathbf{u}) = \sum_{k=0}^N \left( \mathbf{x}[k]^T \mathbf{Q x}[k] + \mathbf{u}[k]^T \mathbf{R u}[k] \right) + \mathbf{x}[N]^T \mathbf{Q_f x}[N]\]

Commonly, the problem is solved for an infinite time horizon meaning $T$ or $N$ are taken to infinity.

Choosing Costs

Haven’t we just moved the problem of tuning $n$ poles to even more tunable parameters in these cost matrices? How is this better?

Well, tuning poles directly is difficult as it is hard to anticipate exactly what moving each one will change in the system response. By contrast, the cost matrices $\mathbf{Q,\, R}$ define quadratic costs for state and control effort over time, respectively which often reflect the design requirements for a system much more clearly. For example, if the system is powered by a battery and the input magnitude is proportional to the power drawn from it then $\mathbf{Q}$ would define a cost for the energy expended in order to regulate the system’s state to 0. If recharging/replacing the battery was difficult or costly and regulating the state could be allowed to happen slowly then it would be appropriate to assign costs relatively low for $\mathbf{Q}$ compared to $\mathbf{R}$. On the other hand, if energy was not an important consideration and power could be drawn at a very high rate then it might be appropriate to set the cost for $\mathbf{Q}$ relatively higher than for $\mathbf{R}$.

It should also be obvious that for effective regulation $\mathbf{R} > 0$ since if it were not then minimising the cost would result in an unbounded control input. Similarly, $\mathbf{Q} \geq 0$ and $\mathbf{Q_f} > 0$ so that the system actually regulates to zero (or the desired set point). Additionally, all of the cost matrices must be positive (semi-) definite. As a result, typically these cost matrices are chosen to be diagonal to make them more intuitive to tune. This way, each element only affects the cost of one input or state and is therefore easier to reason about and adjust as necessary. For example, if the first input $u_1$ is spiking higher than physically possible leading to ineffective control, decreasing $r_{(1, 1)}$ may help by allowing it to expend more energy over a longer period of time. (In reality it is a little more complex than this since lowering this cost may also increase the relative cost of one of the states it is responsible for controlling thereby requiring an increased input regardless but it is still easier to reason about than placing poles directly).

But how can we actually solve this?

Surprisingly, it turns out the solution can be found by solving what is known as an Algebraic Ricatti Equation (ARE) which is, as the name suggests, a linear algebra problem. You can read about the derivation here for the continuous case. The result is that for an infinite time horizon (as is usualkly considered) where the control law is $\mathbf{u = -Kx}

\[\mathbf{K = R^{-1} B^T P}\]

where $\mathbf{P}$ is the solution to the ARE

\[0 = \mathbf{A^T P + PA - PBR^{-1}B^TP + Q}\]

and for discrete cases,

\[\mathbf{K = (R + B^T P B)^{-1} B^T PA}\]

where $\mathbf{P}$ is the solution to

\[\mathbf{P = Q + A^T P A - A^T P B(R + B^T P B)^{-1} B^T P A}\]

Practically, this can be easily solved with software using MATLAB or Python by just providing the system matrices $\mathbf{A,\, B}$ and cost matrices $\mathbf{Q,\, R}$. Sometimes, the cost function is extended to include a third term with cost $2\mathbf{x^T N u}$ to allow for cross weights but often this is not needed.

For more details you can have a look at these slides from an older control course from my uni.