arXiv reader

Technical Report: Adaptive Control for Linearizable Systems Using On-Policy Reinforcement Learning

Tyler Westenbroek, Eric Mazumdar, David Fridovich-Keil, Valmik Prabhu, Claire J. Tomlin and S. Shankar Sastry

arXiv
2004.02766v1 · PDF · 6 April 2020
Categories
cs.LG, math.DS, math.OC, stat.ML
Licence
CC BY 4.0 The site shows the text.

Abstract

This paper proposes a framework for adaptively learning a feedback linearization-based tracking controller for an unknown system using discrete-time model-free policy-gradient parameter update rules. The primary advantage of the scheme over standard model-reference adaptive control techniques is that it does not require the learned inverse model to be invertible at all instances of time. This enables the use of general function approximators to approximate the linearizing controller for the system without having to worry about singularities. However, the discrete-time and stochastic nature of these algorithms precludes the direct application of standard machinery from the adaptive control literature to provide deterministic stability proofs for the system. Nevertheless, we leverage these techniques alongside tools from the stochastic approximation literature to demonstrate that with high probability the tracking and parameter errors concentrate near zero when a certain persistence of excitation condition is satisfied. A simulated example of a double pendulum demonstrates the utility of the proposed theory. 1

Items 12

Lemma

Lemma 1

E9WI

Let the persistence of excitation condition (?) hold and assume that there exists C>0 such that \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\originalleft\lVert W(t)\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\originalright\rVert<C for each t \in \mathbb{R}. Then there exists M>0 and \zeta>0 such that for each t_1, t_2 \in \mathbb{R}

\|\Phi(t_1,t_2)\| \leq Me^{-\zeta(t_1 -t_2)}

with \Phi(t_1,t_2) defined as above.

Uses Assumption 1, Assumption 2
Assumption

Assumption 3

OTVV

The signal y_d(\cdot) is continuous and uniformly bounded. Furthermore, for each j = 1, \dots, q the derivatives \{\dot{y}_{j,d}(\cdot), \ddot{y}_{j,d}(\cdot), \dots , y_{j,d}^{(\gamma_j)}(\cdot)\} are also continuous and uniformly bounded.

Used by Lemma 2
Remark

Remark 1

ZNKM

Typical convergence proofs in the continuous-time adaptive control literature generally only require that (y_{j,d}(\cdot), \dot{y}_{1,d}(\cdot), \dots , y_{j,d}^{\gamma_{j}-1}(\cdot)) be continuous and bounded, but these methods also assume that the input to the plant can be updated continuously. In the sampled data setting, we require the continuity of y_{j,d}^{\gamma_j}(\cdot) to ensure that it does not vary too much within a given sampling period.

Assumption

Assumption 4

V1DZ

There exists a constant C>0 such that \sup_{k\geq0}\| w_k\| < C almost surely.

Used by Lemma 2
Assumption

Assumption 5

91KS

There exists a constant C>0 such \sup_{k \geq 0} \|x_k \| < C and \sup_{k \geq 0} \|\theta_k \| < C almost surely.

Used by Lemma 2
Lemma

Lemma 2

4GIM

Let Assumptions 3-5 hold. Then \Delta \hat{J}_k(\cdot | \theta_k, x_k, e_k) is a sub-Gaussian distribution where

\mathbb{E}[\Delta \hat{J}_k(\cdot | \theta_k, x_k, e_k)] = W_k^TW_k\phi_{k} + O(\Delta t(1 + \sigma + \sigma^2))

and

\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\originalleft\lVert \Delta \hat{J}_k(\cdot | \theta_k, x_k, e_k)\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\originalright\rVert_{\psi_2}= O\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left(\frac{1}{\sigma}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right).
Uses Assumption 1, Assumption 2, Assumption 3, Assumption 4, Assumption 5
Lemma

Lemma 3

0PHH

Let Assumptions 3-5 hold. Then \Delta_{k}^e(\cdot|\theta_k, x_k, e_k) and \Delta_{k}^\phi(\cdot|\theta_k, x_k, e_k) are sub-Gaussian random variables where

\| E[ \Delta_{k}^e(\cdot|\theta_k, x_k, e_k) ]\|_2 =O(\Delta t^2(1 + \sigma+ \sigma^2) ) \vspace{-1.5em}
\| E[ \Delta_{k}^\phi(\cdot|\theta_k, x_k, e_k) ]\|_2 =O(\Delta t^2(1 + \sigma+ \sigma^2)) \vspace{-1em}
\|\Delta_k^e(\cdot |\theta_k, x_k,e_k)\|_{\psi_2} = O(\Delta t \sigma) \vspace{-1em}
\|\Delta_k^\phi(\cdot |\theta_k, x_k,e_k)\|_{\psi_2} = O\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left(\frac{\Delta t }{\sigma}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right).
Uses Assumption 1, Assumption 2
Theorem

Theorem 1

BJUN

Let Assumptions 3-5 hold. Further assume that W^T W is persistently exciting and let M>0 and \zeta>0 be defined as in Lemma 1. Then there exists numerical constants C_1>0 and C_2>0 such that

|\mathbb{E}[X_k]| \leq M\rho^k | X_0| + M C_1\frac{\Delta t(1 + \sigma + \sigma^2)}{\zeta}

and for each \lambda>0 with probability 1-\lambda we have

|X_k - \mathbb{E}[X_k]| \leq C_2 M\sqrt{\frac{ \Delta t \ln\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left( \frac{2}{\lambda}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right)}{\zeta \sigma^2}}
Uses Assumption 1, Assumption 2
Lemma

Lemma 4

LZ8M

3-5 hold. Then there exists a constant C>0 such that

\begin{gathered} \sup_{t\in [t_k, t_{k})}\sup \bigg\{ \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\originalleft\lVert A_p(x(t))\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\originalright\rVert, \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\originalleft\lVert B_p(x(t))\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\originalright\rVert, \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\originalleft\lVert f_p(x(t))\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\originalright\rVert, \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\originalleft\lVert g_p(x(t))\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\originalright\rVert, \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\originalleft\lVert \frac{d}{dx}A_p(x(t))\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\originalright\rVert, \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\originalleft\lVert \frac{d}{dx}B_p(x(t))\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\originalright\rVert, \\ \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\originalleft\lVert \frac{d}{dx}f_p(x(t))\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\originalright\rVert, \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\originalleft\lVert \frac{d}{dx}g_p(x(t))\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\originalright\rVert \bigg \}< C \end{gathered}

and

\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\originalleft\lVert \hat{u}(\theta_k, x_k, e_k)\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\originalright\rVert <C
Lemma

Lemma 5

HP0L

Let Assumptions 3-5 hold. Then there exits C>0 such that for each t \in [t_k, t_{k+1}) we have \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\originalleft\lVert \xi(t) -\xi_k\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\originalright\rVert <C\Delta and \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\originalleft\lVert x(t) - x_k\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\originalright\rVert<C\Delta t and \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\originalleft\lVert \bar{\phi}(t) - \phi_k\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\originalright\rVert C_1 \Delta t..

Source and licence

The text on this page is from arXiv:2004.02766v1 by Tyler Westenbroek, Eric Mazumdar, David Fridovich-Keil, Valmik Prabhu, Claire J. Tomlin and S. Shankar Sastry, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.

The record of each item is in analysis/objects/2004/2004.02766.jsonl.