Efficient Local Planning with Linear Function Approximation

Dong Yin, Botao Hao, Yasin Abbasi-Yadkori, Nevena Lazić and Csaba Szepesvári

arXiv: 2108.05533v3 · PDF · 5 February 2022
Categories: cs.LG, math.OC, stat.ML
Licence: CC BY 4.0 The site shows the text.

Abstract

We study query and computationally efficient planning algorithms with linear function approximation and a simulator. We assume that the agent only has local access to the simulator, meaning that the agent can only query the simulator at states that have been visited before. This setting is more practical than many prior works on reinforcement learning with a generative model. We propose two algorithms, named confident Monte Carlo least square policy iteration (Confident MC-LSPI) and confident Monte Carlo Politex (Confident MC-Politex) for this setting. Under the assumption that the Q-functions of all policies are linear in known features of the state-action pairs, we show that our algorithms have polynomial query and computational costs in the dimension of the features, the effective planning horizon, and the targeted sub-optimality, while these costs are independent of the size of the state space. One technical contribution of our work is the introduction of a novel proof technique that makes use of a virtual policy iteration algorithm. We use this method to leverage existing results on \ell_\infty-bounded approximate policy iteration to show that our algorithm can learn the optimal policy for the given initial state even only with local access to the simulator. We believe that this technique can be extended to broader settings beyond this work.

Items 17

Definition

Definition 1.1 (Local access to the simulator)

U5GN

We say the agent has local access to the simulator if the agent is allowed to query the simulator with a state that the agent has previously seen paired with an arbitrary action.

Assumption

Assumption 3.1 (Bounded features)

4HU9

We assume that \|\phi(s, a)\|_2 \le 1 for all (s, a)\in{\mathcal{S}} \times \mathcal{A}.

Assumption

Assumption 3.2 (Q_\pi-realizability)

WSJ2

There exists b>0 such that for every policy \pi, there exists a weight vector w_\pi\in\mathbb{R}^d, \|w_\pi\|_2\le b, that ensures Q_\pi(s, a) = w_\pi^\top \phi(s, a) for all (s, a)\in{\mathcal{S}} \times \mathcal{A}.

Assumption

Assumption 3.3 (Approximate Q_\pi-realizability)

4P9E

There exists b>0 and model misspecification error \epsilon>0 such that for every policy \pi, there exists a weight vector w_\pi\in\mathbb{R}^d, \|w_\pi\|_2\le b, that ensures |Q_\pi(s, a) - w_\pi^\top \phi(s, a)| \le \epsilon for all (s, a)\in{\mathcal{S}} \times \mathcal{A}.

Definition

Definition 4.1

P17J

Given \lambda,\tau > 0, and feature matrix \Phi_{\mathcal{C}} \in \mathbb{R}^{|\mathcal{C}|\times d}, the good set \mathcal{H}\subset \mathbb{R}^d is defined as

\mathcal{H} := \{\phi \in \mathbb{R}^d : \phi^\top (\Phi_{\mathcal{C}}^\top \Phi_{\mathcal{C}} + \lambda I)^{-1} \phi \le \tau \}.

Theorem

Theorem 5.1 (Main result for )

59R5

If Assumption 3.2 holds, then for an arbitrarily small \kappa>0, by choosing \tau=1, \lambda = \frac{\kappa^2(1-\gamma)^4}{1024 b^2}, n=\frac{3}{1-\gamma}\log(\frac{4(1+\log(1+\lambda^{-1})d)}{\kappa(1-\gamma)}), K=2+\frac{2}{1-\gamma}\log(\frac{3}{\kappa(1-\gamma)}), m=4096 \frac{d(1+\log(1+\lambda^{-1}))}{\kappa^2(1-\gamma)^{6}}\log(\frac{8Kd(1+\log(1+\lambda^{-1}))}{\delta}), we have with probability at least 1-\delta, the policy \pi_{K-1} that Confident MC-LSPI outputs satisfies

\vspace{-2mm} V^*(\rho) - V_{\pi_{K-1}}(\rho) \le \kappa. \vspace{-2mm}

Moreover, the query and computational costs for the algorithm are \operatorname{poly}(d, \frac{1}{1-\gamma}, \frac{1}{\kappa}, \log(\frac{1}{\delta}), \log(b)) and \operatorname{poly}(d, \frac{1}{1-\gamma}, \frac{1}{\kappa}, |\mathcal{A}|, \log(\frac{1}{\delta}), \log(b)), respectively.

Alternatively, if Assumption 3.3 holds, then by choosing \tau=1, \lambda = \frac{\epsilon^2 d}{b^2}, n=\frac{1}{1-\gamma}\log(\frac{1}{\epsilon(1-\gamma)}), K=2 + \frac{1}{1-\gamma} \log\big(\frac{1}{\epsilon\sqrt{d}}\big), m=\frac{1}{\epsilon^2(1-\gamma)^2}\log(\frac{8Kd(1+\log(1+\lambda^{-1}))}{ \delta}), we have with probability at least 1-\delta, the policy \pi_{K-1} that Confident MC-LSPI outputs satisfies

V^*(\rho) - V_{\pi_{K-1}}(\rho) \le \frac{74\epsilon\sqrt{d}}{(1-\gamma)^2}(1+\log(1+b^2\epsilon^{-2}d^{-1})).

Moreover, the query and computational costs for the algorithm are \operatorname{poly}(d, \frac{1}{1-\gamma}, \frac{1}{\epsilon}, \log(\frac{1}{\delta}), \log(b)) and \operatorname{poly}(d, \frac{1}{1-\gamma}, \frac{1}{\epsilon}, |\mathcal{A}|, \log(\frac{1}{\delta}), \log(b)), respectively.

Theorem

Theorem 5.2 (Main result for Confident MC-Politex)

66B0

If Assumption 3.2 holds, then for an arbitrarily small \kappa>0, by choosing \tau = 1, \alpha=(1-\gamma)\sqrt{\frac{2\log(|\mathcal{A}|)}{K}}, \lambda = \frac{\kappa^2(1-\gamma)^2}{256b^2}, K=\frac{32\log(|\mathcal{A}|)}{\kappa^2(1-\gamma)^4}, n=\frac{1}{1-\gamma}\log(\frac{32\sqrt{d}(1+\log(1+\lambda^{-1}))}{(1-\gamma)^2\kappa}), and m=1024\frac{d (1+\log(1+\lambda^{-1}))}{\kappa^2(1-\gamma)^4}\log(\frac{8Kd(1+\log(1+\lambda^{-1}))}{\delta}), we have with probability at least 1-\delta, the mixture policy \overline\pi_{K} that Confident MC-Politex outputs satisfies

V^*(\rho) - V_{\overline{{\pi}}_K}(\rho) \le \kappa.

Alternatively, if Assumption 3.3 holds, then by choosing \tau=1, \alpha=(1-\gamma)\sqrt{\frac{2\log(|\mathcal{A}|)}{K}}, \lambda = \frac{\epsilon^2 d}{b^2}, K=\frac{2\log(|\mathcal{A}|)}{\epsilon^2 d (1-\gamma)^2}, n = \frac{1}{1-\gamma}\log(\frac{1}{\epsilon(1-\gamma)}), and m=\frac{1}{\epsilon^2(1-\gamma)^2}\log( \frac{8Kd(1+\log(1+\lambda^{-1}))}{\delta}), we have with probability at least 1-\delta, the mixture policy \overline\pi_K that Confident MC-Politex outputs satisfies

V^*(\rho) - V_{\overline{\pi}_K}(\rho) \le \frac{42\epsilon\sqrt{d}}{1-\gamma}(1+\log(1+b^2\epsilon^{-2}b^{-1})).

Proof sketch

Proof sketch (sketch)

P8TZ

We now discuss our proof strategy, focusing on LSPI for simplicity.

Lemma

Lemma 5.1

TSFE

Under Assumption 3.1, the size of the core set \mathcal{C} will not exceed

C_{\max}:=\frac{e}{e-1}\frac{1+\tau}{\tau} d \left(\log(1+\frac{1}{\tau}) + \log(1+\frac{1}{\lambda})\right).

Lemma

Lemma B.1

ZGI0

Let \pi be a policy that we run rollout with in an iteration of the virtual algorithm. Then, for any element z\in\mathcal{C} and any \theta>0, we have with probability at least 1-2\exp(-2\theta^2(1-\gamma)^2m),

\begin{aligned} |z_q - Q_\pi(z_s, z_a)| \le \frac{\gamma^{n+1}}{1-\gamma} + \theta. \end{aligned}

Proof of Lemma B.1

By the definition of Q_\pi(z_s, z_a):

Q_\pi(z_s, z_a) = \mathbb E_{s_{t+1}\sim P(\cdot | s_t, a_t), a_{t+1} \sim \pi(\cdot | s_{t+1})} \left[ \sum_{t=0}^\infty \gamma^t r(s_t, a_t) \mid s_0=z_s, a_0=z_a \right],

and define the n-step truncated Q-function:

Q_\pi^n(z_s, z_a) = \mathbb E_{s_{t+1}\sim P(\cdot | s_t, a_t), a_{t+1} \sim \pi(\cdot | s_{t+1})} \left[ \sum_{t=0}^n \gamma^t r(s_t, a_t) \mid s_0=z_s, a_0=z_a \right].

Then we have |Q_\pi^n(s, a) - Q_\pi(s, a)| \le \frac{\gamma^{n+1}}{1-\gamma}. Moreover, the Q-function estimate z_q is an average of m independent and unbiased estimates of Q_\pi^n(s, a), which are all bounded in [0, 1/(1-\gamma)]. By Hoeffding’s inequality we have with probability at least 1-2\exp(-2\theta^2(1-\gamma)^2m), |z_q - Q_\pi^n(s, a)| \le \theta, which completes the proof.

Lemma

Lemma B.2

XAPK

Suppose that Assumption 3.3 holds. Then, with probability at least

1-2C_{\max}\exp(-2\theta^2(1-\gamma)^2m),

for any (s, a) pair such that \phi(s, a) \in \mathcal{H}, we have

\begin{aligned} |\widetilde{Q}_{k-1}(s, a) - Q_{\widetilde\pi_{k-1}}(s, a)| \le b\sqrt{\lambda \tau} + \big(\epsilon + \frac{\gamma^{n+1}}{1-\gamma} + \theta\big) \sqrt{\tau C_{\max}}+\epsilon:=\eta. \end{aligned}

Lemma

Lemma B.3

RJHL

Suppose that we run K approximate policy iterations and generate a sequence of policies \pi_0, \pi_1, \ldots, \pi_K. Suppose that for every k=1,2,\ldots, K, in the k-th iteration, we obtain a function \widetilde{Q}_{k-1} such that, \|\widetilde{Q}_{k-1} - Q_{\pi_{k-1}}\|_\infty\le \eta, and choose \pi_k to be greedy with respect to \widetilde{Q}_{k-1}. Then

\|Q^* - Q_{\pi_{K}}\|_\infty \le \frac{2\eta}{1-\gamma} + \frac{\gamma^K}{1-\gamma}.

Lemma

Lemma B.4

4ZTO

[singh1994upper] Let \pi be greedy with respect to a function Q. Then for any state s,

V^*(s) - V_\pi(s) \le \frac{2}{1-\gamma}\|Q^*-Q\|_\infty.

Lemma

Lemma B.5

IXUX

In the final loop of the main algorithm, all the rollout trajectories in the virtual algorithm are exactly the same as those in the main algorithm, and therefore w_k = \widetilde{w}_k for all 1\le k\le K.

Proof of Lemma B.5

We notice that since we only consider the final loop, in any iteration, for any state s in all the rollout trajectories in the main algorithm, and all action a\in\mathcal{A}, \phi(s, a)\in\mathcal{H}. In the first iteration, since \pi_0=\widetilde\pi_0, and the simulators are coupled, we know that all the rollout trajectories are the same between the main algorithm and the virtual algorithm, and as a result, all the Q-function estimates are the same, and thus w_1 = \widetilde{w}_1. If we have w_k=\widetilde{w}_k, we know that by the definition in (?), the policies \pi_k and \widetilde{\pi}_k always take the same action given s if for all a\in\mathcal{A}, \phi(s, a)\in\mathcal{H}. Again using the fact that the simulators are coupled, the rollout trajectories by \pi_k and \widetilde\pi_k are also the same between the main algorithm and the virtual algorithm, and thus w_{k+1}=\widetilde{w}_{k+1}.

Lemma

Lemma D.1

4320

Given an initial policy \pi_0 and a sequence of functions Q_k:{\mathcal{S}}\times\mathcal{A}\mapsto[0, (1-\gamma)^{-1}], k=0,\ldots, K-1, construct a sequence of policies \pi_1,\ldots, \pi_{K-1} according to (?) with \alpha=(1-\gamma)\sqrt{\frac{2\log(|\mathcal{A}|)}{K}}, then, for any s\in{\mathcal{S}}, the mixture policy \overline{\pi}_K satisfies

V^*(s) - V_{\overline{\pi}_K}(s) \le \frac{1}{(1-\gamma)^2}\sqrt{\frac{2\log(|\mathcal{A}|)}{K}} + \frac{2\max_{0\le k\le K-1} \|Q_k - Q_{\pi_k}\|_\infty}{1-\gamma}.

Source and licence

The text on this page is from arXiv:2108.05533v3 by Dong Yin, Botao Hao, Yasin Abbasi-Yadkori, Nevena Lazić and Csaba Szepesvári, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.

The record of each item is in analysis/objects/2108/2108.05533.jsonl.