Provably Efficient Generalized Lagrangian Policy Optimization for Safe Multi-Agent Reinforcement Learning

The proof is based on Lemma 12. By (12),

where \mathbb{I}{\{{\cdot}\}} is the indicator function that is 1 with probability \pi^t(a\,\vert\,x) \widehat d_1^{\,t}(x) and 0 otherwise.

Let \rho_1^t(x,a) \mathrel{\mathop:}= \Vert{ \widehat P_1^{k_1}(\cdot\,\vert\,x,a)- P_1(\cdot\,\vert\,x,a) }\Vert_1. Clearly, \rho_1^t(x,a) \leq 2. Summing \left\|{\widehat q_1^{\,t} - q_1^t}\right\|_1 from t=0 to t=T-1 leads to,

where the layer \ell depends on episode t implicitly. We next apply the martingale concentration and Lemma 1 to the right-hand side of (?).

Let {\mathcal{F}}_1^{t} be an \sigma-algebra that is generated by the state-action sequence, reward/utility functions for the min-player up to episode t. By the definition of epoch k_1\mathrel{\mathop:}= k_1^t, \rho_1^t(x,a) defines over {\mathcal{F}}_{1}^{t-1} only and thus,

Meanwhile, it is easy to see that

which is bounded by 4 since the summands are probability distributions. Hence,
\sum_{x\,\in\,X_\ell}\sum_{a\,\in\,A} \big(\pi^t(a\,\vert\,x) \widehat d_1^{\,t} (x) - \mathbb{I}{\{{(x_\ell,a_\ell)=(x,a)}\}} \big)\rho_1^t(x,a) is a martingale difference sequence that adapts to the filtration \{{\mathcal{F}}_{1}^{t}\}_{t\geq0}. By the Azuma-Hoeffding inequality, with probability 1-\delta/L it holds that

where \delta\in (0,1). By the union bound, (?) holds with probability 1-\delta for all \ell\in\{ 0,1,\ldots,L-1 \}. Thus, with probability 1-\delta, we have

For the rest, we apply Lemma 1. By the definition of epoch k_1\mathrel{\mathop:}= k_1^t, we have N_1^{k_1^t} (x,a) =\sum_{k\,=\,0}^{k_1^t-1} n_1^{k}(x,a). An application of Lemma 24 yields

We note that \sum_{x\,\in\,X_\ell}\sum_{a\,\in\,A} \mathbb{I}{\{{(x_\ell,a_\ell)=(x,a)}\}} \rho_1^t(x,a) = \Vert{ \widehat P_1^{k_1}(\cdot\,\vert\,x_\ell,a_\ell)- P_1(\cdot\,\vert\,x_\ell,a_\ell) }\Vert_1. By Lemma 1, with probability 1-\delta it holds that

By the definition of N_1^{k_1}\mathrel{\mathop:}= N_1^{k_1^t}, using (?) it is convenient to have

Furthermore, we can make the following simplifications. By the Jensen’s inequality,

We also note that \sum_{x\,\in\,X_\ell}\sum_{a\,\in\,A}N_1^{k^T}(x,a)\leq T and \sqrt{|X_{\ell+1}||X_\ell|}\leq\left(|X_{\ell+1}|+|X_\ell|\right)/2. Thus,

Therefore, with probability 1-\delta it holds that

Finally, we take a union of (?) and (?) and substitute it into (?) to conclude the proof.

By Lemma 1, with probability 1-\delta it holds that

We note that the solution (q_1^\star,q_2^\star) in hindsight to Problem (?) satisfies q_1^\star\in\Delta(P_1) and q_2^\star\in \Delta(P_2). Hence, q_1^\star\in \cap_{t \,=\,0}^{T-1} \Delta(k_1^t) and q_2^\star\in \Delta(P_2) \cap_{t \,=\,0}^{T-1} \Delta(k_2^t) with probability 1-\delta. For episode t, we apply Lemma 13 to the primal update (?) with

and x^\star = \widehat q_1^{\,t}, y^\star= \widehat q_2^{\,t}, x' = \widetilde q_1^{\,t-1}, y'=\widetilde q_2^{\,t-1}, x = q_1^\star, and y=q_2^\star. Thus, with probability 1-\delta it holds for any t that

or, equivalently,

Let \Delta^t\mathrel{\mathop:}= \frac{1}{2} \left((\lambda^{t})^2-(\lambda^{t-1})^2\right) be the drift of the consecutive dual updates. Then,

where the first inequality is due to \max^2(x,0)\leq x^2 and we apply \langle{\widehat q_1^{\,t}},{g^{t-1}}\rangle, \langle{\widehat q_2^{\,t}},{h^{t-1}}\rangle, b\in [0,L] in the last inequality. Adding (?) to (?) from both sides of the inequalities without changing the inequality direction yields

However,

where we apply the Hölder’s inequality and Lemma 14 in the first inequality, the second inequality is due to that

and \Vert{ \widetilde q_2^{\,t-1}- \widehat q_2^{\,t-1}}\Vert_1 \leq 2\theta L that can be proved similarly, and the last inequality is due to -bx+ax^2\geq - b^2/(4a) for any a, b > 0. Therefore, we take the lower bound above for the left-hand side of (?),

By Lemma 15,

and, similarly,

We now simplify (?) into

which leads to the desired result by summing it up from t=1 to T,

which leads to the desired result by noting that

Let {\mathcal{F}}^{t} be an \sigma-algebra that is generated by the state-action sequence, reward/utility functions for both players up to episode t. At the beginning, {\mathcal{F}}^0=\{\emptyset,\Omega \}. We have a discrete-time random process \{\lambda^{t},t\geq0 \} that adapts to {\mathcal{F}}^{t}.

It suffices to check all assumptions in Lemma 22.

By the dual update (?),

where the first inequality is clear from two cases for \max(\cdot) and the second inequality is due to \langle{\widehat q_1^{\,t+1}},{g^{t}}\rangle, \langle{\widehat q_2^{\,t+1}},{h^{t}}\rangle\in [0,L], b\in [0,2L]. Consequently,

which leads to \mathbb{E} [ \, \lambda^{t+t_0} - \lambda^{t} \,\vert\,{\mathcal{F}}^t\, ]\leq 2t_0 L. It is convenient to take \delta_{\max} = 2L in Lemma 22.

We next determine the validity of other assumptions in Lemma 22. Let us denote the event in Lemma 1 by \mathcal{E}_{\text{good}} and we have P(\mathcal{E}_{\text{good}}) \geq 1-\delta. We recall that the proof of Lemma 5 remains to be valid if we replace q_1^\star by \bar q_1 and q_2^\star by \bar q_2 starting from (?). By doing so, it is ready to obtain a similar result as (?): under the good event \mathcal{E}_{\text{good}} it holds for any \tau that

or, equivalently,

We note that |\langle{ \bar q_1\cdot \widehat q_2^{\,\tau}-\widehat q_1^{\,\tau}\cdot \bar q_2 },{r^{\tau}}\rangle|\leq L. By summing both sides of (?) from \tau=t+1 to \tau=t+t_0,

where we omit two non-positive terms. Taking the conditional expectation given {\mathcal{F}}^{t} and \mathcal{E}_{\text{good}} yields,

where the second inequality is due to Lemma 15 and the fact: by the law of total expectation, for any \tau\geq t, {\mathcal{F}}^{t}\subset{\mathcal{F}}^\tau and

where the inequality is due to the strict feasibility assumption on (\bar q_1,\bar q_2); the last inequality is due to that

which follows the fact \lambda^\tau\geq \lambda^t - 2(\tau-t)L for any \tau\geq t\geq 0 if we note that |\lambda^{t+1} - \lambda^{t} | \leq 2L. Hence, we can simplify (?) as

where we apply \lambda^t\geq \Theta for the second inequality and we take \Theta in Lemma 22,

Taking the square root and applying the Jensen’s inequality yield

which shows that \mathbb{E} \left[ \lambda^{t+t_0} -\lambda^{t}\,\vert\, {\mathcal{F}}^{t}, \mathcal{E}_{\text{good}} \right] \leq - \frac{1}{2}\xi t_0. Application of law of total expectation to this inequality and (?) with \delta < \frac{1}{12} yields

which verifies the assumption of Lemma 22 if we take \zeta=\xi/4.

We now have verified all assumptions of Lemma 22 with appropriate parameters \Theta, \delta_{\max}, \zeta. For episode t, with probability 1-\delta it holds that

We complete the proof by taking a union bound over t = 1,\cdots,T.

Let Z^t \mathrel{\mathop:}= \sum_{\tau\,=\,0}^{t-1} \lambda^{\tau} \big(\langle{ q_1^\star},{g^{\tau}}\rangle + \langle{ q_2^\star},{h^{\tau}}\rangle -b\big). We note that

where the inequality is because of \mathbb{E} \left[\big(\langle{ q_1^\star},{g^{t-1}}\rangle + \langle{ q_2^\star},{h^{t-1}}\rangle -b\big)\,\vert\,{\mathcal{F}}^{t-1}\right] = \langle{ q_1^\star},{g}\rangle + \langle{ q_2^\star},{h}\rangle -b\leq 0. Hence, \{Z^t,t\geq0 \} a supermartingale.

We also note that \left|Z^{t+1}-Z^{t}\right| = \lambda^{t} \left|\langle{ q_1^\star},{g^{t}}\rangle + \langle{ q_2^\star},{h^{t}}\rangle -b\right| \leq 2\lambda^{t} L. Thus, if |Z^{t+1}-Z^t|> c for some c\in\mathbb{R}^+, then \lambda^t> c/(2L). Let Y^t \mathrel{\mathop:}= \lambda^t - c/(2L). Therefore,

By Lemma 23,

By Lemma 6, with probability 1-\delta it holds for any t that

or, equivalently,

If we take

then (?) becomes

which proves the desired result.

Using the given parameters V, \eta, t_0, and \theta for Lemma 5, {\widehat{\text{\normalfont Regret}}(T)} is upper bounded by \frac{1}{L\sqrt{T}} \sum_{t\,=\,0}^{T-1} \lambda^{t} \big(\langle{ q_1^\star},{g^{t}}\rangle + \langle{ q_2^\star},{h^{t}}\rangle -b\big) + \widetilde O(L\sqrt{T }) with probability 1-\delta. We note that \Theta\leq \widetilde O(L^2\sqrt{T}) and T\geq \max(|X||A|,|B||Y|). Using parameters in Lemma 7, with probability 1-2T\delta,

We complete the proof by noting L\leq |X|+|Y|.

In (?), we have two standard mirror descent problems. Since two problems enjoy the same structure, we only prove an efficient solution to the first problem (?).

By the online mirror descent optimization [zimin2013online], Problem (?) is equivalent to

where \bar q_1^{\,t} is a solution to an unconstrained problem and \widehat q_1^{\,t} simply takes the projection of \bar q_1^{\,t} to the domain \Delta(k_1^t) in the unnormalized Kullback-Leibler divergence.

It is straightforward to compute a closed-form solution for the unconstrained problem,

To compute the projection of \bar q_1^{\,t}, we recall that the domain set \Delta(k_1^t) explicitly takes the following linear constraints on q_1: X\times A\to [0,1],

• (i) q_1(x, a) = \sum_{x'\,\in\,X_{\ell+1}} q_1(x,a,x') for (x,a)\in X_\ell\times A and \ell \in\{0,1,\cdots,L-1\};

• (ii) \sum_{x\in X_\ell}\sum_{a\in A}\sum_{x'\in X_{\ell+1}} q_1(x, a, x') = 1 for \ell \in\{0,1,\cdots,L-1\};

• (iii) \sum_{x\in X_{\ell-1}}\sum_{a\in A} q_1(x, a, x') =\sum_{a\in A}\sum_{x''\in X_{\ell+1}} q_1(x', a, x'') for x'\in X_\ell and \ell \in\{1,\cdots,L-1\};

• (iv) q_1(x,a,x')-\bar P_1^{k_1}(x'\,\vert\,x,a) \sum_{x''\,\in\,X_{\ell+1}}q_1(x,a,x'') \leq \epsilon(x,a,x'),

  \bar P_1^{k_1}(x'\,\vert\,x,a) \sum_{x''\,\in\,X_{\ell+1}}q_1(x,a,x'')-q_1(x,a,x') \leq \epsilon(x,a,x'),

  and \sum_{x'\,\in\,X_{\ell+1}} \epsilon(x,a,x') \leq \epsilon_1^{k_1}(x,a) \sum_{x'\,\in\,X_{\ell+1}}q_1(x,a,x') for (x,a,x')\in X_\ell \times A\times X_{\ell+1} and \ell \in\{0,1,\cdots,L-1\}.

where (ii) and (iii) follow the occupancy measure’s property and (iv) displays the confidence set condition for q_1\in \Delta(k_1^t),

and we also introduce \epsilon: X\times A\times X\to [0,\infty) additionally. Therefore, the projection problem is a convex optimization with the linear constraints. By the method of Lagrange multipliers, we have the following Lagrangian {\mathcal{L}}(q_1,\epsilon; \alpha, \lambda,\beta,\mu^+,\mu^-,\mu),

where \alpha (x,a), \lambda_\ell, \beta(x), \mu^+(x,a,x')\geq0, \mu^-(x,a,x')\geq0, and \mu(x,a,x')\geq0 for (x,a,x')\in X_\ell\times A\times X_{\ell+1} are Lagrange multipliers associated to the linear constraints.

By the Lagrange duality theory, the strong duality holds. To find the optimal solution to the projection problem in (Proof), it suffices to check the first-order stationary conditions. We first take the derivative over \epsilon(x,a,x') for (x,a,x')\in X_\ell \times A\times X_{\ell+1},

which is zero if we take \mu(x,a)=\mu^+(x,a,x')+\mu^-(x,a,x'). Using this stationary condition, we simplify the Lagrangian {\mathcal{L}}(q_1,\epsilon; \alpha, \lambda,\beta,\mu^+,\mu^-,\mu) by eliminating \mu and \epsilon into,

For the notational simplicity, we take \beta(x_0)=\beta(x_L) = 0. We next check the first-order stationary conditions of {\mathcal{L}}(q_1; \alpha, \lambda,\beta,\mu^+,\mu^-) and solve them for the stationary point. We first take the derivative over q_1(x,a,x') and q_1(x,a) for (x,a,x')\in X_\ell \times A\times X_{\ell+1}, respectively,

By setting the second derivative above to be zero, we have \alpha(x,a) = -\ln q_1(x,a)+\ln \bar q_1^{\,t}(x,a). Then, substituting it into the first zero-derivative by eliminating \alpha(x,a) yields,

The solution q_1^\star(x,a) leads to an explicit formula for \widehat q_1^{\,t},

where the last equality is due to (?) and x\neq x_L. We note that it is not unique to determine \alpha(x,a) since it takes the form \alpha^\star(x,a)=-\eta \phi_1^{t-1}(x,a)+\lambda_\ell+B^{t}(x,a,x') for some x'. It remains to determine the optimal \beta, \mu^+, and \mu^-.

Bofore showing the optimal \beta, \mu^+, and \mu^-, we take another derivative over \lambda_\ell at q_1=\widehat q_1^t and set it to be zero,

or, equivalently,

which shows that \lambda_\ell^\star = \ln Z_\ell^t. It also leads to \alpha^\star(x,a)=-\eta \phi_1^{t-1}(x,a)+\lambda_\ell^\star+B^{t}(x,a,x').

We note that

We now collect all previously determined optimal dual variables and apply the strong duality,

where the third equality is due to: \frac{\partial {\mathcal{L}}}{\partial q_1(x,a,x') } \vert_{q_1^\star(x,a,x')}=0 and \frac{\partial {\mathcal{L}}}{\partial q_1(x,a) } \vert_{q_1^\star(x,a)}= 0, and we ignore all constants that are independent of \beta, \mu^+, and \mu^- for the last equality; we note that this minimization problem is a convex optimization problem over the nonnegative orthant. Hence, we have proved the update (11) as an efficient update (?). Similarly, we have an efficient update (11) for the second problem (?) and the proof is complete.

Since two players have the independent transitions, it suffices to just prove one of two players. We next prove (12) for the min-player. By the definitions, we can bound \left\|{\widehat q_1^{\,t} - q_1^t}\right\|_1 by

where we apply the triangle inequality to obtain the inequality. We add and subtract \widehat d_1^{\,t} (x)P_1(\cdot\,\vert\,x,a) into the norm \Vert{ \widehat d_1^{\,t} (x)\widehat P_1^{k_1}(\cdot\,\vert\,x,a)- d_1^t(x) P_1(\cdot\,\vert\,x,a) }\Vert_1, and apply the triangle inequality again,

Therefore,

We can further simplify the upper bound in (?). Using \left\|{P_1(\cdot\,\vert\,x,a) }\right\|_1=1 and \sum_{a\,\in\,A} \pi^t(a\,\vert\,x) =1, we have

By the definitions, \widehat d_1^{\,t}(x) = d_1^t(x) =1 for x\in X_0, and \widehat d_1^{\,t}(x)=\sum_{x^\circ\,\in\,X_{\ell-1}}\sum_{a\,\in\, A}\widehat q_1^{\,t} (x^\circ,a,x) and d_1^t(x)=\sum_{x^\circ\,\in\,X_{\ell-1}}\sum_{a\,\in\, A} q_1^t (x^\circ,a,x) for x\in X_\ell. Thus,

We now return back to (?),

which is a recursive formula for \sum_{\ell\,=\,0}^{j}\sum_{x\,\in\,X_{\ell}}\sum_{a\,\in\, A} \left\|{ \widehat q_1^{\,t} (x,a,\cdot) - q_1^t (x,a,\cdot) }\right\|_1 over j\in\{ 0,1,\ldots,L-1 \}. By the recursion,

Finally, we complete the proof by using (Proof).

For the smooth convex-concave function f, it is necessary to have the first-order stationary condition on (x^\star,y^\star). There exist \nabla_x f(x^\star,y^\star) and \nabla_y(x^\star,y^\star) such that

By the definition of D(\cdot\,\vert\,\cdot),

Application of (Proof) leads to

where the last inequality is due to the convexity fo f(x,y^\star) in x: f(x,y^\star)\geq f(x^\star,y^\star)+\langle\nabla_x f(x^\star,y^\star),x-x^\star\rangle.

Similarly, we work on \eta^{-1} \big(D(y,y') - D(y,y^\star)\big) and (Proof).

Finally, we conclude the proof by adding (?) to (?) from both sides.

We recall q(x,a) and q'(x,a),

where \ell is the layer that x belongs to. We note that q(x,a) and q'(x,a) define probability laws for each \ell\in\{ 0,1,\ldots,L-1 \}, and \sum_{x\,\in\,X_\ell}\sum_{a\,\in\,A}q(x,a) = \sum_{x\,\in\,X_\ell}\sum_{a\,\in\,A}q'(x,a) = 1.

By the definition,

where we apply the Pinsker’s inequality to \sum_{x\,\in\,X_\ell} \sum_{a\,\in\,A} q(x,a) \log\frac{q(x,a)}{q'(x,a)} in the first inequality.

By the definition,

By the Jensen’s inequality,

where the second inequality is due to that a negative entropy is non-positive.

We next prove the second inequality. By the definition,

where the last inequality is due to that a negative entropy is non-positive. We note that -\log (\cdot) is a non-increasing function. We can simplify the upper bound on D(q,\widetilde{q}{\,'}) above by,

By Lemma 1, with probability 1-\delta it holds that

We note that the solution (q_1^\star,q_2^\star) in hindsight to Problem (?) satisfies q_1^\star\in\Delta(P_1) and q_2^\star\in \Delta(P_2). Hence, q_1^\star\in \cap_{t \,=\,0}^{T-1} \Delta(k_1^t) and q_2^\star\in \Delta(P_2) \cap_{t \,=\,0}^{T-1} \Delta(k_2^t) with probability 1-\delta. For episode t, we apply Lemma 13 to the primal update (?) with

and x^\star = \widehat q_1^{\,t}, y^\star= \widehat q_2^{\,t}, x' = \widetilde q_1^{\,t-1}, y'=\widetilde q_2^{\,t-1}, x = q_1^\star, and y=q_2^\star. Thus, with probability 1-\delta it holds for any t that

or, equivalently,

Let \Delta_1^t\mathrel{\mathop:}= \frac{1}{2} \left((\lambda_1^{t})^2-(\lambda_1^{t-1})^2\right) be the drift of the first consecutive dual updates. Then,

where the first inequality is due to \max^2(x,0)\leq x^2 and we apply \langle{\widehat q_1^{\,t}},{g^{t-1}}\rangle \in [0,L], b_1\in [0,L] in the last inequality. Similarly, if we let \Delta_2^t\mathrel{\mathop:}= \frac{1}{2} \left((\lambda_2^{t})^2-(\lambda_2^{t-1})^2\right), then

Adding (?) and (?) to (?) from both sides of the inequalities without changing the inequality direction yields

However,

where we apply the Hölder’s inequality and Lemma 14 in the first inequality, the second inequality is due to that

and \Vert{ \widetilde q_2^{\,t-1}- \widehat q_2^{\,t-1}}\Vert_1 \leq 2\theta L that can be proved similarly, and the last inequality is due to -bx+ax^2\geq - b^2/(4a) for any a, b > 0. Therefore, we take the lower bound above for the left-hand side of (?),

By Lemma 15,

and, similarly,

We now simplify (?) into

which leads to the desired result by summing it up from t=1 to T,

which leads to the desired result by noting that

Let {\mathcal{F}}^{t} be an \sigma-algebra that is generated by the state-action sequence, reward/utility functions for both players up to episode t. At the beginning, {\mathcal{F}}^0=\{\emptyset,\Omega \}. We have a discrete-time random process \{\left\|{\lambda^t}\right\|,t\geq0 \} that adapts to {\mathcal{F}}^{t}. It suffices to check all assumptions in Lemma 22.

By the dual update (?),

where the first inequality is clear from two cases for \max(\cdot) and the second inequality is due to \langle{\widehat q_1^{\,t+1}},{g^{t}}\rangle\in [0,L], b_1\in [0,L]. Similarly, \left| \lambda_2^{t+1} - \lambda_2^{t} \right| \leq L. Hence,

Consequently,

which leads to \mathbb{E} [ \, \left\|{\lambda}\right\|^{t+t_0} - \left\|{\lambda}\right\|^{t} \,\vert\,{\mathcal{F}}^t\, ]\leq 2t_0 L. It is convenient to take \delta_{\max} = 2L in Lemma 22.

We next determine the validity of other assumptions in Lemma 22. Let us denote the event in Lemma 1 by \mathcal{E}_{\text{good}} and we have P(\mathcal{E}_{\text{good}})\geq 1-\delta. Let \Delta^t \mathrel{\mathop:}= \frac{1}{2} \big( \left\|{\lambda^t}\right\|^2-\left\|{\lambda^{t-1}}\right\|^2 \big). Clearly, \Delta^t = \Delta_1^t+\Delta_2^t. We recall that the proof of Lemma 5 remains to be valid if we replace q_1^\star by \bar q_1 and q_2^\star by \bar q_2 starting from (?). By doing so, it is ready to obtain a similar result as (?): under the good event \mathcal{E}_{\text{good}} it holds for any \tau that

or, equivalently,

We note that |\langle{ \bar q_1\cdot \widehat q_2^{\,\tau}-\widehat q_1^{\,\tau}\cdot \bar q_2 },{r^{\tau}}\rangle|\leq L. By summing both sides of (?) from \tau=t+1 to \tau=t+t_0,

where we omit two non-positive terms. Taking the conditional expectation given {\mathcal{F}}^{t} and \mathcal{E}_{\text{good}} yields,

where the second inequality is due to Lemma 15 and the fact: by the law of total expectation, for any \tau\geq t, {\mathcal{F}}^{t}\subset{\mathcal{F}}^\tau and

where the inequality is due to the strict feasibility assumption on (\bar q_1,\bar q_2); the last inequality is due to that

which follows the fact \left\|{\lambda^\tau}\right\| \geq \left\|{\lambda^t}\right\| - 2(\tau-t)L for any \tau\geq t\geq 0 if we note that \left| \left\|{\lambda^{t+1}}\right\| - \left\|{\lambda^{t}}\right\| \right| \leq 2L. Hence, we can simplify (?) as

where we apply \lambda^t\geq \Theta for the second inequality and we take \Theta in Lemma 22,

Taking the square root and applying the Jensen’s inequality yield

which shows that \mathbb{E} \left[ \left\|{\lambda^{t+t_0}}\right\| - \left\|{\lambda^{t}}\right\|\,\vert\, {\mathcal{F}}^{t}, \mathcal{E}_{\text{good}} \right] \leq - \frac{1}{2}\xi t_0. Application of law of total expectation to this inequality and (?) with \delta<\frac{1}{12} yields

which verifies the assumption of Lemma 22 if we take \zeta=\xi/4.

We now have verified all assumptions of Lemma 22 with appropriate parameters \Theta, \delta_{\max}, \zeta. For episode t, with probability 1-\delta it holds that

We complete the proof by taking a union bound over t = 1,\cdots,T.

Let Z^t \mathrel{\mathop:}= \sum_{\tau\,=\,0}^{t-1} \left( \lambda_1^{\tau} \big(\langle{ q_1^\star},{g^{\tau}}\rangle -b_1\big)+\lambda_2^{\tau} \big(\langle{ q_2^\star},{h^{\tau}}\rangle -b_2\big) \right). We note that

where the inequality is because of \mathbb{E} \left[\big(\langle{ q_1^\star},{g^{t-1}}\rangle -b_1\big)\,\vert\,{\mathcal{F}}^{t-1}\right] = \langle{ q_1^\star},{g}\rangle -b_1\leq 0 and
\mathbb{E} \left[\big( \langle{ q_2^\star},{h^{t-1}}\rangle -b_1\big)\,\vert\,{\mathcal{F}}^{t-1}\right] \leq \langle{ q_2^\star},{h}\rangle -b_2 \leq 0. Hence, \{Z^t,t\geq0 \} a supermartingale.

We also note that

Thus, if |Z^{t+1}-Z^t|> c for some c\in\mathbb{R}^+, then \left\|{\lambda^t}\right\|> c/(2L). Let Y^t \mathrel{\mathop:}= \left\|{\lambda^t}\right\| - c/(2L). Therefore,

By Lemma 23,

By Lemma 18, with probability 1-\delta it holds for any t that

or, equivalently,

If we take

then (?) becomes

which proves the desired result.

Using the given parameters V, \eta, t_0, and \theta for Lemma 17, {\widehat{\text{\normalfont Regret}}(T)} is upper bounded by \frac{1}{L\sqrt{T}} \sum_{t\,=\,0}^{T-1} \left( \lambda_1^{t} \big(\langle{ q_1^\star},{g^{t}}\rangle -b_1\big) + \lambda_2^{t} \big( \langle{ q_2^\star},{h^{t}}\rangle -b_2\big) \right) + \widetilde O(L\sqrt{T })

with probability 1-\delta.

We note that \Theta\leq \widetilde O(L^2\sqrt{T}) and T\geq \max(|X||A|,|B||Y|). Using parameters in Lemma 19, with probability 1-2T\delta,

We complete the proof by noting L\leq |X|+|Y|.

By the dual update (?),

We note that \lambda_1^0=\lambda_2^0=0 from the initialization. Summing up both sides of (Proof) from t=1 to t=T leads to

Hence,

We recall \widehat q_1^{\,t}\in \Delta(k_1^t), \widehat q_2^{\,t}\in \Delta(k_2^t) in the primal update (?) and \Delta(k_1^t) and \Delta(k_2^t) in the confidence sets (?). To bound \left\|{\widehat q_1^{\,t} - \widehat q_1^{\,t-1}}\right\|_1 +\left\|{\widehat q_2^{\,t}-\widehat q_2^{\,t-1}}\right\|_1, we consider two cases: (i) k_1^t = k_1^{t-1} and k_2^t = k_2^{t-1}; (ii) either k_1^t \neq k_1^{t-1} or k_2^t \neq k_2^{t-1}.

Case (i). In this case, we have: \widehat q_1^{\,t}, \widehat q_1^{\,t-1}\in \Delta(k_1^t), \widehat q_2^{\,t}, \widehat q_2^{\,t-1}\in \Delta(k_2^t). We begin with the primal update (?) and apply Lemma 13 with,

and x^\star = \widehat q_1^{\,t}, y^\star= \widehat q_2^{\,t}, x' = \widetilde q_1^{\,t-1}, y'=\widetilde q_2^{\,t-1}, x = \widetilde q_1^{\,t-1}, and y=\widetilde q_2^{\,t-1}. Thus,

or, equivalently,

We note that \langle{ (\widetilde q_1^{\,t-1} -\widehat q_1^{\,t} )\cdot \widehat q_2^{\,t-1}},{r^{t-1}}\rangle \leq \Vert (\widetilde q_1^{\,t-1} -\widehat q_1^{\,t} )\cdot \widehat q_2^{\,t-1}\Vert_1\Vert{r^{t-1}}\Vert_\infty\leq \Vert\widetilde q_1^{\,t-1} -\widehat q_1^{\,t} \Vert_1, and, similarly, \langle\widehat q_1^{\,t-1}\cdot (\widehat q_2^{\,t} - \widetilde q_2^{\,t-1} ),{r^{t-1}}\rangle\leq\Vert\widehat q_2^{\,t} - \widetilde q_2^{\,t-1}\Vert_1. Thus, we can reduce (?) into

where the left-hand side can be lower bounded by Lemma 14,

Then, we apply the inequality (x+y)^2\leq2(x^2+y^2) and cancel a non-negative term to obtain

By the definition of \widetilde q_1^{\,t-1} and \widetilde q_2^{\,t-1},

Similarly, we have \Vert{ \widetilde q_2^{\,t-1}-\widehat q_2^{\,t} }\Vert_1 \leq (1-\theta) \Vert{\widehat {q}_2^{\,t-1}-\widehat q_2^{\,t}}\Vert_1 -2\theta L. Thus, we can further reduce (?) into

Case (ii). In this case, either \widehat q_1^{\,t}, \widehat q_1^{\,t-1} or \widehat q_2^{\,t}, \widehat q_2^{\,t-1} might not have the same domain. For instance, when k_1^t>k_1^{t-1}, it is possible that \Delta(k_1^{t}) becomes different from \Delta(k_1^{t-1}). We note that k_1^t>k_1^{t-1} only happens when episode t is the first one that belongs to epoch k_1^t. By Lemma 25, k_1^T \leq \sqrt{ T|X||A|}\log (8T/(|X||A|)) and k_2^T \leq \sqrt{ T|Y||B|}\log (8T/(|Y||B|)) if we are given T\geq \max(|X||A|, |Y||B|).

We now combine two cases above for (Proof),

where the first inequality is due to: \left\|{\widehat q_1^{\,t} - \widehat q_1^{\,t-1}}\right\|_1 \leq 2L and \left\|{\widehat q_2^{\,t}-\widehat q_2^{\,t-1}}\right\|_1\leq 2L, and we apply (?) from the case (i) for the last inequality. Using the bounds on k_1^T, k_2^T in the case (ii), we conclude the desired bound for (Proof),

We complete the proof by noting \lambda_1^0=\lambda_2^0=0, V=L\sqrt{T}, \eta = 1/(TL), and \theta = 1/T.