Kernel Ridge Riesz Representers: Generalization, Mis-specification, and the Counterfactual Effective Dimension

To begin, write

The first term does not depend on \alpha. Kernel boundedness in Assumption 5 implies Bochner integrability [steinwart2008support, Definition A.5.20], which allows us to exchange expectation and RKHS inner product in the second and third terms. By Lemma 1, the definition of counterfactual features, and boundedness of the counterfactual kernel, the second term has

By the reproducing property and boundedness of the actual kernel, the third term is

Take derivatives of the various losses in Definitions 1 and 2. See [de2005risk, Proof of Proposition 2] for technical details of differentiation in the RKHS.

Rearranging Lemma 4 gives \hat{T}\tilde{\alpha}-\hat{\mu}^{(m)}=-\lambda \tilde{\alpha}. Taking the RKHS norm of both sides, \|\hat{T}\tilde{\alpha}-\hat{\mu}^{(m)}\|_H=\lambda \|\tilde{\alpha}\|_H. Finally, write

and \hat{\mu}^{(m)}=\mathbb{E}_n\{\phi^{(m)}(W)\}.

The standard formula for ridge regression is \tilde{\gamma}=(\hat{T}+\lambda)^{-1}\mathbb{E}_n\{Y\phi(W)\}. Comparing to the formula \tilde{\alpha}=(\hat{T}+\lambda)^{-1}\hat{\mu}^{(m)} in Lemma 4, \mathbb{E}_n\{Y\phi(W)\} equals \hat{\mu}^{(m)}=\mathbb{E}\{\phi^{(m)}(W)\} when m(w,f)=yf(w).

I restate the proof of [bruns2023augmented, p. 5] for completeness. Define

so that \tilde{\mathcal{L}}_{\lambda}^n(\beta)=\{\textsc{imbalance}(\beta)\}^2+\lambda n^{-1}\beta^{\top}\beta.

To begin, use the reproducing property to write

Since H is self-dual,

where \Phi^* is the adjoint of the operator \Phi:H\rightarrow \mathbb{R}^n whose ith component is \langle \phi(W_i),\cdot\rangle_H.

Therefore the loss \tilde{\mathcal{L}}_{\lambda}^n(\beta) becomes

Its first order condition yields \tilde{\beta}=(\Phi\Phi^*/n+\lambda)^{-1}\Phi\hat{\mu}^{(m)} whose ith component is \beta_i.

By Lemma 4, \Phi\tilde{\alpha}=\Phi(\Phi^*\Phi/n+\lambda)^{-1}\hat{\mu}^{(m)} whose ith component is \langle \phi(W_i),\tilde{\alpha}\rangle_H=\tilde{\alpha}(W_i). Thus what remains is to show (\Phi\Phi^*/n+\lambda)^{-1}\Phi=\Phi(\Phi^*\Phi/n+\lambda)^{-1}, which follows by expressing \Phi via its singular value decomposition U\Sigma V^*. In particular,

and (\Sigma^2n^{-1}+\lambda)^{-1}\Sigma=\Sigma (\Sigma^2 n^{-1}+\lambda)^{-1} by diagonality of each factor.

By the reproducing property, boundedness of the counterfactual kernel, and the well known expression for kernel ridge regression, write

By the reproducing property, boundedness of the actual kernel, and Lemma 4, write

Finally observe that (\hat{T}+\lambda)^{-1} is self adjoint.

See the proof of Corollary 5.

The argument is based on [chernozhukov2020adversarial, Lemma 2], which instead finds a solution in \mathbb{R}^{2n} for an adversary that tries to violate Riesz representation.

By Definition 2, the definitions of \hat{\mu}^{(m)} and \hat{T}, and boundedness of the kernels,

Due to the ridge penalty, \mathcal{L}^n_{\lambda} is coercive and strongly convex. Hence it has a unique minimizer \tilde{\alpha}. Write \tilde{\alpha}=\tilde{\alpha}_n+\tilde{\alpha}^{\perp}_n where \tilde{\alpha}_n\in row(\Psi) and \tilde{\alpha}_n^{\perp}\in null(\Psi). In words, \tilde{\alpha}^{\perp}_n is orthogonal to the features and counterfactual features evaluated at data. I use this orthogonality and the final expression in the display to conclude that \mathcal{L}^n_{\lambda}(\tilde{\alpha})=\mathcal{L}^n_{\lambda}(\tilde{\alpha}_n)+\lambda \|\tilde{\alpha}_n^{\perp}\|^2_H. Therefore \mathcal{L}^n_{\lambda}(\tilde{\alpha})\geq \mathcal{L}^n_{\lambda}(\tilde{\alpha}_n). Since \tilde{\alpha} is the unique minimizer, \tilde{\alpha}=\tilde{\alpha}_n.

Using Definition 2 and Lemma 5, we have

The first order condition yields

To evaluate the estimator at a test location w, compute \tilde{\alpha}(w)=\langle \tilde{\alpha}, \phi(w)\rangle_H =\rho^{\top}\Psi\phi(w). Finally, note that \Omega=\Psi \Phi^*\Phi \Psi^*, K=\Psi \Psi^*, v=n\Psi\hat{\mu}^{(m)}, and u(w)=\Psi\phi(w).

I combine the nonasymptotic bound in Proposition 5 with the learning theory bounds of Lemma 11, following [sutherland2017fixing, Section 2]. The absolute constant depends only on the quantities (R,\kappa,\kappa^{(m)},\Upsilon,\Sigma,\underline{B}, \bar{B},b,c). Note that (\Upsilon,\Sigma) introduce dependence on (\|\alpha_H\|_H,\bar{M}). Finally appeal to Lemma 8.

By Corollary 3, KRRR nests kernel ridge regression. If |Y|\leq \bar{Y}, then Assumption 1 holds with \bar{M}=\bar{Y}^2. Finally appeal to Theorem 2.

The conclusion is immediate.

Suppose \gamma_{\mathcal{G}}=\gamma_0. Then by the law of iterated expectations,

and \sigma_{\textsc{mis}}^2=var\{\psi(W,\theta_0,\gamma_0,\alpha_{\mathcal{A}})\}.

Suppose instead that \alpha_{\mathcal{A}}=\alpha_0. Then by Riesz representation and the law of iterated expectations,

and \sigma_{\textsc{mis}}^2=var\{\psi(W,\theta_0,\gamma_{\mathcal{G}},\alpha_0)\}.

Finally appeal to Proposition 6.

The result follows from Parseval’s identity.

By Assumption 1, there exists some \bar{M}<\infty such that for all f\in H, \mathbb{E}\{m(W,f)^2\}\leq \bar{M} \mathbb{E}\{f(W)^2\}. By the reproducing property, boundedness of the counterfactual kernel, and the definition of T^{(m)},

By the reproducing property, boundedness of the actual kernel, and the definition of T,

In summary, for all f\in H, \langle f,T^{(m)}f \rangle_H \leq \bar{M} \langle f,Tf \rangle_H i.e. \bar{M} T-T^{(m)} \succeq 0 in the sense of Loewner ordering. Therefore by properties of trace,

The argument for \mathcal{E}_1 immediately precedes [caponnetto2007optimal, eq. 41]. The argument for \mathcal{E}_2 is identical to [caponnetto2007optimal, eq. 43]. I prove a new result for \mathcal{E}_3 appealing to Lemmas 9 and 15, the former of which appears to be new.

Define

By Lemma 4, \mathbb{E}(\xi_i)=(T+\lambda)^{-\frac{1}{2}}\{\mu^{(m)}-T\alpha_H\}=0. Towards an application of Lemma 15, I analyze \|\xi\|_H and \mathbb{E}(\|\xi\|^2_H).

1. To bound \|\xi\|_H, write

   \begin{aligned} \|\xi_i\|_H &\leq\|(T+\lambda)^{-\frac{1}{2}}\|_{op}\|\phi^{(m)}(W_i)-\{\phi(W_i)\otimes \phi(W_i)^*\}\alpha_H\|_H \\ &\leq \lambda^{-1/2}\{\sqrt{\kappa^{(m)}}+\kappa\|\alpha_H\|_H\} \end{aligned}

   i.e. \bar{a}=2\lambda^{-1/2}\{\sqrt{\kappa^{(m)}}+\kappa\|\alpha_H\|_H\}.

2. To bound \mathbb{E}(\|\xi\|^2_H), write

   \xi_i=A-B,\quad A=(T+\lambda)^{-\frac{1}{2}}\phi^{(m)}(W_i),\quad B=(T+\lambda)^{-\frac{1}{2}}\alpha_H(W_i)\phi(W_i)

   where I use the reproducing property to simplify

   \{\phi(W_i)\otimes \phi(W_i)^*\}\alpha_H=\phi(W_i) \langle \phi(W_i),\alpha_H\rangle_H=\alpha_H(W_i)\phi(W_i).

   Hence \|\xi_i\|^2_H\leq 2\|A\|_H^2+2\|B\|_H^2. Focusing on the former term, by properties of trace,

   \begin{aligned} \|A\|_H^2 &=\langle (T+\lambda)^{-\frac{1}{2}}\phi^{(m)}(W_i),(T+\lambda)^{-\frac{1}{2}}\phi^{(m)}(W_i) \rangle_H \\ &=\langle (T+\lambda)^{-1}\phi^{(m)}(W_i),\phi^{(m)}(W_i) \rangle_H \\ &=Tr\{\langle (T+\lambda)^{-1}\phi^{(m)}(W_i),\phi^{(m)}(W_i) \rangle_H\} \\ &=Tr((T+\lambda)^{-1} [\phi^{(m)}(W_i) \otimes \{\phi^{(m)}(W_i)\}^*]). \end{aligned}

   Therefore by linearity of trace, the definition of T^{(m)}, and Lemma 9,

   \mathbb{E}(\|A\|^2_H)=\mathcal{N}^{(m)}(\lambda)\leq \bar{M} \mathcal{N}(\lambda).

   Focusing on the latter term, by properties of trace,

   \begin{aligned} \|B\|_H^2 &=\langle (T+\lambda)^{-\frac{1}{2}}\alpha_H(W_i)\phi(W_i),(T+\lambda)^{-\frac{1}{2}}\alpha_H(W_i)\phi(W_i) \rangle_H \\ &=\langle (T+\lambda)^{-1}\alpha_H(W_i)\phi(W_i),\alpha_H(W_i)\phi(W_i) \rangle_H \\ &=Tr\{\langle (T+\lambda)^{-1}\alpha_H(W_i)\phi(W_i),\alpha_H(W_i)\phi(W_i) \rangle_H\} \\ &=\{\alpha_H(W_i)\}^2 Tr[(T+\lambda)^{-1} \{\phi(W_i) \otimes \phi(W_i)^*\}]. \end{aligned}

   By the reproducing property, Cauchy-Schwartz inequality, and boundedness of the kernel,

   |\alpha_H(W_i)|=\langle \alpha_H,\phi(W_i)\rangle_H \leq \|\alpha_H\|_H\cdot \|\phi(W_i)\|_H\leq \sqrt{\kappa}\|\alpha_H\|_H.

   Therefore by linearity of trace and the definition of T,

   \mathbb{E}(\|B\|_H^2) \leq \kappa \|\alpha_H\|^2_H \mathcal{N}(\lambda).

   In summary,

   \mathbb{E}(\|\xi\|^2_H)\leq 2(\bar{M}+\kappa \|\alpha_H\|^2_H ) \cdot \mathcal{N}(\lambda),

   i.e. \bar{b}=\sqrt{2(\bar{M}+\kappa \|\alpha_H\|^2_H)} \sqrt{\mathcal{N}(\lambda)}.

3. Finally I appeal to Lemma 15. With probability 1-\delta/3,

   \begin{aligned} &\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left\|(T+\lambda)^{-\frac{1}{2}}(\hat{\mu}^{(m)}-\hat{T}\alpha_H)\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right\|_H \leq 2 \ln(6/\delta)\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left(\frac{\bar{a}}{n}+\frac{\bar{b}}{\sqrt{n}}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right) \\ &= 2 \ln(6/\delta)\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left\{\frac{2(\sqrt{\kappa^{(m)}}+\kappa\|\alpha_H\|_H)}{n\sqrt{\lambda}}+\frac{\sqrt{2(\bar{M}+\kappa \|\alpha_H\|^2_H)}\sqrt{\mathcal{N}(\lambda)}}{\sqrt{n}}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right\} \\ &\leq2\ln(6/\delta)\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left\{\frac{1}{n}\sqrt{\Upsilon^2\frac{\kappa'}{\lambda}}+\sqrt{\frac{\Sigma^2\mathcal{N}(\lambda)}{n}}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right\}. \end{aligned}

I follow the structure of [caponnetto2007optimal, Proof of Theorem 4], appealing to Proposition 4 and the union bound.

1. By a decomposition that mirrors [caponnetto2007optimal, eq. 36],

   \|T^{1/2}(\tilde{\alpha}-\alpha_H)\|^2_H\leq 3\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left[\mathcal{A}(\lambda)+\mathcal{S}_1\{\lambda,(W_i)\}+\mathcal{S}_2\{\lambda,(W_i)\}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right].

   The middle and final terms depend on data (W_i), and the work of this proposition is to bound them. They are

   \begin{aligned} \mathcal{S}_1\{\lambda,(W_i)\}&=\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left\|T^{\frac{1}{2}}(\hat{T}+\lambda)^{-1}\{\hat{\mu}^{(m)}-\hat{T}\alpha_H\}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right\|^2_H; \\ \mathcal{S}_2\{\lambda,(W_i)\}&=\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left\|T^{\frac{1}{2}}(\hat{T}+\lambda)^{-1}(T-\hat{T})(\alpha_{\lambda}-\alpha_H)\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right\|^2_H. \end{aligned}

2. Consider the final term. Write

   \mathcal{S}_2\{\lambda,(W_i)\}\leq\|T^{\frac{1}{2}}(\hat{T}+\lambda)^{-1}\|^2_{op}\|(T-\hat{T})(\alpha_{\lambda}-\alpha_H)\|^2_H.

   By [caponnetto2007optimal, eq. 39], under \mathcal{E}_1, \|T^{\frac{1}{2}}(\hat{T}+\lambda)^{-1}\|_{op} \leq \lambda^{-1/2}. The event \mathcal{E}_2 directly guarantees a bound on the other factor. In summary, under \mathcal{E}_1 and \mathcal{E}_2,

   \begin{aligned} \mathcal{S}_2\{\lambda,(W_i)\}& \leq \frac{1}{\lambda}\cdot \{2\ln(6/\delta)\}^2\cdot2\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left[\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left\{\frac{2\kappa\sqrt{\mathcal{B}(\lambda)}}{n}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right\}^2+\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left\{\sqrt{\frac{\kappa\mathcal{A}(\lambda)}{n}}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right\}^2\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right] \\ &=8\ln^2(6/\delta)\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left\{\frac{4\kappa^2\mathcal{B}(\lambda)}{n^2\lambda }+\frac{\kappa\mathcal{A}(\lambda)}{n\lambda}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right\}. \end{aligned}

3. Consider the middle term. Write

   \mathcal{S}_1\{\lambda,(W_i)\}\leq \|T^{\frac{1}{2}}(\hat{T}+\lambda)^{-1}(T+\lambda)^{\frac{1}{2}}\|^2_{op}\|(T+\lambda)^{-\frac{1}{2}}\{\hat{\mu}^{(m)}-\hat{T}\alpha_H\}\|^2_H.

   By [caponnetto2007optimal, eq. 47], under \mathcal{E}_1, \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left\|T^{\frac{1}{2}}(\hat{T}+\lambda)^{-1}(T+\lambda)^{\frac{1}{2}}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right\|_{op} \leq 2. The event \mathcal{E}_3 directly guarantees a bound on the other factor, and it is designed to resemble [caponnetto2007optimal, eq. 48]. Therefore by [caponnetto2007optimal, eq. 49], under \mathcal{E}_1 and \mathcal{E}_3,

   \mathcal{S}_1\{\lambda,(W_i)\}\leq 32 \ln^2(6/\delta)\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left\{\frac{\kappa' \Upsilon^2}{n^2\lambda}+\frac{\Sigma^2 \mathcal{N}(\lambda)}{n}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right\}.

Both \mathcal{A}(\lambda) and \mathcal{B}(\lambda) are standard. See [sutherland2017fixing, Section 1] for \mathcal{N}(\lambda).

Write the difference within the empirical mean as

Averaging over observations in I_{\ell},

Finally, apply Jensen’s inequality to each term.

I proceed in steps.

1. Decomposition. Write

   \begin{aligned} &\bar{\theta}-\theta_{\textsc{mis}} \\ &=\mathbb{E}_n [m(W,\gamma_{\mathcal{G}})+\alpha_{\mathcal{A}}(W)\{Y-\gamma_{\mathcal{G}}(W)\}]-\mathbb{E}[m(W,\gamma_{\mathcal{G}})+\alpha_{\mathcal{A}}(W)\{Y-\gamma_{\mathcal{G}}(W)\}] \\ &=\mathbb{E}_n \{\psi_{\textsc{mis}}(W)\}-\mathbb{E}\{\psi_{\textsc{mis}}(W)\}=\mathbb{E}_n\{\xi(W)\} \end{aligned}

   where \xi(W)=\psi_{\textsc{mis}}(W)-\mathbb{E}\{\psi_{\textsc{mis}}(W)\} is mean zero by construction.

2. Bounding moments. Since observations are independent and identically distributed,

   \begin{aligned} \mathbb{E}(\Delta^2) &=\mathbb{E} ([\mathbb{E}_n\{\xi(W)\}]^2) =\mathbb{E} \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left\{\frac{1}{n^2} \sum_{i,j\in [n]} \xi(W_i)\xi(W_j)\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right\} \\ &=\frac{1}{n^2} \sum_{i,j\in [n]} \mathbb{E} \{\xi(W_i)\xi(W_j)\} =\frac{1}{n^2} \sum_{i\in [n]} \mathbb{E} \{\xi(W_i)^2\} =\frac{1}{n}\mathbb{E} \{\xi(W)^2\}=\frac{1}{n}\sigma_{\textsc{mis}}^2. \end{aligned}

3. Markov inequality implies

   \mathbb{P}(\Delta>t)\leq \frac{\mathbb{E}(\Delta^2)}{t^2}= \frac{\sigma_{\textsc{mis}}^2}{nt^2} =\frac{\epsilon}{2}.

   Solving for t gives the desired result.

I proceed in steps.

1. Decomposition. By Lemma 12 and the triangle inequality, write

   \begin{aligned} |\hat{\theta}-\theta_{\textsc{mis}}| &\leq |\hat{\theta}-\bar{\theta}|+|\bar{\theta}-\theta_{\textsc{mis}}| \leq\frac{1}{L} \sum_{\ell=1}^L \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left| \hat{\theta}_{\ell}-\bar{\theta}_{\ell}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right|+|\bar{\theta}-\theta_{\textsc{mis}}| \leq \frac{1}{L} \sum_{\ell=1}^L \sum_{j=1}^3 \Delta_{j\ell}^{1/2}+\Delta. \end{aligned}

2. Union bound.

   Define the event \mathcal{E}=(\Delta \leq t). By Lemma 14, \mathbb{P}(\mathcal{E}^c)\leq \frac{\epsilon}{2}. Define the events

   \mathcal{E}'_{\ell}=(\Delta_{1\ell}\leq t_1, \Delta_{2\ell}\leq t_2, \Delta_{3\ell}\leq t_3),\quad \mathcal{E}'=\cap_{\ell=1}^L \mathcal{E}'_{\ell},\quad (\mathcal{E}')^c=\cup_{\ell=1}^L (\mathcal{E}_{\ell}')^c.

   Hence by the union bound and Lemma 13,

   \mathbb{P}\{(\mathcal{E}')^c\}\leq \sum_{\ell=1}^L \mathbb{P}\{(\mathcal{E}_{\ell}')^c\} \leq L\frac{\epsilon}{2L}=\frac{\epsilon}{2}.

3. Collecting results. Therefore with probability 1-\epsilon,

   \begin{aligned} &|\hat{\theta}-\theta_{\textsc{mis}}| \leq \frac{1}{L} \sum_{\ell=1}^L \sum_{j=1}^3 \Delta_{j\ell}^{1/2}+\Delta \leq \frac{1}{L} \sum_{\ell=1}^L \sum_{j=1}^3 t_j^{1/2}+t =\sum_{j=1}^3 t_j^{1/2}+t \\ &=\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left\{\frac{6L}{\epsilon}\bar{M}\mathcal{R}(\hat{\gamma}_{\ell})\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right\}^{1/2} +\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left\{\frac{6L}{\epsilon}(\bar{\alpha}')^2\mathcal{R}(\hat{\gamma}_{\ell})\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right\}^{1/2} +\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left\{\frac{6L}{\epsilon}\bar{\sigma}^2\mathcal{R}(\hat{\alpha}_{\ell})\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right\}^{1/2} +\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left(\frac{2}{\epsilon}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right)^{1/2}\frac{\sigma_{\textsc{mis}}}{n^{1/2}} \\ &\leq \mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left(\frac{6L}{\epsilon}\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right)^{1/2}\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\mathopen{}\mathclose\bgroup\left\{ \bar{M}^{1/2}\mathcal{R}(\hat{\gamma}_{\ell})^{1/2} +\bar{\alpha}'\mathcal{R}(\hat{\gamma}_{\ell})^{1/2} +\bar{\sigma}\mathcal{R}(\hat{\alpha}_{\ell})^{1/2} +\frac{\sigma_{\textsc{mis}}}{n^{1/2}} \aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\aftergroup\egroup\right\}. \end{aligned}