arXiv reader

Source-Condition Analysis of Kernel Adversarial Estimators

Antonio Olivas-Martinez and Andrea Rotnitzky

arXiv
2508.17181v1 · PDF · 24 August 2025
Categories
math.ST, stat.TH
Licence
CC BY 4.0 The site shows the text.

Abstract

In many applications, the target parameter depends on a nuisance function defined by a conditional moment restriction, whose estimation often leads to an ill-posed inverse problem. Classical approaches, such as sieve-based GMM, approximate the restriction using a fixed set of test functions and may fail to capture important aspects of the solution. Adversarial estimators address this limitation by framing estimation as a game between an estimator and an adaptive critic. We study the class of Regularized Adversarial Stabilized (RAS) estimators that employ reproducing kernel Hilbert spaces (RKHSs) for both estimation and testing, with regularization via the RKHS norm. Our first contribution is a novel analysis that establishes finite-sample bounds for both the weak error and the root mean squared error (RMSE) of these estimators under interpretable source conditions, in contrast to existing results. Our second contribution is a detailed comparison of the assumptions underlying this RKHS-norm-regularized approach with those required for (i) RAS estimators using \mathcal{L}^2 penalties, and (ii) recently proposed, computationally stable Kernel Maximal Moment estimators.

Items 56

Example

Example 1 (Proximal Inference for Generalized Average Causal Effects)

RJXE

Suppose that O=(L,A,Z',W',Y), where A is a discrete or continuous treatment, Y an outcome, and L a vector of observed baseline covariates, and Z' and W' are correlates of an unmeasured confounder U, referred to as negative control treatments and negative control outcomes, respectively, satisfying Z' \perp Y \mid A,L,U and (A,Z') \perp W' \mid L,U. For each a\in\mathcal{A}, let Y(a) denote the potential outcome under treatment level a and Y ( \cdot):= \left\{Y(a):a \in \mathcal{A} \right\}. Let O_{\text{full}}=(L,A,Z',W',Y, Y(\cdot)). Let P_0 and P_{\text{full},0} denote the unknown distributions of the factual random vector O and the factual/counterfactual vector O_{\text{full}}. Suppose the goal is to estimate a parameter \gamma(P_{\text{full},0}) := \mathbb{E}_{P_{\text{full},0}}\left\{ Y(A)\pi(A,L)\right\} where \pi:\mathcal{A}\times\mathcal{L}\mapsto\mathbb{R} is a given specified weighting function. In particular, when A is binary and \pi(a,l) = 2a - 1, \gamma(P_{\text{full},0}) is the average treatment effect (ATE), and when \pi(a|l)=a, \gamma(P_{\text{full},0})=\mathbb{E}_{P_{full,0}}\{Y(1)\}. Under the consistency assumption Y(A)=Y and additional regularity conditions which essentially ensure that Z' and W' are strong proxies of the unmeasured confounder U, it can be shown (see [miao2018identifying, tchetgen2020introduction, kallus2021causal, ghassami2022minimax, cui2024semiparametric]) that \gamma(P_{\text{full},0}) is identified and equal to \theta(P_0) of the form (?), with W=(L,A,W'), Z=(L,A,Z'), r(o) = 1, \tilde{m}(o;h) =\pi(a,l) \cdot h(w), m(o;g) = y \cdot g(z) and corresponding Riesz representers \tilde\rho_{P_0}(w) = \pi(a,l)/p_{0,A|L,W'}(a|l,w') and \rho_{P_0}(z) = \mathbb{E}_{P_0}\left(Y \mid Z=z\right), where p_{0,A|L,W'} denotes the conditional density or probability mass function of A given (L,W'). In this example X:=(A,L). Note that while \rho_{P_0} does not have structural zeros, \tilde\rho_{P_0} will have exactly the same structural zeros as the weight function \pi(a,l). For instance, if A is binary and \pi(a,l)=a, then \tilde\rho_{P_0}(w)=0 for w=(a=0,l,w').

Example

Example 2 (Proximal Inference for Policy Shift Interventions)

N8C1

Suppose that under the same assumptions as in Example 1, with A being continuous, we are interested in estimating the parameter \gamma(P_{\text{full},0}) := \mathbb{E}_{P_{\text{full},0}}\left[ Y\{q(A)\}\right] where q:\mathcal{A} \rightarrow \mathcal{A} is a known function of the actually received treatment [haneuse2013estimation, diaz2023nonparametric]. For instance, q may take the form q(a)=\left(a+\delta\right) I\{a\in[c,d-\delta-\varepsilon)\} + \left\{a+\delta(d-a)/(\delta+\varepsilon)\right\} I\{a\in [d-\delta-\varepsilon, d]\} where [c,d] is the support of A and \delta and \varepsilon are fixed pre-specified constants such that d-\delta-\varepsilon >c. [olivas2025chap3] showed that under certain regularity conditions which include the requirement that q be invertible and differentiable almost everywhere, \theta(P_{full,0}) is identified and admits the representation (?), with r(o) = 1, \tilde{m}(o;h) =h\{q(a),l,w'\}, m(o;g) = y \cdot g(z) and corresponding Riesz representers \rho_{P_0} as in Example 1—since the map m(o;g) is the same as in that example—and \tilde\rho_{P_0}(w) = I\{a\in[c+\delta,d]\}\cdot\frac{dq^{-1}(a)}{da}\cdot\frac{p_{0,A|L,W}\{q^{-1}(a)\mid l,w'\}}{p_{0,A|L,W}(a|l,w')}. Recalling that in this problem X:=(L,A), we see that \tilde\rho_{P_0} is zero outside the set \{(l,a):a\in[c+\delta,d]\}, so this last set plays the role of \mathcal{X}_0 for estimation of the solution of equation (?).

Used by Proof
Example

Example 3 (Missing Not At Random Data with Shadow Variables)

9GZR

Suppose that O=(\Delta,\Delta Y,X,Z'), where \Delta is binary, Y is an outcome observed if and only if \Delta = 1, X is a vector of covariates, and Z' is a variable informative of Y, referred to as shadow variable, satisfying Z'\perp \Delta\mid X,Y and Z'\not\perp Y\mid X,\Delta=1. Let O_{full}=(\Delta, Y,X,Z) denote the full data vector. Let P_0 and P_{\text{full},0} denote the unknown distributions of the observed random vector O and the full data vector O_{\text{full}}. Suppose the missingness mechanism is not at random and that the goal is to estimate a parameter \gamma(P_{\text{full},0}) := \mathbb{E}_{P_{\text{full},0}}\left\{ (1-\Delta)\pi(X, Y) \right\} where \pi:\mathcal{X}\times\mathcal{Y}\mapsto\mathbb{R} is a given function. In particular, when \pi(X, Y) = Y, \gamma(P_{\text{full},0})/P_0(\Delta=0) is the mean outcome among individuals with missing outcomes. Under certain structural assumptions on \pi,—namely, that \pi(X,Y) depends only on features of Y that are indirectly captured through Z' given X—it can be shown [miao2016varieties, li2023non, miao2024identification] that \gamma(P_{\text{full},0}) is identified and equals \theta(P_0) of the form (?), with W=(X,\Delta Y), Z=(X,Z'), r(o)=\delta, \widetilde{m}(o;h)=\delta\pi(x,\delta y)h(x,\delta y), m(o;g)=(1-\delta)g(x,z) and corresponding Riesz representers \tilde\rho_{P_0}(x,\delta y)=\pi(x,\delta y)\mathbb{E}(\Delta\mid X=x,\Delta Y=\delta y) and \rho_{P_0}(x,z)=\mathbb{E}(1-\Delta\mid X=x,Z=z).

Theorem

Theorem 1

S25R

Suppose that h_{0,P_0} and g_{0,P_0} solve equations (?) and (?), respectively, at P=P_0. Suppose that \int\{\phi(o;\hat {h}^{(-k)},\hat{g}^{(-k)}) - \phi(o; h_{0,P_0},g_{0,P_0})\}^2 dP_0(o) = o_P(1) holds for each k \in \{1,...,K\}. Then, \sqrt{n}\{ \hat\theta-\theta(P_0)\} =\sqrt{n}\mathbb{E}_{n} \left[ \phi (O;h_{0,P_0},g_{0,P_0})-\theta(P_0) \right] +\sqrt{n}\:R_{n}+o_{P}(1), where

R_n:=\sum_{k=1}^K\frac{n_k}{n} \:\int r(o) \left\{\hat h^{(-k)}(w)-h_{0,P_0}(w)\right\}\left\{g_{0,P_0}(z)-\hat g^{(-k)}(z)\right\}dP_0(o),

with n_k denoting the size of the k^{th} subsample. In particular, if \mathbb{E}_{P_0}\{\phi(O;h_{0,P_0},g_{0,P_0})^2\}<\infty and R_n=o_p(n^{-1/2}), then \sqrt{n}\{ \hat\theta-\theta(P_0)\}\stackrel{\text{d}}{\longrightarrow}\mathcal{N}(0,\tau_0^2), where \tau_0 ^{2}=\mathbb{E}_{P_0}[ \{ \phi (O;h_{0,P_0},g_{0,P_0})-\theta(P_0)\} ^{2}].

Proposition

Proposition 1

Z21U

Suppose \{g_j=I_0g_j^\dag:g_j^\dag \in \mathcal{G},j\in\mathbb{J}\} is an \mathcal{L}^2(P_Z)–orthonormal basis of \overline{I_0\mathcal{G}}. Let \mathcal{P}_{\overline{I_0\mathcal{G}}}:\mathcal{L}^2(P_Z) \rightarrow \overline{I_0\mathcal{G}} denote the projection operator. Then Q_{\mathbb{J}}(h)=\Vert\mathcal{P}_{\overline{I_0\mathcal{G}}} \mathcal{T} I_0(h_0^\dag-h)\Vert_2^2.

Used by Proof
Proof of Proposition 1

If \{g_j=I_0 g_j^\dag : g_j^\dag \in \mathcal{G},\: j\in\mathbb{J}\} is an orthonormal basis of \overline{I_0 \mathcal{G}}, then \Vert \mathcal{P}_{\overline{I_0\mathcal{G}}}\circ \mathcal{T}I_0(h_0^\dag-h)\Vert_2^2=\sum_{j\in\mathbb{J}} \langle I_0\mathcal{T}(h_{0}^\dag-h)\: ,\: I_0 g_j^\dag \rangle_2^2 =\sum_{j\in\mathbb{J}} \{\mathbb{E} [ \rho(Z)g_j^\dag(Z)- I_0(X)r(O)h(W)\: g_j^\dag(Z) ]\}^2 =\sum_{j\in\mathbb{J}}\{\mathbb{E} [ m(O,g_j^\dag)- I_0(X)r(O)h(W)\: g_j^\dag(Z) ]\}^2=Q_\mathbb{J}(h)

Proposition

Proposition 2

RDA9

For any constant c \ne0, any h_0^\dag such that h_0=I_0h_0^\dag is a solution of equation (?), and any h\in \mathcal{L}^2(P_{W}), it holds that:

  1. If \mathcal{G}\subset\mathcal{L}^2(P_Z) such that (2c^2)^{-1}\mathcal{T}(h_0^\dag-h)\in\mathcal{G} then M_P(h) =\Vert I_0\mathcal{T}(h_0^\dag-h)\Vert_2^2.

  2. Suppose \mathcal{G} is a subspace of \mathcal{L}^2(P_{Z}). Let I_0\mathcal{G}:=\left\{I_0g\::\:g\in\mathcal{G}\right\} and let \overline{I_0\mathcal{G}} denote its closure in \mathcal{L}^2(P_Z). Let \{g_j=I_0g_j^\dag:g_j^\dag \in \mathcal{G},\:j\in\mathbb{J}\} denote an \mathcal{L}^2(P_Z)–orthonormal basis of \overline{I_0\mathcal{G}} where \mathbb{J}\subseteq\mathbb{N}. If \mathcal{P}_{\overline{I_0\mathcal{G}}}\circ\mathcal{T}_{P}I_0(h_0^\dag-h)\in\ I_0\:\mathcal{G}, then Q_{\mathbb{J}}(h)=M_P(h).

Used by Proof
Proof of Proposition 2

To show part 1, write

\begin{aligned} M_P(h) &\stackrel{(i)}{=}4c^2 \max_{g\in \mathcal{G}} \mathbb{E}\left\{\rho(Z)g(Z)-I_0(X)r(O)h(W)g(Z)-c^2I_0(X)g(Z)^2\right\} \\ &\stackrel{(ii)}{=}4c^2 \max_{g\in \mathcal{G}} \mathbb{E}\Big\{I_0(X)g(Z)\mathcal{T}\big(h_{0}^\dag-h\big)(Z)-c^2I_0(X)g(Z)^2\Big\} \\ &=4c^2 \max_{g\in \mathcal{G}} \mathbb{E}\Big[I_0(X)g(Z)\mathcal{T}\big(h_{0}^\dag-h\big)(Z)-c^2I_0(X)g(Z)^2-\frac{1}{4c^2}I_0(X)\big\{\mathcal{T}\big(h_{0}^\dag-h\big)(Z)\big\}^2\Big] \\ & \quad +\mathbb{E}\Big[I_0(X)\big\{\mathcal{T}\big(h_{0}^\dag-h\big)(Z)\big\}^2\Big] \\ &=\max_{g\in \mathcal{G}} -\mathbb{E}\Big[I_0(X)\big\{2c^2g(Z)-\mathcal{T}\big(h_{0}^\dag-h\big)(Z)\big\}^2\Big] +\mathbb{E}\Big[I_0(X)\big\{\mathcal{T}\big(h_{0}^\dag-h\big)(Z)\big\}^2\Big] \\ &\stackrel{(iii)}{=}\mathbb{E}\Big[I_0(X)\big\{\mathcal{T}\big(h_{0}^\dag-h\big)(Z)\big\}^2\Big] \end{aligned}

where the equality (i) holds because \rho is the Riesz representer of the map g \mapsto \mathbb{E}\left[m(O;g)\right], the equality (ii) is because I_0 h_{0}^\dag is a solution to equation (?), and the equality (iii) is because by assumption, (2c^2)^{-1}\cdot\mathcal{T}(h_0^\dag-h)\in\mathcal{G}. Note that if \mathcal{G} was unrestricted, part 1 would be, on expectation, the Frenchel conjugate representation of the squared function. This concludes the proof of part 1. To show part 2 write

\begin{aligned} \max_{g\in \mathcal{G}}& -\mathbb{E}\Big[I_0(X)\big\{2c^2g(Z)-\mathcal{T}\big(h_{0}^\dag-h\big)(Z)\big\}^2\Big] +\mathbb{E}\Big[I_0(X)\big\{\mathcal{T}\big(h_{0}^\dag-h\big)(Z)\big\}^2\Big]\\ &\stackrel{(i)}{=}-\min_{g\in \mathcal{G}} \mathbb{E}\Big[\big\{I_0(X)g(Z)-I_0(X)\mathcal{T}\big(h_{0}^\dag-h\big)(Z)\big\}^2\Big] +\mathbb{E}\Big[I_0(X)\big\{\mathcal{T}\big(h_{0}^\dag-h\big)(Z)\big\}^2\Big] \\ &\stackrel{(ii)}{=}-\big\Vert I_0\mathcal{T}\big(h_{0}^\dag-h\big) - \mathcal{P}_{\overline{I_0\mathcal{G}}}\circ \mathcal{T}I_0(h_0^\dag-h)\big\Vert_2^2 + \big\Vert I_0\mathcal{T}\big(h_{0}^\dag-h\big) \big\Vert_2^2 \\ &\stackrel{(iii)}{=}\big\Vert \mathcal{P}_{\overline{I_0\mathcal{G}}}\circ \mathcal{T}I_0(h_0^\dag-h)\big\Vert_2^2 \end{aligned}

where the equality (i) holds because \mathcal{G} is a linear space, the equality (ii) holds by the definition of the projection operator and the assumption that \mathcal{P}_{\overline{I_0\mathcal{G}}}\circ \mathcal{T}I_0(h_0^\dag-h) \in I_0\mathcal{G}, and the equality (iii) holds by Pythagorean Theorem. Hence, by Proposition 1, we have M_P(h)=\Vert \mathcal{P}_{\overline{I_0\mathcal{G}}}\circ \mathcal{T}I_0(h_0^\dag-h)\Vert_2^2=Q_\mathbb{J}(h). This concludes the proof of part 2.

Corollary

Corollary 1

J53Q

Let \mathcal{G}\subset\mathcal{L}^2(P_{Z}) and let c \ne 0 be a constant. If there exists h_0^\dag such that h_0=I_0h_0^\dag solves equation (?), and for any h\in \mathcal{H} it holds that (2c^2)^{-1}\mathcal{T}_{P}(h_0^\dag-h)\in\mathcal{G}, then

\begin{aligned} &\arg\min_{h \in \mathcal{H}}\left\{\Vert \rho-\mathcal{T}I_0h\Vert_2^2+\lambda_\mathcal{H}\Vert h\Vert^2\right\}\\ &=\arg\min_{h \in \mathcal{H}}\big[\max_{g\in \mathcal{G}} \mathbb{E}_P\left\{m(O;g)-I_0(X)r(O)h(W)g(Z)-c^2I_0(X)g(Z)^2\right\}+\lambda_\mathcal{H}\Vert h\Vert^2\big]. \end{aligned}
Theorem

Theorem 2

WSJ0

Let \mathcal{A} : \mathcal{E}_1 \rightarrow \mathcal{E}_2 be a bounded linear operator between the Hilbert spaces \mathcal{E}_1 and \mathcal{E}_2 and let \rho \in \mathcal{E}_2 be in the range of \mathcal{A}. Suppose the solution h_0 to the equation \mathcal{A}h=\rho with minimal \mathcal{E}_1–norm satisfies the \beta–source condition (?). Let h_\lambda be defined as in (?). Then, there exist constants c_1 and c_2 such that \Vert h_0 - h_{\lambda} \Vert_{\mathcal{E}_1}^2 < c_1 \:\lambda^{\min\left\{\beta ,2\right\}} and \Vert \mathcal{A} ( h_0 - h_{\lambda}) \Vert_{\mathcal{E}_2}^2 < c_2 \:\lambda^{\min\left\{\beta +1 ,2\right\}}.

Used by Proof
Assumption

Assumption 3

5KES

There exists \beta >0 such that h_{0,KRAS}^\dag = (T_{\mathcal{H}}\circ\mathcal{T}^* \circ I_0^{op}\circ \mathcal{T})^{\beta/2} \: h^{*} for some h^{*} \in \mathcal{H}. Equivalently,

h_{0,KRAS}^\dag = (T_{\mathcal{H}}\circ\mathcal{T}^* \circ I_0^{op}\circ \mathcal{T})^{\beta/2} \: T_\mathcal{H}^{1/2} \: h^{**} \quad \text{for some } \: h^{**} \in \mathcal{L}^2(P_W).
Assumption

Assumption 4

YFJ2

There exists \beta>0 and h^{**}\in\mathcal{L}^2(P_{W}) such that

\begin{aligned} h_{0,KRAS}^\dag =T^{1/2}_{\mathcal{H}}\circ\left(\widetilde{\mathcal{T}}^*\circ\widetilde{\mathcal{T}}\right)^{\beta/2} h^{**}=T^{1/2}_{\mathcal{H}}\circ \left( T_\mathcal{H}^{1/2} \circ \mathcal{T}^*\circ I_0^{op} \circ \mathcal{T} \circ T_\mathcal{H}^{1/2}\right)^{\beta/2} \: h^{**}. \end{aligned}
Theorem

Theorem 3 (Regularization Bias)

OF60

Suppose that Assumptions 1 and 2 hold. If, in addition, Assumption 3 or Assumption 4 hold, then there exist constants c_1^* and c_2^* such that

\begin{aligned} \Vert h_{\lambda_{\mathcal{H}},KRAS}^\dag - h_{0,KRAS}^\dag \Vert_{\mathcal{H}}^2 \le c_1^* \: \lambda_{\mathcal{H}}^{\min\left\{\beta,2\right\}} \: \text{and}\: \Vert \mathcal{T}I_0(h_{\lambda_{\mathcal{H}},KRAS}^\dag - h_{0,KRAS}^\dag )\Vert_2^2\le c_2^* \:\lambda_{\mathcal{H}}^{\min\left\{\beta+1,2\right\}}. \end{aligned}
Proof of Theorem 3

Recall that h'_{0,KRAS} is the solution to equation (?) with minimal \mathcal{L}^2(P_W)–norm and that h_{\lambda_\mathcal{H},KRAS}'=\arg\min_{h'\in\mathcal{N}(T_\mathcal{H})^\perp}\{\Vert\rho -\widetilde{\mathcal{T}}h'\Vert_2^2+4\lambda_{\mathcal{H}}\Vert h'\Vert_2^2\} (see Section ?). Now, Lemma 6 establishes that h'_{0,KRAS}\in \mathcal{N}(T_\mathcal{H})^\perp and h^\dag_{0,KRAS}=T_\mathcal{H}^{1/2}h'_{0,KRAS}. On the other hand, we know that h'_{\lambda_\mathcal{H},KRAS}\in \mathcal{N}(T_\mathcal{H})^\perp and h^\dag_{\lambda_\mathcal{H},KRAS}=T_\mathcal{H}^{1/2}h'_{\lambda_\mathcal{H},KRAS}. It then follows from Lemma 3 (d) that \|h_{\lambda_\mathcal{H},KRAS}^\dag - h_{0,KRAS}^\dag\|_\mathcal{H} = \|h_{\lambda_\mathcal{H},KRAS}' - h_{0,KRAS}'\|_2 and \|\widetilde{\mathcal{T}}(h_{\lambda_\mathcal{H},KRAS}'-h_{0,KRAS}')\|_2 = \|I_0^{op}\circ\mathcal{T}(h_{\lambda_\mathcal{H},KRAS}^\dag - h_{0,KRAS}^\dag)\|_2. Hence, under Assumption 4, Theorem 2 with \mathcal{A}=\widetilde{\mathcal{T}} ensures that there exist constants c_1^* and c_2^* such that

\begin{aligned} \|h_{\lambda_\mathcal{H},KRAS}^\dag - h_{0,KRAS}^\dag\|_\mathcal{H}^2&=\|h_{\lambda_\mathcal{H},KRAS}' - h_{0,KRAS}'\|_2 \le c_1^*\lambda_\mathcal{H}^{\min\{\beta,2\}}\quad\text{and}\\ \|I_0^{op}\circ\mathcal{T}(h_{\lambda_\mathcal{H},KRAS}^\dag - h_{0,KRAS}^\dag)\|_2^2&=\|\widetilde{\mathcal{T}}(h_{\lambda_\mathcal{H},KRAS}'-h_{0,KRAS}')\|_2 \le c_2^*\lambda_\mathcal{H}^{\min\{\beta+1,2\}}. \end{aligned}
Assumption

Assumption 5 (Local closeness and boundedness of \mc{T})

3OET

There exists B>\big\Vert h_{0,KRAS}^\dag\big\Vert_{\mathcal{H}} and B'>0 such that if \Vert h\Vert_{\mathcal{H}}\le 2B then \mathcal{T}h\in\mathcal{G} and \left\Vert\mathcal{T}h\right\Vert_{\mathcal{G}}\le B'.

Theorem

Theorem 4

1WGV

Suppose m(o;g) = I_0(x)s(o) g(z) for some fixed real-valued function s satisfying \Vert s \Vert_{\infty} < \infty. Suppose Assumptions 1, 2, 5, 6 hold and either Assumption 3 or 4 hold when P=P_0. Suppose that the constant B in Assumption 5 is taken large enough so that \big\Vert h_{0,KRAS}^\dag\big\Vert_{\mathcal{H}}\le B. Let B_1=B'/c^2. Define the function class I_0\cdot\mathcal{G}_{B_1}:=\{I_0\cdot g\: :\: g\in\mathcal{G}_{B_1}\} and let \delta_n be an upper bound on the critical radius of I_0\cdot\mathcal{G}_{B_1}. If \lambda_{\mathcal{H}}\le 1, and \lambda_\mathcal{G}\ge157c^2\delta_n^2/B_1^2, then with probability at least \kappa(\delta_n) :=1-[36+18\cdot I\{\delta_n < 5b/6 \}\cdot\lfloor\{\log(6/5)\}^{-1}\log(b/\delta_n)\rfloor]\exp(-c_1n\delta_n^2) where c_1=\{8b^2(1+17e)\}^{-1} and b=k_2B_1, it holds that \hat{h}_{KRAS} := I_0\:\tilde{h}_{KRAS} and h_{0,KRAS}:=I_0\:h_{0,KRAS}^\dag satisfy

\left\Vert \hat {h}_{KRAS}-h_{0,KRAS}\right\Vert_2^2\le c_2\left(\frac{\delta_n^2}{\lambda_{\mathcal{H}}}+\frac{\lambda_{\mathcal{G}}}{\lambda_{\mathcal{H}}}+\lambda_{\mathcal{H}}^{\min\left\{\beta,1\right\}}\right),

and

\left\Vert\mathcal{T} \left(\hat {h}_{KRAS}-h_{0,KRAS}\right)\right\Vert_2^2\le c_3\left(\delta_n^2+\lambda_{\mathcal{G}}+\lambda_{\mathcal{H}}^{\min\left\{\beta+1,2\right\}}\right),

where c_2 and c_3 are universal constants that depend neither on n nor on \delta_n.

Proof of Theorem 4

We adapted the proof of Theorem 4 of [bennett2023source] to our estimator. To simplify notation, we omit the subscript KRAS from the functions h_{0,KRAS}^\dag, h_{\lambda_\mathcal{H},KRAS}^\dag, and \tilde h_{KRAS} defined in (?), (?), and (?). Define L(\tau):=\frac{1}{4c^2}\big\Vert I_0\mathcal{T}\big\{h_{0}^\dag-h_{\lambda_\mathcal{H}}^\dag-\tau\big(\tilde h-h_{\lambda_\mathcal{H}}^\dag\big)\big\}\big\Vert_2^2+\lambda_{\mathcal{H}}\big\Vert h_{\lambda_\mathcal{H}}^\dag+\tau\big(\tilde h- h_{\lambda_\mathcal{H}}^\dag\big)\big\Vert_{\mathcal{H}}^2. L(\tau) attains its unique global minimum at \tau=0 because h_{\lambda_\mathcal{H}}^\dag is the unique minimizer in \mathcal{H} of the criterion in (?). Then, L'(0)=0 and since L(\tau) is quadratic, a Taylor’s expansion yields

\begin{aligned} L(1)-L(0)&=L'(0)\cdot 1+\frac{1}{2}L''(0)\cdot 1^2=\frac{1}{2}L''(0)=\frac{1}{4c^2}\big\Vert I_0\mathcal{T}\big(\tilde h-h_{\lambda_\mathcal{H}}^\dag\big)\big\Vert_2^2+\lambda_{\mathcal{H}}\big\Vert \tilde h- h_{\lambda_\mathcal{H}}^\dag\big\Vert_{\mathcal{H}}^2. \end{aligned}

On the other hand, L(0) =\frac{1}{4c^2}\Vert I_0\mathcal{T}(h_0^\dag-h_{\lambda_\mathcal{H}}^\dag)\Vert_2^2+\lambda_{\mathcal{H}}\Vert h_{\lambda_\mathcal{H}}^\dag\Vert_{\mathcal{H}}^2 and L(1)=\frac{1}{4c^2}\Vert I_0\mathcal{T}(h_0^\dag-\tilde h)\Vert_2^2+\lambda_{\mathcal{H}}\Vert \tilde h\Vert_{\mathcal{H}}^2. Thus, we arrive at

\begin{aligned} \frac{1}{4c^2}\big\Vert I_0\mathcal{T}\big(\tilde h-&h_{\lambda_\mathcal{H}}^\dag\big)\big\Vert_2^2+\lambda_{\mathcal{H}}\big\Vert \tilde h- h_{\lambda_\mathcal{H}}^\dag\big\Vert_{\mathcal{H}}^2\\ &= \frac{1}{4c^2}\Big\{\big\Vert I_0\mathcal{T}\big(h_0^\dag-\tilde h\big)\big\Vert_2^2-\big\Vert I_0\mathcal{T}\big(h_0^\dag-h_{\lambda_\mathcal{H}}^\dag\big)\big\Vert_2^2\Big\}+\lambda_{\mathcal{H}}\Big(\big\Vert \tilde h\big\Vert_{\mathcal{H}}^2-\big\Vert h_{\lambda_\mathcal{H}}^\dag\big\Vert_{\mathcal{H}}^2\Big). \end{aligned}

To control the right-hand side of (Proof), we invoke Lemma 12 which establishes that with probability at least \kappa(\delta_n) the following event happens

\begin{aligned} \frac{1}{4c^2}\Big\{\big\Vert I_0 \mathcal{T} \big(h_0^\dag-&\tilde h\big)\big\Vert^2_2-\big\Vert I_0\mathcal{T} \big(h_0^\dag-h\big)\big\Vert^2_2\Big\}+\lambda_{\mathcal{H}}\Big(\big\Vert \tilde h\big\Vert_{\mathcal{H}}^2-\big\Vert h\big\Vert_{\mathcal{H}}^2\Big)\\ &\quad\quad\quad\le \frac{1}{2c^2}\big\Vert I_0\mathcal{T} \big(h_0^\dag-h\big)\big\Vert^2_2+c_0\delta_n\big\Vert I_0\mathcal{T}\big(\tilde h-h\big)\big\Vert_2+\lambda_{\mathcal{G}}B_1^2+c_0'\delta^2_n \end{aligned}

where c_0=9\cdot\max\{\Vert s\Vert_\infty,\:\Vert r\Vert_\infty\cdot k_1B,\: 2c^2b\}/(c^2 b) and c'_0=c^2\cdot c_0(2_3c_0+\max\{2,2c_0/9\}).

Hence, identity (Proof) implies that, with probability at least \kappa(\delta_n), the following holds

\begin{aligned} \frac{1}{4c^2}\big\Vert I_0\mathcal{T}\big(\tilde h-h_{\lambda_\mathcal{H}}^\dag\big)\big\Vert_2^2&+\lambda_{\mathcal{H}}\big\Vert \tilde h- h_{\lambda_\mathcal{H}}^\dag\big\Vert_{\mathcal{H}}^2\\ &\le \frac{1}{2c^2}\big\Vert I_0\mathcal{T}\big(h_0^\dag-h_{\lambda_\mathcal{H}}^\dag\big)\big\Vert_2^2+c_0\delta_n\big\Vert I_0\mathcal{T}\big(\tilde h-h_{\lambda_\mathcal{H}}^\dag\big)\big\Vert_2+\lambda_{\mathcal{G}}B_1^2+c'_0\delta_n^2. \end{aligned}

By the AM-GM inequality, c_0\delta_n\Vert I_0\mathcal{T}(\tilde h-h_{\lambda_\mathcal{H}}^\dag)\Vert_2 \le 2c^2\cdot c_0^2 \delta_n^2 +\frac{1}{8c^2}\Vert I_0\mathcal{T}(\tilde h-h_{\lambda_\mathcal{H}}^\dag)\Vert_{2}^2, which combined with the right-hand side of the last display yields that

\begin{aligned} \lambda_{\mathcal{H}}\big\Vert \tilde h-h_{\lambda_\mathcal{H}}^\dag\big\Vert_{\mathcal{H}}^2+\frac{1}{8c^2}\big\Vert I_0\mathcal{T}\big(\tilde h-h_{\lambda_\mathcal{H}}^\dag\big)\big\Vert_2^2&\le \frac{1}{2c^2}\big\Vert I_0\mathcal{T}\big(h_0^\dag-h_{\lambda_\mathcal{H}}^\dag\big)\big\Vert_2^2+\lambda_{\mathcal{G}}B_1^2+c_0''\delta_n^2, \end{aligned}

with c_0''=c'_0+2c^2\cdot c_0^2, happens with probability at least \kappa(\delta_n), which, in turn, implies that \lambda_{\mathcal{H}}\Vert \tilde h-h_{\lambda_\mathcal{H}}^\dag\Vert_{\mathcal{H}}^2\le \frac{1}{2c^2}\Vert I_0\mathcal{T} (h_0^\dag-h_{\lambda_\mathcal{H}}^\dag)\Vert_2^2+\lambda_{\mathcal{G}}B_1^2+c_0''\delta_n^2 also holds with probability at least \kappa(\delta_n).

Applying the inequality \Vert h_1+h_2\Vert_{\mathcal{H}}^2\le 2\Vert h_1\Vert_{\mathcal{H}}^2+2\Vert h_2\Vert_{\mathcal{H}}^2 we then obtain that the event

\begin{aligned} \big\Vert \tilde h-h_0^\dag\big\Vert_{\mathcal{H}}^2 \le& \:2\big\Vert \tilde h-h_{\lambda_\mathcal{H}}^\dag\big\Vert_{\mathcal{H}}^2+2\big\Vert h_{\lambda_\mathcal{H}}^\dag-h_0^\dag\big\Vert_{\mathcal{H}}^2\\ \le& \frac{1}{c^2\cdot\lambda_{\mathcal{H}}}\big\Vert I_0\mathcal{T}\big(h_0^\dag-h_{\lambda_\mathcal{H}}^\dag\big)\big\Vert_2^2+\frac{2\lambda_{\mathcal{G}}B_1^2}{\lambda_{\mathcal{H}}}+\frac{2c_0''\delta_n^2}{\lambda_{\mathcal{H}}}+2\big\Vert h_{\lambda_\mathcal{H}}^\dag-h_0^\dag\big\Vert_{\mathcal{H}}^2, \end{aligned}

happens with probability at least \kappa(\delta_n). Because \tilde h-h_0^\dag\in\mathcal{H}, it follows that \Vert \tilde h-h_0^\dag\Vert_2^2\le k_1^2 \: \Vert \tilde h-h_0^\dag\Vert_{\mathcal{H}}^2. Hence, invoking Theorem 3 to bound the terms \Vert I_0\mathcal{T}(h_0^\dag-h_{\lambda_{\mathcal{H}}}^\dag)\Vert_2^2 and \Vert h_{\lambda_{\mathcal{H}}}^\dag-h_0^\dag\Vert_{\mathcal{H}}^2, we conclude that, with probability at least \kappa(\delta_n), \Vert \tilde h-h_0^\dag\Vert_2^2\le c_2\big(\frac{\delta_n^2}{\lambda_{\mathcal{H}}}+\frac{\lambda_{\mathcal{G}}}{\lambda_{\mathcal{H}}}+\lambda_{\mathcal{H}}^{\min\left\{\beta,1\right\}}\big), where c_2=k_1^2\cdot\max\{2c_1^*+c_2^*c^{-2}\:,\: 2B_1^2\:,\:2c_0''\}, and c_1^* and c_2^* are the constants of Theorem 3.

To derive the bound for \Vert I_0\mathcal{T}(\tilde h-h_0^\dag)\Vert_2^2, first note that from inequality (?) we have that the event

\begin{aligned} \frac{1}{8c^2}\big\Vert I_0\mathcal{T}\big(\tilde h-h_{\lambda_{\mathcal{H}}}^\dag\big)\big\Vert_2^2&\le \frac{1}{2c^2}\big\Vert I_0\mathcal{T}\big(h_0^\dag-h_{\lambda_{\mathcal{H}}}^\dag\big)\big\Vert_2^2+\lambda_{\mathcal{G}}B_1^2+c_0''\delta_n^2 \end{aligned}

happens with probability at least \kappa(\delta_n). Combining this inequality with the inequality \Vert I_0\mathcal{T}(\tilde h-h_0^\dag)\Vert_2^2\le 2\Vert I_0\mathcal{T}(\tilde h-h_{\lambda_{\mathcal{H}}}^\dag)\Vert_2^2+2\Vert I_0\mathcal{T}(h_{\lambda_{\mathcal{H}}}^\dag-h_0^\dag)\Vert_2^2 we conclude that

\begin{aligned} \big\Vert I_0\mathcal{T}\big(\tilde h-h_0^\dag\big)\big\Vert_2^2\le 10\big\Vert I_0\mathcal{T}\big(h_0^\dag-h_{\lambda_{\mathcal{H}}}^\dag\big)\big\Vert_2^2+16c^2B_1^2\lambda_{\mathcal{G}}+16c^2c_0''\delta_n^2 \end{aligned}

holds with probability at least \kappa(\delta_n). Invoking again Theorem 3 to bound \Vert I_0\mathcal{T}(h_0^\dag-h_{\lambda_{\mathcal{H}}}^\dag)\Vert_2^2, we conclude that the event \Vert I_0\mathcal{T}(\tilde h-h_0^\dag)\Vert_2^2\le c_3\left(\delta_n^2+\lambda_{\mathcal{G}}+\lambda_{\mathcal{H}}^{\min\left\{\beta+1,2\right\}}\right) happens with probability at least \kappa(\delta_n) where c_3=\max\{10 c_2^*\:,\:16c^2B_1^2\:,\:16c^2c_0''\}. This completes the proof.

Proof of Theorem 4

When \delta_n<5b/6, we show that \log(\log(b/\delta_n))\le c_1n\delta_n^2/2, where c_1 is the constant defined in Theorem 4, for a suitable choice of \delta_n. Suppose \delta_n^2\ge c_0\log(\log(n))/n with c_0=\max\{b^2/\log(\log(3)),2/c_1\}. Since \delta_n/b<1, it holds that \delta_n/b\ge(\delta_n/b)^2\ge c_0\log(\log(n))/(nb^2). If n\ge 3, the assumption c_0\ge b^2/\log(\log(3)) implies \delta_n/b\ge \log(\log(n))/\{n\log(\log(3))\}\ge1/n which then implies n\ge b/\delta_n. Furthermore, since c_0\ge 2/c_1, we have \log(\log(b/\delta_n))\le \log(\log(n))\le n\delta_n^2/c_0\le c_1n\delta_n^2/2, as desired.

Lemma

Lemma 1

E5JJ

Let \mathcal{F} denote a class of real-valued measurable functions f:\mathcal{O}\mapsto\mathbb{R} such that \Vert f\Vert_\infty\le b for all f\in\mathcal{F}. Let \delta'_n and \delta_n denote the critical radii of I_0\cdot \mathcal{F}:=\{I_0\cdot f \::\: f\in\mathcal{F}\} and \mathcal{F}, respectively, where I_0 is the indicator that X\in\mathcal{X}_0 and X is a subvector of O. Assume that \delta_n^2=o(\exp\{-dn\}) for any fixed constant d>0 and \delta_n=O(\alpha_n) where \alpha_n decreases with n. Then, \delta'_n\le O(\alpha_{\lfloor n\mu_0/2\rfloor}) where \mu_0:=P(X\in\mathcal{X}_0).

Proof of Lemma 1

Let Q denote the conditional law of O given X\in\mathcal{X}_0, where O\sim P. Then, for any f\in\mathcal{L}^2(P), we have \Vert I_0\cdot f\Vert_{\mathcal{L}^2(P)}^2=\mu_0\Vert f\Vert_{\mathcal{L}^2(Q)}^2. Let N := \sum_{i=1}^N I_0(X_i) and let \mathcal{F}_0(\delta):=\{f\in\mathcal{F}:\Vert f\Vert_{\mathcal{L}^2(Q)}\le\delta/\sqrt{\mu_0}\}. Then, letting s=\delta/\sqrt{\mu_0}, we have

\begin{aligned} \mathcal{R}_n(I_0\cdot\mathcal{F},\:P,\:\delta) &=\sum_{m=1}^n \mathbb{E}_{P,P_\varepsilon}\Big\{\sup_{f\in\mathcal{F}_0(\delta)}\frac{1}{n}\sum_{i=1}^n\varepsilon_iI_0(X_i)f(O_i)\Big\vert N=m\Big\}P(N=m)\\ &=\sum_{m=1}^n \mathbb{E}_{P,P_\varepsilon}\Big\{\sup_{f\in\mathcal{F}_0(\delta)}\frac{1}{m}\sum_{i=1}^m\varepsilon_if(O_i)\Big\vert X_i\in\mathcal{X}_0,\forall \:i=1,\dots,m\Big\}\frac{m}{n}P(N=m)\\ &=\sum_{m=1}^n \mathbb{E}_{Q,P_\varepsilon}\Big\{\sup_{f\in\mathcal{F}_0(\delta)}\frac{1}{m}\sum_{i=1}^m\varepsilon_if(O_i)\Big\}\frac{m}{n}P(N=m)\\ &=\sum_{m=1}^n \mathcal{R}_m(\mathcal{F},Q,\:\delta/\sqrt{\mu_0})\:\frac{m}{n}\:P(N=m)=\mathbb{E}\left\{\mathcal{R}_N(\mathcal{F},Q,s)\frac{N}{n}\right\}, \end{aligned}

where the second equality follows because the O_1,\dots,O_n are i.i.d.

Let \mathcal{D}_n be the event \{N-n\mu_0<-n\mu_0/2\}. By Hoeffding inequality we have P(D_n)<\exp\{-2n^2\mu_0^2/(4n)\}=\exp(-dn) where d:=\mu_0^2/2. Then for s:=\sup\{\delta_m: n\mu_0/2\le m\le n\}

\begin{aligned} \mathcal{R}_n(I_0\cdot\mathcal{F},P,\delta)&=\mathbb{E}\left\{\mathcal{R}_N(\mathcal{F},Q,s)\frac{N}{n}I_{\mathcal{D}_N}(N)\right\}+E\left\{\mathcal{R}_N(\mathcal{F},Q,s)\frac{N}{n}I_{\mathcal{D}_N^c}(N)\right\}\\ &\le bP(\mathcal{D}_n)+\sup_{n\mu_0/2\le m\le n}\mathcal{R}_m(\mathcal{F},Q,s)\\ &\le b\exp(-dn)+s^2. \end{aligned}

Let \varepsilon be such that 1/\mu_0>1+\varepsilon^2. Because by assumption \delta_n^2=o(\exp\{-dn\}), then there exists n_0 such that if n\ge n_0, b\exp(-dn)\le\varepsilon^2s^2. On the other hand, our choice of \varepsilon implies that \mathcal{R}_n(I_0\cdot\mathcal{F},P,\delta)>\mathcal{R}_n(I_0\cdot\mathcal{F},P,s\sqrt{1+\varepsilon^2}). Then, the last display implies that \mathcal{R}_n(I_0\cdot\mathcal{F},P,s\sqrt{1+\varepsilon^2})<(1+\varepsilon^2)s^2. Therefore, \delta_n'\le s\sqrt{1+\varepsilon^2}=O(\alpha_{\lfloor n\mu_0/2\rfloor}) by the assumed monotonicity of \alpha_n.

Assumption

Assumption 7

06L8
  1. There exists B_2>0 such that, for any g\in\mathcal{L}^2(P_{Z}), it holds that \left\Vert m(\:\cdot\:;g)\right\Vert_2\le B_2 \: \left\Vert I_0 \:g\right\Vert_2.

  2. The function class m\circ \mathcal{G}:=\left\{o \mapsto m(o;g)\::\:g\in\mathcal{G}\right\} can be equipped with a metric d under which it is separable. Moreover, d is controlled by the RKHS norm in the sense that d(m(\:\cdot\:;g),m(\:\cdot\:,g'))\le \Vert g-g'\Vert_\mathcal{G} for all g,g'\in\mathcal{G} and it dominates the metric induced by the sup-norm on \mathcal{O}, meaning there exists a constant B_3>0 such that \sup_{o\in\mathcal{O}}|f(o)-f'(o)|\le B_3\cdot d(f,f')\quad\text{for all } f,f'\in m\circ\mathcal{G}.

Theorem

Theorem 5

SQGV

Suppose Assumptions 1, 2, 5, 6, ? hold and either Assumption 3 or 4 hold for P=P_0. Suppose that the constant B in Assumption 5 is taken large enough so that \big\Vert h_{0,KRAS}^\dag\big\Vert_{\mathcal{H}}\le B. Let B_1=B'/c^2. Define m\circ \mathcal{G}_{B_1}:=\left\{o \mapsto m(o;g)\::\:g\in\mathcal{G}_{B_1}\right\}, and let the function class I_0\cdot\mathcal{G}_{B_1} be defined as in Theorem 4. Let \delta_n be an upper bound on the critical radius of I_0\cdot\mathcal{G}_{B_1} and of star(m\circ \mathcal{G}_{B_1}). If \lambda_{\mathcal{H}}\le 1, and \lambda_\mathcal{G}\ge157c^2\delta_n^2/B_1^2, then inequalities (4) and (4) hold for some universal constants c_2 and c_3, with probability at least as \kappa(\delta_n) as given in Theorem 4 with c_1 as defined in that Theorem, but with b:=B_1\max\{k_2,B_3\}, where B_3 is the constant specified in Assumption ?.

Proof of Theorem 5

The proof follows analogously to that of Theorem 4, but applies Lemma 13 in place of Lemma 12, and uses the updated constants defined in Lemma 13.

Lemma

Lemma 2

SMVI

Suppose m(o;g)=g\{q(z)\} for some measurable map q:\mathcal{Z}\rightarrow\mathcal{Z}, where z is a subvector of o. Suppose that \mathcal{G}_{B_1} is star-shaped. Then star(m\circ\mathcal{G}_{B_1}) is also star-shaped. Furthermore, letting \delta_n and \delta'_n denote the critical radii of (m\circ\mathcal{G}_{B_1}) and \mathcal{G}_{B_1}, respectively, there exists a universal constant c_1'' such that \delta_n\le c_1''\delta'_n.

Proof of Lemma 2

Let P_0^q:=P_0\circ q^{-1} be the pushforward measure of P_0 under q. For any g\in\mathcal{G}_{B_1}, the composition m(\:\cdot\:,g)=g\circ q(\cdot) satisfies

\Vert m(\:\cdot\:,g)\Vert_{\mathcal{L}^2(P_0)}^2=\Vert g\circ q \Vert_{\mathcal{L}^2(P_0)}^2=\int_\mathcal{Z}g\{q(z)\}^2dP_0(z)=\int_{q(\mathcal{Z})}g(\tilde z)^2d(P_0\circ q^{-1})(\tilde z)=\Vert g\Vert_{\mathcal{L}^2(P_0^q)}^2,

so the map g\mapsto g\circ q=m(\:\cdot\:,g) is an isometry between (\mathcal{G}_{B_1},\mathcal{L}^2(P_0^q)) and (m\circ\mathcal{G}_{B_1},\mathcal{L}^2(P_0)).

Observe that the Rademacher complexity is preserved under reparameterization: if Z_1,\dots,Z_n\sim P_{0}, \tilde Z_1,\dots,\tilde Z_n\sim P_0^q, and \varepsilon_1,\dots,\varepsilon_n are i.i.d. Rademacher variables, then \frac{1}{n}\sum_{i=1}^n\varepsilon_ig\{q(Z_i)\}^2=\frac{1}{n}\sum_{i=1}^n\varepsilon_ig(\tilde Z_i)^2, so for all \delta>0, \mathcal{R}_n(m\circ \mathcal{G}_{B_1},\:P_0,\:\delta) = \mathcal{R}_n(\mathcal{G}_{B_1},\:P_0^q,\:\delta). Then the critical radius of m\circ\mathcal{G}_{B_1} under P_0 equals the critical radius of \mathcal{G}_{B_1} under P_0^q. The result follows because the critical radius of an RKHS class is preserved up to multiplicative constants under changes of measure. That o \mapsto m(o, g) = g(q(z)) is star shaped follows because o \mapsto m(o, g) = g(q(z)) is linear in g and \mathcal{G}_{B_1} is star-shaped.

Remark

Remark 1

6Z1U

When \delta_n\ge 5b/6 for b=k_2B_1 (=B_1\max\{k_2,B_3\}), the bounds on \Vert \hat {h}_{KRAS}-h_{0,KRAS}\Vert_2^2 and \Vert \mathcal{T}(\hat {h}_{KRAS}-h_{0,KRAS})\Vert_2^2 provided in Theorem 4 (Theorem 5) hold with probability at least 1-36\exp\left(-c_1n\delta_n^2\right). On the other hand, if c_0\log(\log(n))/n\le \delta_n^2<\left(5b/6\right)^2 with c_0=\max\{b^2/\log(\log(3)),2/c_1\}, and n\ge 3, the bounds in Theorem 4 and Theorem 5 remain valid with probability at least 1-18\cdot \{\log(6/5)\}^{-1}\exp\left(-\frac{c_1}{2}n\delta_n^2\right), (see Appendix ?), which tends to 1 as n \rightarrow \infty because n \:\delta_n^2 \ge c_0 \:\log(\log(n))\rightarrow\infty. The condition c_0\log(\log(n))/n\le \delta_n^2 holds for a sufficiently large n, in particular, when \mathcal{G} is an RKHS with Gaussian kernel because in such case \delta_n^2 = O(\log(n)/n).

Corollary

Corollary 2

K2FA

Suppose that the conditions of Theorems 4 or 5 hold. If the penalization parameters are chosen as \lambda_{\mathcal{H}}=O(\delta_n^{2/(1+\min\{\beta,1\})}) and \lambda_{\mathcal{G}}=O\left(\delta_n^2\right), then

\left\Vert \hat {h}_{KRAS}-h_{0,KRAS}\right\Vert_2=O_p\left(\delta_n^{\frac{\min\left\{\beta,1\right\}}{1+\min\left\{\beta,1\right\}}}\right)\quad\text{and}\quad \left\Vert \mathcal{T}(\hat {h}_{KRAS}-h_{0,KRAS})\right\Vert_2=O_p\left(\delta_n\right).
Remark

Remark 2

BTH9

The errors bounds established in Theorems 4 and 5 require that the estimator \tilde {h}_{KRAS} has RKHS norm at most B, where B is the constant specified in Assumption 5. While this constant is not typically known in practice, one can select a sufficiently large B and solve the optimization problem in (?) over the constrained class \mathcal{H}_{B}. The computation of the solution in the restricted class \mathcal{H}_{B} reduces to the problem of optimizing a quadratic objective function in \mathbb{R}_n subject to a quadratic constraint. As such, enforcement of the constraint \Vert \tilde{h}_{KRAS} \Vert_{\mathcal{H}} < B can be conducted as follows. First a solution to the unconstrained optimization problem is found. If the solution satisfies the constraint, then this is the solution to the constrained problem. If not, the unconstrained optimization problem is altered as indicated in Exercise 4.22 of [boyd2004convex], and the unconstrained solution to this new problem agrees with the solution to the original constrained problem.

Proof

Proof

RB3F

We denote the support of random variables (V_1,\dots,V_k) and their conditional versions V_1|(V_2=v_2,\dots,V_k=v_k), for k\ge2, as \text{supp}(V_1,\dots,V_k) and \text{supp}(V_1|V_2=v_2,\dots,V_k=v_k).

Observe that q, defined in Example 2, is strictly monotone and differentiable a.e. with respect to P_{0,A}, and admits an inverse that is differentiable a.e. with positive derivative.

Suppose that \tilde\rho_{P_0}(a,l,w) = I_0(a,l)\cdot\frac{dq^{-1}(a)}{da}\cdot\frac{p_{0,A|L,W}\{q^{-1}(a)\mid l,w\}}{p_{0,A|L,W}(a|l,w)}, where I_0(a,l):=I\{a\in[c+\delta,d]\}, is bounded by some B_0>0. Then, for any h\in\mathcal{L}^2(A,L,W),

\begin{aligned} \Vert\widetilde{m}(O;h)\Vert_2^2&=\iint \left[\int h\{q(a),l,w\}^2p_{0,A|L,W}(a|l,w)da\right]dP_0(l,w)\\ &\stackrel{(i)}{=}\iiint h(\tilde a,l,w)^2\cdot I\{\tilde a\in[c+\delta,d]\}\cdot \frac{dq^{-1}(\tilde a)}{d\tilde a}p_{0,A|L,W}(q^{-1}(\tilde a)|l,w)\:d\tilde a\cdot dP_{0}(l,w)\\ &\stackrel{(ii)}{=}\iiint h(\tilde a,l,w)^2\cdot I_0(\tilde a,l)\cdot\tilde \rho_{P_0}(\tilde a,l,w)p_{0,A|L,W}(\tilde a|l,w)\:d\tilde a\cdot dP_{0}(l,w)\\ &\stackrel{(iii)}{\le} B_0\iiint I_0(\tilde a,l)\cdot h(\tilde a,l,w)^2p_{0,A|L,W}(\tilde a|l,w)\:d\tilde a\cdot dP_0(l,w)=B_0\cdot \Vert I_0h\Vert_2^2, \end{aligned}

where the equality in (i) follows from the change of variable \tilde a = q(a), which is valid since q has an a.e. differentiable inverse with strictly positive derivative and maps [c,d] onto [c+\delta,d]. Equality (ii) follows directly from the definition of \tilde\rho_{P_0}, and the inequality (iii) uses the assumption that |\tilde\rho_{P_0}(a,l,w)|\le B_0 for all (a,l,w)\in\text{supp}(A,L,W). This proves the first inequality in part 1 of Assumption ? with constant B_2=\sqrt{B_0}.

To verify part 2 of Assumption ?, we impose the following common support assumption: \text{supp}(A|L=l,U=u)=\text{supp}(A|L=l) for all (l,u)\in\text{supp}(L,U), which is standard in proximal inference (see [olivas2025chap3]). This assumption, together with the fact that W is a negative control outcome– i.e., (A,Z)\perp W|L,U – imply that \text{supp}(A|L=l,W=w)=\text{supp}(A|L=l) for all (l,w)\in\text{supp}(L,W). Moreover, \text{supp}(A,L,W)=[c,d]\times\text{supp}(L,W) given that \text{supp}(A|L=l)=[c,d] for all l\in\mathcal{L}.

Next, we define an equivalence relation on \mathcal{H} as follows: h\leftrightarrow h' if and only if h(a,l,w)=h'(a,l,w) for all (a,l,w)\in[c+\delta, d]\times\text{supp}(L,W). Let \widetilde{\mathcal{H}}:=\{[h]:h\in\mathcal{H}\} denote the set of equivalences classes in \mathcal{H} under this relation. Note that h\leftrightarrow h' iff h-h'\in[0]:=\{h\in\mathcal{H}:h(a,l,w)=0\:\forall (a,l,w)\in[c+\delta,d]\times\text{supp}(L,W)\}. In the special case of a Gaussian RKHS, the zero function is the only function that vanishes on a non-empty open subset (see Corollary 4.44 of [steinwart2008support]); hence each equivalence class contains a single function, and \widetilde{\mathcal{H}} coincides with \mathcal{H}. Define the map \tilde d:\widetilde{\mathcal{H}}\times\widetilde{\mathcal{H}}\mapsto\mathbb{R} by

\tilde d([h_1],[h_2]):=\inf\{\Vert h_1-h_2+v\Vert_\mathcal{H}\mid v\in[0]\}.

Note that \tilde d is well-defined because the set in the right hand side is invariant to the choice of representer h_1\in[h_1] and h_2\in[h_2]. We now verify that \tilde d is a metric on \widetilde{\mathcal{H}}. By construction, \tilde d is symmetric. To see that the triangle inequality holds, given \epsilon >0 and [h_1],[h_2],[h_3]\in\widetilde{\mathcal{H}}, pick v,v'\in[0] such that \Vert h_1-h_2+v\Vert_\mathcal{H} < \tilde d([h_1],[h_2]) + \epsilon and \Vert h_2-h_3+v'\Vert_\mathcal{H} < \tilde d([h_2],[h_3]) + \epsilon. Then, \Vert h_1-h_3+v+v'\Vert_\mathcal{H} \le \tilde d([h_1],[h_2])+d([h_2],[h_3])+ 2 \epsilon. Because v+v'\in[0], it follows that \tilde d([h_1],[h_3]) \le \tilde d([h_1],[h_2])+\tilde d([h_2],[h_3]) + 2 \epsilon. Since this holds for all \epsilon >0, the triangular inequality follows.

Finally, we show that \tilde d([h_1],[h_2])=0 if and only if [h_1]=[h_2]. The if direction is trivial since \tilde d([h],[h])=\inf\{\Vert v\Vert\mid v\in[0]\}=0 for all [h]\in\widetilde{\mathcal{H}}. To show the only if direction suppose that \tilde d([h_1],[h_2])=0 but [h_1]\neq[h_2]. Then there exist representatives h_1\in[h_1], h_2\in[h_2], and a point (a_0,l_0,w_0)\in[c+\delta,d]\times\text{supp}(L,W) such that h_1(a_0,l_0,w_0)\neq h_2(a_0,l_0,w_0). Let \epsilon:=|h_1(a_0,l_0,w_0)- h_2(a_0,l_0,w_0)|>0, and k:=[\sup_{(a,l,w)\in\text{supp}(A,L,W)}K_{\mathcal{H}}\{(a,l,w),(a,l,w)\}]^{1/2}<\infty. By the definition of \tilde d and the fact that \tilde d([h_1],[h_2])=0, there exists v'\in[0] such that \Vert h_1-h_2+v'\Vert_\mathcal{H}<\epsilon/k. Let h_1':=h_1+v'. Then we arrive at the following contradiction:

\begin{aligned} k\Vert h_1'-h_2\Vert_\mathcal{H}<|h_1(a_0,l_0,w_0)-h_2(a_0,l_0,w_0)|=|h'_1(a_0,l_0,w_0)-h_2(a_0,l_0,w_0)|\le k\Vert h_1'-h_2\Vert_\mathcal{H}, \end{aligned}

where the equality holds because h_1\leftrightarrow h'_1 and the final inequality because for any h\in\mathcal{H}, \Vert h\Vert_\infty\le k\:\Vert h\Vert_\mathcal{H}. Hence, \tilde d([h_1],[h_2])=0 implies [h_1]=[h_2].

We now show that (\widetilde{\mathcal{H}},d) is separable. Since K_\mathcal{H} is continuous and \text{supp}(A,L,W) is compact, \mathcal{H} is separable under the metric induced by its norm (see Proposition 11.7 of [paulsen2016introduction]). Let \{h_n\}\subset\mathcal{H} be a countable dense subset, and consider the set of equivalence classes \{[h_n]\}. We claim this set is dense in \widetilde{\mathcal{H}}. Indeed, for any [h]\in\widetilde{\mathcal{H}} and \epsilon>0, there exists a representative h'\in[h] and some h_n\in\mathcal{H} such that \Vert h'-h_n\Vert_\mathcal{H}<\epsilon. Then d([h],[h_n])\le\Vert h-h_n\Vert_\mathcal{H}<\epsilon, so \{[h_n]\} is dense in \widetilde{\mathcal{H}}, and thus \widetilde{\mathcal{H}} is separable.

Next, consider the class \widetilde{m}\circ \mathcal{H}:=\{o\mapsto h(q(a),l,w)\}, which is in a one-to-one correspondence with \widetilde{\mathcal{H}}. More precisely, the map h\mapsto \widetilde{m}(\cdot,h) is well-defined on equivalence classes, since \widetilde{m}(o;h_1)=\widetilde{m}(o;h_2) for all o\in\mathcal{O} iff h_1(q(a),l,w)=h_1(q(a),l,w) for all (a,l,w)\in[c,d]\times\text{supp}(L,W), h_1(a,l,w)=h_2(a,l,w) for all (a,l,w)\in[c+\delta,d]\times\text{supp}(L,W) (since q \left([c,d]\right)=[c+\delta,d]) iff h_1\leftrightarrow h_2. Thus, we can equip \widetilde{m}\circ\mathcal{G} with the metric d defined as:

d(f_1,f_2):=\tilde d([h_{f_1}],[h_{f_2}])\quad \text{ for any }f_1,f_2\in\widetilde{\mathcal{H}},

where [h_{f_1}] and [h_{f_2}] are their corresponding equivalence classes in \widetilde{\mathcal{H}}. This yields (\widetilde{m}\circ\mathcal{H},d) a separable metric space. By construction, the metric d is controlled by the RKHS norm.

Finally, we show that the metric d dominates the sup-norm on \widetilde{m}\circ\mathcal{H}. For any f_1,f_2\in \widetilde{m}\circ\mathcal{H}, let [h_{f_1}] and [h_{f_2}] be their corresponding equivalence classes, and take any representatives h_{f_1}\in [h_{f_1}] and h_{f_2}\in [h_{f_2}]. Then, for any o\in\mathcal{O}, we have

\begin{aligned} |f_1(o)-f_2(o)|:=|\widetilde{m}(o;h_{f_1})-\widetilde{m}(o;h_{f_2})|&=|h_{f_1}(q(a),l,w)-h_{f_2}(a,l,w)|\le k\Vert h_{f_1}-h_{f_2}\Vert_\mathcal{H}. \end{aligned}

Since the representatives were arbitrary, we conclude that \sup_{o\in\mathcal{O}} |f_1(o)-f_2(o)|\le k\cdot \tilde d([h_1],[h_2])=k\cdot d(f_1,f_2), implying that d dominates the metric induced by the sup-norm with constant B_3=k. This completes the verification of part 2 of Assumption ?.

Proof

Proof (Deriving the adjoint of T_\mc{G} \circ \mc{T} : \mc{H} \rightarrow \mc{G})

G0QT

For h\in\mathcal{H} and g\in\mathcal{G}, applying Lemma 3(g) with the integral operators induced by K_\mathcal{H} and K_\mathcal{G} yields \left\langle T_\mathcal{G} \circ \mathcal{T} h,g \right\rangle_\mathcal{G}=\langle \mathcal{T}h,g\rangle_{\mathcal{L}^2(P_Z)}=\langle h,\mathcal{T}^*g\rangle_{\mathcal{L}^2(P_W)}=\left\langle h,T_\mathcal{H}\circ \mathcal{T}^*g\right\rangle_\mathcal{H}. Hence, the adjoint of T_\mathcal{G} \circ \mathcal{T} is T_\mathcal{H} \circ \mathcal{T}^*.

Uses Lemma 3
Lemma

Lemma 3

1TV7

Suppose Assumption 2 holds. The following holds:

  1. a) \mathcal{N}(T_{\mathcal{H}})=\mathcal{N}(T_{\mathcal{H}}^{1/2}).

  2. b) For any h\in \mathcal{L}^2(P_{0,W}), it holds that T_{\mathcal{H}}^{1/2}h\in\mathcal{H}.

  3. c) The linear operator T_{\mathcal{H}}^{1/2}:\mathcal{L}^2(P_{0,W})\rightarrow \mathcal{H} is bounded and satisfies \Vert\mathcal{T}_{\mathcal{H}}^{1/2}h\Vert_{\mathcal{H}} \leq \Vert h\Vert_2.

  4. d) The restriction of T_{\mathcal{H}}^{1/2} to \mathcal{N}(T_\mathcal{H})^\perp is an isometry, i.e. \left\Vert T_{\mathcal{H}}^{1/2} h\right\Vert_{\mathcal{H}}=\Vert h\Vert_2 for all h\in\mathcal{N}(T_\mathcal{H})^\perp.

  5. e) The operator T_{\mathcal{H}}^{1/2}:\mathcal{N}(T_\mathcal{H})^\perp \rightarrow\mathcal{H} is bijective.

  6. f) The operator T_{\mathcal{H}}^{1/2}:\mathcal{L}^2(P_{0,W}) \rightarrow \mathcal{H}, with its range endowed with the \mathcal{L}^2(P_{0,W})–norm, is compact.

  7. g) For any h'\in\mathcal{L}^2(P_W) and h\in\mathcal{H}, it holds that \langle T_\mathcal{H}h',h\rangle_\mathcal{H}=\langle h',h\rangle_{\mathcal{L}^2(P_W)}.

Used by Proof, Proof, Proof, Proof
Proof of Lemma 3

As discussed in Section ?, Assumption 2 implies that the integral operator T_\mathcal{H}:\mathcal{L}^2(P_{0,W})\mapsto\mathcal{L}^2(P_{0,W}) is compact and positive. Therefore, by the spectral decomposition theorem (see, e.g., Theorem 4.3 of [hohage2002lecture]) there exists eigenfunctions (\varphi_j)_{j=1}^N corresponding to all the non-zero eigenvalues (\eta_j)_{j=1}^N of T_{\mathcal{H}}, where N\in\mathbb{N}\cup\left\{+\infty\right\}. Such eigenfunctions are orthonormal in \mathcal{L}^2(P_{0,W}) and satisfy that for any h \in \mathcal{L}^2(P_{0,W}), it holds that T_{\mathcal{H}}h=\sum_{j=1}^N\eta_j\langle h,\varphi_j\rangle_2\varphi_j. Because the integral operator T_\mathcal{H} is positive, its non-zero eigenvalues (\eta_j)_{j=1}^N are positive. When N=\infty, \eta_j \rightarrow 0 as j \rightarrow \infty and the sequence (\eta_j)_{j=1}^N is arranged in decreasing order. The square root of T_{\mathcal{H}}, denoted by T_{\mathcal{H}}^{1/2}, is defined by T_{\mathcal{H}}^{1/2}h:=\sum_{j=1}^N \sqrt{\eta_j}\langle h,\varphi_j\rangle_2\varphi_j.

Assertion (a) follows from the fact that for any h\in\mathcal{L}^2(P_{0,W}): T_{\mathcal{H}}h=\sum_{j=1}^N\eta_j\langle h,\varphi_j\rangle_2\varphi_j=0 iff \langle h,\varphi_j\rangle_2=0\:\forall j=1,\dots,N iff T_{\mathcal{H}}^{1/2}h=\sum_{j=1}^N\sqrt{\eta_j}\langle h,\varphi_j\rangle_2\varphi_j=0.

Another result (see Theorem 11.18 of [paulsen2016introduction]), establishes that under Assumption 2, the sequence (\sqrt{\eta_j}\varphi_j)_{j=1}^N forms an orthogonal basis of the RKHS \mathcal{H}. Furthermore, \mathcal{H} admits the representation

\begin{aligned} \mathcal{H}=\left\{h=\sum_{j=1}^N\gamma_j\varphi_j \: : \: \text{ for some } (\gamma_j)_{j=1}^{\infty}\text{ with }\sum_{j=1}^N\frac{\gamma_j^2}{\eta_j}<\infty\right\}, \end{aligned}

where if N=\infty, the limit of the finite sums is with respect to the \mathcal{L}^2(P_{0,W})-metric. Additionally, its inner product decomposes as (see Theorem 11.3 of [paulsen2016introduction]) \langle h,h'\rangle_{\mathcal{H}}=\sum_{j=1}^N\langle h,\varphi_j\rangle_2\langle h',\varphi_j\rangle_2/\eta_j.

Assertion (b) follows from the representation of \mathcal{H} in (?) and the definition of T_{\mathcal{H}}^{1/2} h. Specifically, for any h \in \mathcal{L}^2(P_{0,W}), we have T_{\mathcal{H}}^{1/2}h=\sum_{j=1}^N\sqrt{\eta_j}\langle h,\varphi_j\rangle_2\varphi_j, so to show that T_{\mathcal{H}}^{1/2}h is in \mathcal{H} we need to show that \sum_{j=1}^N(\sqrt{\eta_j}\langle h,\varphi_j\rangle_2)^2/\eta_j < \infty. This holds because

\sum_{j=1}^N\frac{\left(\sqrt{\eta_j}\langle h,\varphi_j\rangle_2\right)^2}{\eta_j}=\sum_{j=1}^N\langle h,\varphi_j\rangle_2^2\le \Vert h\Vert_2^2<\infty.

Assertion (c) follows from (?) since \Vert T^{1/2}_\mathcal{H}h\Vert_\mathcal{H}^2 = \sum_{j=1}^N\left(\sqrt{\eta_j}\langle h,\varphi_j\rangle\right)^2/\eta_j, thus proving that T^{1/2}_\mathcal{H} is bounded with operator norm at most one. The linearity of T^{1/2}_\mathcal{H} is immediate from the linearity of the inner products \langle h, \varphi_j \rangle_2, completing the proof of (c).

To show Assertions (d) and (e), we use that \mathcal{N}(T_\mathcal{H})^\perp=\text{span}(\varphi_1,\dots,\varphi_N). Assertion (d) follows because, for h\in\text{span}(\varphi_1,\dots,\varphi_N), the first inequality in (?) becomes an equality.

To prove Assertion (e), we first show that T_{\mathcal{H}}^{1/2}:\mathcal{N}(T_\mathcal{H})^\perp \rightarrow \mathcal{H} is injective. Let h_1,h_2\in\mathcal{N}(T_\mathcal{H})^\perp be such that T_\mathcal{H}^{1/2}h_1=T_\mathcal{H}^{1/2}h_2. Then, h_1-h_2\in\mathcal{N}(T_\mathcal{H}^{1/2}). By part (a), \mathcal{N}(T_\mathcal{H}^{1/2}) = \mathcal{N}(T_\mathcal{H}). Then, h_1-h_2\in\mathcal{N}(T_\mathcal{H}) \cap \mathcal{N}(T_\mathcal{H})^\perp =\{0\}. This shows injectivity. Surjectivity follows from the representation of \mathcal{H} in (?). Specifically, any h\in\mathcal{H} can be written as h=\sum_{j=1}^N\gamma_j\varphi_j with \sum_{j=1}^N\frac{\gamma_j^2}{\eta_j}<\infty. Then T_\mathcal{H}^{1/2}h'=h where h':=\sum_{j=1}^N\gamma_j\varphi_j/\sqrt{\eta_j}\in\mathcal{N}(T_\mathcal{H})^\perp.

Assertion (f) follows from the definition of \mathcal{T}_{\mathcal{H}}^{1/2}:\mathcal{L}^2(P_{0,W})\mapsto\mathcal{L}^2(P_{0,W}), as it is the norm-limit of finite-rank operators (see Proposition 5.6.5 of [conway2012course]).

To prove assertion (g), observe that for any h'\in\mathcal{L}^2(P_W) and h\in\mathcal{H}, it holds that \langle T_\mathcal{H}h',h\rangle_\mathcal{H}=\sum_{j=1}^N\langle T_\mathcal{H}h',\varphi_j\rangle_2\langle h,\varphi_j\rangle_2/\eta_j=\sum_{j=1}^N\langle h',\varphi_j\rangle_2\langle h,\varphi_j\rangle_2=\langle h',\sum_{j=1}^N\langle h,\varphi_j\rangle_2\varphi_j\rangle_2. For every h\in\mathcal{H}, h=\sum_{j=1}^N\langle h,\varphi_j\rangle_2\varphi_j. Hence, the last expression equals \langle h',h\rangle_2.

Lemma

Lemma 4

FUSJ

Suppose Assumption 2 holds. Then the operators I_0^{op}\circ\mathcal{T}\mid_\mathcal{H}:\mathcal{H} \rightarrow \mathcal{L}^2(P_{Z}) and I_0^{op}\circ\mathcal{T}\circ T_\mathcal{H}^{1/2}:\mathcal{L}^2(P_{W})\rightarrow\mathcal{L}^2(P_{Z}) are bounded and linear, and their adjoints (I_0^{op}\circ\mathcal{T}\mid_\mathcal{H})^*:\mathcal{L}^2(P_Z)\rightarrow\mathcal{H} and (I_0^{op}\circ\mathcal{T}\circ T_\mathcal{H}^{1/2})^*:\mathcal{L}^2(P_Z)\rightarrow\mathcal{L}^2(P_W) are given by T_\mathcal{H}\circ\mathcal{T}^*\circ I_0^{op} and T_\mathcal{H}^{1/2}\circ\mathcal{T}^*\circ I_0^{op}.

Proof of Lemma 4

Linearity is immediate from that of I_0^{op}, \mathcal{T}, and T_\mathcal{H}^{1/2}. Boundedness of I_0^{op}\circ\mathcal{T}\mid_\mathcal{H} follows because for h \in \mathcal{H}, it holds that \Vert I_0^{op}\mathcal{T}h\Vert_2^2\le\Vert r\Vert_\infty^2\Vert h\Vert_2^2\le\Vert r\Vert_\infty^2\Vert h\Vert_\infty^2\le k_1^2\Vert r\Vert_\infty^2\Vert h\Vert_\mathcal{H}^2, where k_1^2=\sup_{w\in\mathcal{W}}K_\mathcal{H}(w,w)<\infty. Boundedness of I_0^{op}\circ\mathcal{T}\circ T_\mathcal{H}^{1/2} follows because T_\mathcal H^{1/2}(by Lemma 3(f)), \mathcal T, and I_0^{op} are all bounded, and so is their composition.

To determine the adjoint of I_0^{op}\circ\mathcal{T}\mid_\mathcal{H}, let h\in\mathcal{H} and g\in\mathcal{L}^2(P_Z). Then, applying Lemma 3(g) yields \langle I_0^{op}\circ\mathcal{T}h,g\rangle_{\mathcal{L}^2(P_Z)}=\langle \mathcal{T}h,I_0^{op}g\rangle_{\mathcal{L}^2(P_Z)}=\left\langle h,\mathcal{T}^*\circ I_0^{op}g\right\rangle_{\mathcal{L}^2(P_W)}=\left\langle h,T_\mathcal{H}\circ\mathcal{T}^*\circ I_0^{op}g\right\rangle_\mathcal{H}. Hence, (I_0^{op}\circ\mathcal{T}\mid_\mathcal{H})^*=T_\mathcal{H}\circ\mathcal{T}^*\circ I_0^{op}. Now let h\in\mathcal{L}^2(P_W) and g\in\mathcal{L}^2(P_Z). Since T_\mathcal{H}^{1/2} is self-adjoint, we have \langle I_0^{op}\circ\mathcal{T}\circ T_\mathcal{H}^{1/2}h,g\rangle_{\mathcal{L}^2(P_Z)}=\langle T_\mathcal{H}^{1/2}h,\mathcal{T}^*\circ I_0^{op}g\rangle_{\mathcal{L}^2(P_W)}=\langle h,T_\mathcal{H}^{1/2}\circ\mathcal{T}^*\circ I_0^{op}g\rangle_{\mathcal{L}^2(P_W)}. Thus (I_0^{op}\circ\mathcal{T}\circ T_\mathcal{H}^{1/2})^*=T_\mathcal{H}^{1/2}\circ\mathcal{T}^*\circ I_0^{op}.

Uses Lemma 3
Lemma

Lemma 5

D3CE

The second equality in display (?) holds.

Proof of Lemma 5

To show the second equality in (?) it suffices to show that \Vert h^{\dag\dag}_{\lambda_\mathcal{H},KRAS}\Vert_\mathcal{H}\le B where h^{\dag\dag}_{\lambda_\mathcal{H},KRAS}:=\arg\min_{h\in\mathcal{H}}\left\{\frac{1}{4}\left\Vert\rho- I_0^{op} \circ \mathcal{T}\vert_\mathcal{H} \:h\right\Vert_2^2+\lambda_{\mathcal{H}}\left\Vert h\right\Vert_{\mathcal{H}}^2\right\}. This holds because \Vert h^{\dag}_{0,KRAS}\Vert_\mathcal{H}<B by definition of B and \Vert h^{\dag\dag}_{\lambda_\mathcal{H},KRAS}\Vert_\mathcal{H}\le \Vert h^{\dag}_{0,KRAS}\Vert_\mathcal{H} because otherwise \frac{1}{4c^2}\Vert I_0\mathcal{T}(h_{0,KRAS}^\dag-h_{0,KRAS}^\dag)\Vert_2^2+\lambda_{\mathcal{H}}\big\Vert h_{0,KRAS}^\dag\big\Vert_{\mathcal{H}}^2<\frac{1}{4c^2}\Vert I_0\mathcal{T}(h_{0,KRAS}^\dag-h_{\lambda_\mathcal{H},KRAS}^{\dag\dag})\Vert_2^2+\lambda_{\mathcal{H}}\big\Vert h_{\lambda_\mathcal{H},KRAS}^{\dag\dag}\big\Vert_{\mathcal{H}}^2 violating the definition of h_{\lambda_\mathcal{H},KRAS}^{\dag\dag}.

Lemma

Lemma 6

RLJ9

Under Assumptions 1 and 2, h_{0,KRAS}^\dag=T_{\mathcal{H}}^{1/2}h'_{0,KRAS} and h'_{0,KRAS}\in\mathcal{N}(T_\mathcal{H})^\perp.

Used by Proof
Proof of Lemma 6

To simplify notation, we drop the subscripts KRAS from h_{0,KRAS}^\dag and h'_{0,KRAS}. Under Assumptions 1–2, h_0^\dag in (?) is well-defined, and \widetilde{\mathcal{T}} is bounded (see Section ?).

By Lemma 3 (e), there exists a unique h_0'\in\mathcal{N}(T_\mathcal{H})^\perp such that h_0^\dag=T_\mathcal{H}^{1/2}h_0'. Since \rho=\mathcal{T}\left(I_0h_0^\dag\right)=I_0^{op}\circ\mathcal{T}h_0^\dag=I_0^{op}\circ\mathcal{T}\circ T_\mathcal{H}^{1/2}h_0'=\widetilde{\mathcal{T}}h_0' it follows that h_0' solves equation (?). It remains to show that it is the solution of minimal \mathcal{L}^2(P_{W})-norm. Let h_1\in\mathcal{L}^2(P_{W}) be any solution to (?). Decompose h_1 as h_1'+h_1^\perp, where h_1'\in\mathcal{N}(T_\mathcal{H}) and h_1^\perp\in\mathcal{N}(T_\mathcal{H})^\perp. Since \widetilde{\mathcal{T}}h_1^\perp =\widetilde{\mathcal{T}}h_1, it follows that h_1^\perp also solves equation (?). Moreover, \rho=\widetilde{\mathcal{T}}h_1^\perp=I_0^{op}\circ\mathcal{T}\circ T_\mathcal{H}^{1/2}h_1^\perp =\mathcal{T}(I_0T_\mathcal{H}^{1/2}h_1^\perp), so T_\mathcal{H}^{1/2}h_1^\perp\in\mathcal{H}_0. By Lemma 3 (d), T_\mathcal{H}^{1/2} restricted to \mathcal{N}(T_\mathcal{H})^\perp is an isometry, so \Vert h_1^\perp\Vert_2=\Vert T^{1/2}_\mathcal{H}h_1^\perp\Vert_\mathcal{H}\ge \Vert h_0^\dag\Vert_\mathcal{H}^2=\Vert h_0'\Vert_2^2 with equality iff T^{1/2}_{\mathcal{H}}h_1^\perp=h_0^\dag=T_\mathcal{H}^{1/2}h_0', which by Lemma 3 (e) implies h_1^\perp=h_0'. Therefore, \Vert h_1\Vert_2^2=\Vert h_1'\Vert_2^2+\Vert h_1^\perp\Vert_2^2\ge \Vert h_0'\Vert_2^2, with equality, i.e., \Vert h_1\Vert_2=\Vert h_0'\Vert_2, if and only if h_1'=0 and h_1^\perp=h_0'. This shows that h_0' is the unique solution to equation (?) of minimal \mathcal{L}^2(P_{W})-norm.

Lemma

Lemma 7 (Talagrand inequality, Theorem 3.27)

BQA3

Suppose \mathcal{F} is countable and \left\{t_f:f\in\mathcal{F}\right\} is uniformly bounded by b. Define Z:=\sup_{f\in\mathcal{F}}\left\{\frac{1}{n}\sum_{i=1}^nt_f(O_i)\right\}. Then, for all u>0, \text{P}\left\{Z\ge\mathbb{E}(Z)+u\right\}\le2\exp\Big(-\frac{nu^2}{16eb\mathbb{E}(Z)+8e\sup_{f\in\mathcal{F}}\mathbb{E}\left\{t_f(O)^2\right\}+4bu}\Big) where e:=exp(1).

Used by Proof
Lemma

Lemma 8

9XKU

Suppose that:

  1. \mathcal{F} is a separable metric space with metric d, and let \mathcal{S}\subset\mathcal{F} be a countable dense set,

  2. For each f\in\mathcal{F}, t_f:\mathcal{O}\mapsto\mathbb{R} is a measurable function, where \mathcal{O} is the sample space of a random vector O,

  3. For each o\in\mathcal{O}, the map f\mapsto t_f(o) is continuous with respect to d at all f \in \mathcal{F}.

Then for all o_1,\dots,o_n\in\mathcal{O}, \sup_{f\in\mathcal{F}}\left\{\frac{1}{n}\sum_{i=1}^nt_f(o_i) \right \} = \sup_{f\in\mathcal{S}}\left\{\frac{1}{n}\sum_{i=1}^nt_f(o_i)\right\} and, consequently, for O_1,\cdots,O_n i.i.d. copies of O, \sup_{f\in\mathcal{F}}\left\{\frac{1}{n}\sum_{i=1}^nt_f(O_i)\right\} is a well-defined random variable.

Used by Proof
Proof of Lemma 8

Fix \underset{\sim}{o}:=(o_1,\dots, o_n)\in\mathcal{O}^n and define g(\underset{\sim}{o};f):=\frac{1}{n}\sum_{i=1}^nt_f(o_i), A_1(\underset{\sim}{o}):=\sup_{f\in \mathcal{F}} g(\underset{\sim}{o};f), and A_2(\underset{\sim}{o}):=\sup_{f\in \mathcal{S}}g(\underset{\sim}{o};f). We will show that A_1(\underset{\sim}{o})=A_2(\underset{\sim}{o}).

Clearly A_2(\underset{\sim}{o})\le A_1(\underset{\sim}{o}) because \mathcal{S}\subset \mathcal{F}. Suppose that A_2(\underset{\sim}{o})<A_1(\underset{\sim}{o}) and set \varepsilon:=A_1(\underset{\sim}{o})-A_2(\underset{\sim}{o})>0. By the definition of the supremum, there exists f_\varepsilon\in\mathcal{F} such that A_1(\underset{\sim}{o})-\frac{\varepsilon}{2}< g(\underset{\sim}{o};f_{\varepsilon}). Now, the map f\mapsto g(\underset{\sim}{o};f) is continuous because the map f\mapsto t_f(o_i) is continuous for each i. Then, there exists \delta>0 such that d(f,f_\varepsilon)<\delta implies |g(\underset{\sim}{o};f)-g(\underset{\sim}{o};f_{\varepsilon})|<\frac{\varepsilon}{2}. Since \mathcal{S} is dense in \mathcal{F}, there exits f_s\in \mathcal{S} such that d(f_s,f_\varepsilon)<\delta. Then, g(\underset{\sim}{o};f_s)>g(\underset{\sim}{o};f_{\varepsilon})-\frac{\varepsilon}{2}>A_1(\underset{\sim}{o})-\varepsilon=A_2(\underset{\sim}{o}), which contradicts the definition of A_2(\underset{\sim}{o}). We thus conclude that A_1(\underset{\sim}{o})=A_2(\underset{\sim}{o}).

Next, since g(\cdot;f):\mathcal{O}^n\mapsto\mathbb{R} is measurable for all f\in \mathcal{S}, and the map f\mapsto g(\underset{\sim}{o};f) is continuous for all \underset{\sim}{o}\in\mathcal{O}^n, Lemma A.3.17 of [steinwart2008support] ensures that the mapping (\underset{\sim}{o},f)\mapsto g(\underset{\sim}{o};f) is measurable on \mathcal{O}^n\times\mathcal{S}. Then, the map \underset{\sim}{o}\mapsto\sup_{f\in\mathcal{S}}g(\underset{\sim}{o};f) is also measurable because \mathcal{S} is countable. Hence, \sup_{f\in\mathcal{S}}g(O_1,\dots,O_n;f)= \sup_{f\in\mathcal{F}}g(O_1,\dots,O_n;f) is a well-defined random variable. This completes the proof.

Corollary

Corollary 3

VRFO

Let \mathcal{O}_0\subset\mathcal{O} be a measurable set. Suppose that:

  1. \mathcal{F} is a separable metric space with metric denoted by d such that \mathcal{F}\subset\mathcal{L}^2(P_O), and let \mathcal{S} denote a countable dense subset of it,

  2. For each f\in\mathcal{L}^2(P_O), t_f:\mathcal{O}\mapsto\mathbb{R} is a measurable function, where \mathcal{O} is the sample space of a random vector O,

  3. For each o\in\mathcal{O}, the map f\mapsto t_{I_{\mathcal{O}_0}\cdot f}(o) is continuous with respect to d at all f \in \mathcal{F}.

  4. The map f\mapsto t_{I_{\mathcal{O}_0}\cdot f} is continuous from (\mathcal{F},d) into \mathcal{L}^1(P_O).

For given fixed \beta\in\mathbb{R} and any o\in\mathcal{O}, define the map f\in\mathcal{F}\mapsto h_f(o):= t_{I_{\mathcal{O}_0}\cdot f}(o)-\beta\cdot\mathbb{E}\{t_{I_{\mathcal{O}_0}\cdot f}(O)\}. Given \mathcal{F}'\subset\mathcal{F}, define S':=\mathcal{F}'\cap\mathcal{S}. Then,

  1. i) \sup_{f\in \mathcal{F}'}\left\vert \frac{1}{n}\sum_{i=1}^nh_f(O_i) \right\vert = \sup_{f \in \mathcal{S}'}\left\vert \frac{1}{n}\sum_{i=1}^nh_f(O_i) \right\vert. In particular, the left-hand side is a well-defined random variable.

  2. ii) When \beta=0, the conclusion in i) holds even if assumption 4 fails.

Used by Lemma 9, Proof
Proof of Corollary 3

Fix o\in\mathcal{O}. First, we show that the map f\mapsto h_f(o) is continuous with respect to the metric d. Take f_1,f_2\in\mathcal{F}. By triangle inequality, we have \vert h_{f_1}(o)-h_{f_2}(o)\vert\le \vert t_{I_{\mathcal{O}_0}\cdot f_1}(o)-t_{I_{\mathcal{O}_0}\cdot f_2}(o)\vert +|\beta|\cdot\mathbb{E}_P\{\vert t_{I_{\mathcal{O}_0}\cdot f_1}(O)-t_{I_{\mathcal{O}_0}\cdot f_2}(O)\vert\}.

Let \varepsilon>0. By assumption 3, there exists \delta>0 such that if d(f_1,f_2)<\delta, then \left\vert t_{I_{\mathcal{O}_0}\cdot f_1}(o)-t_{I_{\mathcal{O}_0}\cdot f_2}(o)\right\vert <\varepsilon/2. By assumption 4, there exists \delta'>0 such that if d(f_1,f_2)<\delta', then |\beta|\cdot\mathbb{E}_P\{\vert t_{I_{\mathcal{O}_0}\cdot f_1}(O)-t_{I_{\mathcal{O}_0}\cdot f_2}(O)\vert\}<\varepsilon/2. Then, if d(f_1,f_2)<\min\{\delta,\delta'\}, we have \vert h_{f_1}(o)-h_{f_2}(o)\vert<\varepsilon.

By Assumption 1 and definition of \mathcal{F}', \mathcal{F}' is separable with respect to d with dense subset \mathcal{S}'. Then, it follows from Lemma 8 that

\sup_{f\in \mathcal{F}'} \Big\{\frac{1}{n}\sum_{i=1}^nh_f(O_i) \Big\} = \sup_{f \in \mathcal{S}'} \Big\{\frac{1}{n}\sum_{i=1}^nh_f(O_i) \Big\},\: \sup_{f\in \mathcal{F}'} \Big\{-\frac{1}{n}\sum_{i=1}^nh_f(O_i) \Big\} = \sup_{f \in \mathcal{S}'} \Big\{-\frac{1}{n}\sum_{i=1}^nh_f(O_i) \Big\},

and the left hand sides of both equalities are well-defined random variables. Therefore

\begin{aligned} \sup_{f\in \mathcal{F}'} \Big\vert\frac{1}{n}\sum_{i=1}^nh_f&(O_i) \Big\vert = \max\Bigg\{\sup_{f\in \mathcal{F}'} \Big[\frac{1}{n}\sum_{i=1}^nh_f(O_i) \Big],\sup_{f\in \mathcal{F}'} \Big[-\frac{1}{n}\sum_{i=1}^nh_f(O_i) \Big]\Bigg\} \\ &=\max\Bigg\{\sup_{f\in \mathcal{S}'} \Big[\frac{1}{n}\sum_{i=1}^nh_f(O_i) \Big],\sup_{f\in \mathcal{S}'} \Big[-\frac{1}{n}\sum_{i=1}^nh_f(O_i) \Big]\Bigg\}=\sup_{f\in \mathcal{S}'} \Big\vert\frac{1}{n}\sum_{i=1}^nh_f(O_i) \Big\vert \end{aligned}

and the leftmost hand side is a well-defined random variable. This concludes the proof.

Lemma

Lemma 9

1QXU

Suppose that:

  1. The assumptions of Corollary 3 hold with \mathcal{F} satisfying: (1.1) \mathcal{F} is uniformly bounded by b, (1.2) for each o\in\mathcal{O}, the evaluation functional f\mapsto f(o) is continuous relative to d, and (1.3) f=0 belongs to \mathcal{F}.

  2. t_f satisfies t_f=0 when f=0.

  3. There exists a constant L such that for any f_1,f_2\in\mathcal{L}^2(P_O), |t_{f_1}(o)-t_{f_2}(o)|\le L\cdot\left\vert f_1(o)-f_2(o)\right\vert for all o\in\mathcal{O} (o\in\mathcal{O}_0).

Define h_f as in Corollary 3 with \beta = 1. For any r>0 define \mathcal{F}_r:=\left\{f \in \mathcal{F}\::\:\Vert I_{\mathcal{O}_0}\cdot f\Vert_2 \le r \right\} and Z_n(r):=\sup_{f\in \mathcal{F}_r} \big\vert\frac{1}{n}\sum_{i=1}^nh_f(O_i) \big\vert. Then, Z_n(r) is a well-defined random variable and for all u>0, \text{P}\left\{Z_n(r)\ge 4L\mathcal{R}_n\left(I_{\mathcal{O}_0}\cdot\mathcal{F}, r\right)+u\right\}\le 2\exp\Big(-\frac{nu^2}{8eL^2r^2+128ebL^2\mathcal{R}_n\left(I_{\mathcal{O}_0}\cdot \mathcal{F},r\right)+8bLu}\Big), where e=\exp(1) and \mathcal{R}_n(I_{\mathcal{O}_0}\cdot\mathcal{F},r) is the localized Rademacher complexity of radius r of the function class I_{\mathcal{O}_0}\cdot\mathcal{F}:=\{I_{\mathcal{O}_0}\cdot f\:|\:f\in\mathcal{F}\}. Moreover, if \delta_n is any solution to the inequality \mathcal{R}_n\left(star\left(I_{\mathcal{O}_0}\cdot\mathcal{F}\right),\delta\right)\le\frac{\delta^2}{b}, then for each r\ge\delta_n, \text{P}\{Z_n(r)\ge\frac{4Lr\delta_n}{b}+u\}\le 2\exp\Big(-\frac{nu^2}{8eL^2r^2+128eL^2r\delta_n+8bLu}\Big).

Proof of Lemma 9

By assumption 1, the conclusion of Corollary 3 holds; i.e., Z_n(r) is well-defined. Furthermore, Z_n(r)=\sup_{f\in \mathcal{S}_r} \big\vert\frac{1}{n}\sum_{i=1}^nh_f(O_i) \big\vert. where \mathcal{S}_r := \mathcal{S} \cap \mathcal{F}_r. Now define the function class \mathcal{C}:=\left\{\alpha h_f\::\:\alpha\in\{-1,1\}, f\in\mathcal{S}_r\right\} which is countable by construction. With \mathcal{C} so defined we have that Z_n(r)=\sup_{h\in\mathcal{C}}\frac{1}{n}\sum_{i=1}^nh(O_i).

For any h\in\mathcal{C}, write h=\alpha h_f for some \alpha\in\{-1,1\} and f\in\mathcal{S}_r. Letting f'=0, we have

\begin{aligned} |h(o)|&\le |t_{I_{\mathcal{O}_0}\cdot f}(o)|+\mathbb{E}\big\{\big\vert t_{I_{\mathcal{O}_0}\cdot f}(O)\big\vert\big\}\\ &=|t_{I_{\mathcal{O}_0}\cdot f}(o)-t_{I_{\mathcal{O}_0}\cdot f'}(o)|+\mathbb{E}\big\{\big\vert t_{I_{\mathcal{O}_0}\cdot f}(O)-t_{I_{\mathcal{O}_0}\cdot f'}(O)\big\vert\big\}\\ &\le L\cdot I_{\mathcal{O}_0}(o)\big\vert f(o)-f'(o)\big\vert + L\cdot \mathbb{E}\big\{I_{\mathcal{O}_0}(O)\big\vert f(O)-f'(O)\big\vert \big\}\le 2Lb, \end{aligned}

where the first inequality is by the triangle inequality, the first and second equalities are by assumptions 1.3 and 2, the second inequality is by assumption 3, and the third inequality is by assumption 1.1. We thus conclude that \mathcal{C} is uniformly bounded by 2Lb. Since \mathcal{C} is countable, applying Talagrand inequality (Lemma 7) we obtain that, for any u>0,

\text{P}\left[Z_n(r)\ge \mathbb{E}\left\{Z_n(r)\right\}+u\right]\le 2\exp\left(-\frac{nu^2}{8e\sup_{h\in\mathcal{C}}\mathbb{E}[h(O)^2]+32ebL\mathbb{E}\left\{Z_n(r)\right\}+8bLu}\right).

Again, take h\in\mathcal{C} and write h=\alpha h_f for some \alpha\in\{-1,1\} and f\in\mathcal{S}_r. Thus \mathbb{E}[h(O)^2]=\text{var}[t_{I_{\mathcal{O}_0}\cdot f}(O)]\le \mathbb{E}[\{ t_{I_{\mathcal{O}_0}\cdot f}(O)\}^2]\le L^2\mathbb{E}[\{I_{\mathcal{O}_0}(O)\cdot f(O)\}^2]\le L^2r^2, where the second inequality follows by invoking assumption 3 applied to f_1=I_{\mathcal{O}_0}\cdot f and f_2=0 and noting that by assumption 2, t_{f_2}=0, and the third inequality is because f\in\mathcal{F}_r.

We now bound \mathbb{E}\{Z_n(r)\}=\mathbb{E}\{\sup_{f\in\mathcal{S}_r}\vert\frac{1}{n}\sum_{i=1}^n[t_{I_{\mathcal{O}_0}\cdot f}(O_i)-\mathbb{E}\{t_{I_{\mathcal{O}_0}\cdot f}(O)\}]\vert\} using a symmetrization argument. Letting \{O'_i\}_{i=1}^n be independent copies of O and independent of \{O_i\}_{i=1}^n, we can write \mathbb{E}\{Z_n(r)\}=\mathbb{E}\{\sup_{f\in\mathcal{S}_r}\vert\mathbb{E}[\frac{1}{n}\sum_{i=1}^n\{t_{I_{\mathcal{O}_0}\cdot f}(O_i)-t_{I_{\mathcal{O}_0}\cdot f}(O_i')\}]\vert\}. By the inequality \sup_{d\in\mathcal{D}}\mathbb{E}\left\{|d(O)|\right\}\le\mathbb{E}\left\{\sup_{d\in\mathcal{D}}|d(O)|\right\}, which is valid for any countable class \mathcal{D} of real-valued measurable functions, we conclude \mathbb{E}\{Z_n(r)\}\le\mathbb{E}[\sup_{f\in\mathcal{S}_r}\vert\frac{1}{n}\sum_{i=1}^n\{t_{I_{\mathcal{O}_0}\cdot f}(O_i)-t_{I_{\mathcal{O}_0}\cdot f}(O_i')\}\vert].

Now, let \varepsilon_1,\dots,\varepsilon_n be n i.i.d. Rademacher random variables, independent of O_i and O'_i for all i=1,\dots,n. Given this independence assumption, for any function f\in\mathcal{F}, the n \times 1 random vector with i^{th} component \varepsilon_i\{t_{I_{\mathcal{O}_0}\cdot f}(O_i)-t_{I_{\mathcal{O}_0}\cdot f}(O_i')\} has the same joint distribution as the n \times 1 random vector with i^{th} component t_{I_{\mathcal{O}_0}\cdot f}(O_i)-t_{I_{\mathcal{O}_0}\cdot f}(O_i'). Hence, \mathbb{E}[\sup_{f\in\mathcal{S}_r}\vert\frac{1}{n}\sum_{i=1}^n\varepsilon_i\{t_{I_{\mathcal{O}_0}\cdot f}(O_i)-t_{I_{\mathcal{O}_0}\cdot f}(O_i')\}\vert]\le2\mathbb{E}\{\sup_{f\in\mathcal{S}_r}\vert\frac{1}{n}\sum_{i=1}^n\epsilon_i t_{I_{\mathcal{O}_0}\cdot f}(O_i)\vert\}\\=2\mathbb{E}\{\sup_{f\in\mathcal{S}'_r}\vert\frac{1}{n}\sum_{i=1}^n\epsilon_i t_f(O_i)\vert\}. where \mathcal{S}_r':=\left\{I_{\mathcal{O}_0}\cdot f:f\in\mathcal{S}_r\right\}. Next, invoking the Ledoux-Talagrand contraction inequality (see inequality (5.61) of [wainwright2019high]), we obtain 2\mathbb{E}\{\sup_{f\in\mathcal{S}'_r}\vert\frac{1}{n}\sum_{i=1}^n\epsilon_i t_f(O_i)\vert\}\le 4L\mathbb{E}\{\sup_{f\in\mathcal{S}'_r}\vert\frac{1}{n}\sum_{i=1}^n\epsilon_i f(O_i)\vert\}\\=4L\mathbb{E}\{\sup_{f\in\mathcal{S}_r}\vert\frac{1}{n}\sum_{i=1}^n\epsilon_i I_{\mathcal{O}_0}(O_i)\cdot f(O_i)\vert\}=4L\mathbb{E}\{\sup_{f\in\mathcal{F}_r}\vert\frac{1}{n}\sum_{i=1}^n\epsilon_i I_{\mathcal{O}_0}(O_i)\cdot f(O_i)\vert\}=4L \mathcal{R}_n\left(I_{\mathcal{O}_0}\cdot\mathcal{F},r\right) where the second equality follows by applying Corollary 3 to the functions t_f(o,\varepsilon):=\varepsilon f(o) and \beta=0, noting that assumption 3 of the corollary holds because the map f\mapsto f(o) is continuous (assumption 1.2). Thus, \mathbb{E}\left\{Z_n(r)\right\}\le4L\cdot \mathcal{R}_n\left(I_{\mathcal{O}_0}\cdot\mathcal{F},r\right). Therefore, \text{P}[Z_n(r)\ge 4L\cdot \mathcal{R}_n(I_{\mathcal{O}_0}\cdot\mathcal{F},r)+u]\le \text{P}[Z_n(r)\ge \mathbb{E}\{Z_n(r)\}+u]\\ \le2\exp\Big(-\frac{nu^2}{8eL^2r^2+32ebL\mathbb{E}\left\{Z_n(r)\right\}+8bLu}\Big)\le 2\exp\Big(-\frac{nu^2}{8eL^2r^2+128ebL^2\mathcal{R}_n(I_{\mathcal{O}_0}\cdot\mathcal{F},r)+8bLu}\Big).

Because star\left(I_{\mathcal{O}_0}\cdot\mathcal{F}\right) is star-shaped, by the arguments in the proof of Lemma 13.6 of [wainwright2019high], the function \delta\mapsto\frac{\mathcal{R}_n\left(star\left(I_{\mathcal{O}_0}\cdot\mathcal{F}\right),\delta\right)}{\delta} is non-decreasing on the interval (0,\infty). Also, since I_{\mathcal{O}_0}\cdot\mathcal{F}\subset star\left(I_{\mathcal{O}_0}\cdot\mathcal{F}\right), then, for any r\ge\delta_n, it holds \mathcal{R}_n\left(I_{\mathcal{O}_0}\cdot\mathcal{F},r\right)/r\le\mathcal{R}_n\left(star\left(I_{\mathcal{O}_0}\cdot\mathcal{F}\right),r\right)/r\le\mathcal{R}_n\left(star\left(I_{\mathcal{O}_0}\cdot\mathcal{F}\right),\delta_n\right)/\delta_n\le\frac{\delta_n}{b}. Then, \text{P}\{Z_n(r)\ge \frac{4Lr\delta_n}{b}+u\}\le\text{P}\{Z_n(r)\ge 4L \mathcal{R}_n(I_{\mathcal{O}_0}\cdot\mathcal{F},r)+u\}\le 2\exp\Big(-\frac{nu^2}{8eL^2r^2+128eL^2r\delta_n+8bLu}\Big).

Lemma

Lemma 10

QLCH

Suppose the assumptions of Lemma 9 hold. Let h_f and Z_n(r) be defined as in Lemma 9 and let \delta_n be any solution to \mathcal{R}_n\left(star\left(I_{\mathcal{O}_0}\cdot\mathcal{F}\right),\delta\right)\le\frac{\delta^2}{b}. For fixed \theta>1 define the event \mathcal{E}:=\{\exists f\in\mathcal{F}:\Vert I_{\mathcal{O}_0}\cdot f\Vert_2\ge\delta_n\text{ and } \big\vert\frac{1}{n}\sum_{i=1}^nh_f(O_i) \big\vert\ge \frac{5\theta L\delta_n}{b}\left\Vert I_{\mathcal{O}_0}\cdot f\right\Vert_2\}. Also, set c_1=\left\{8b^2(1+17e)\right\}^{-1}. Then, \text{P}(\mathcal{E})\le2\{ 1+I\{\delta_n<b/\theta\}\lfloor\{\log(\theta)\}^{-1}\log(b/\delta_n)\rfloor\} \:\exp(-c_1n\delta_n^2) where \lfloor x \rfloor denotes the greatest integer smaller than or equal to x.

Uses Lemma 9 · Used by Proof
Proof of Lemma 10

This proof uses a peeling argument as in the proof of Lemma 12 of [foster2023orthogonal]. Since \Vert I_{\mathcal{O}_0}\cdot f\Vert_2\le \Vert f\Vert_{2}\le\Vert f\Vert_{\infty}, Assumption 1.1 of Lemma 9 implies that \Vert I_{\mathcal{O}_0}\cdot f\Vert_{2}\le b for any f\in\mathcal{F}. Then, any f\in\mathcal{F} satisfying \Vert I_{\mathcal{O}_0}\cdot f\Vert_2\ge \delta_n belongs to the set \mathcal{S}_m:=\{f\in\mathcal{F}\text{ }\::\:\text{ }\theta^{m-1}\delta_n\le\Vert I_{\mathcal{O}_0}\cdot f\Vert_2\le \theta^m\delta_n\} for some m\in\left\{1,\cdots,M\right\} where M is the smallest integer satisfying b\le \theta^M \delta_n. We find such M as follows. If \delta_n\ge b/\theta then M=1 is such constant because \theta>1. If \delta_n<b/\theta then b\le \theta^M \delta_n iff \log(b/\delta_n)\le M\log(\theta) iff \left\{\log(\theta)\right\}^{-1}\log(b/\delta_n)\le M. Therefore, we conclude that M=1+I\{\delta_n<b/\theta\}\lfloor\{\log(\theta)\}^{-1}\log(b/\delta_n)\rfloor. Note that M is a positive integer because \theta>1 and b/\delta_n>\theta when \delta_n<b/\theta.

Now, if the event \mathcal{E}\cap\mathcal{S}_m occurs for some m\in\{1,\dots,M\}, then there exists f with \theta^{m-1}\delta_n\le\Vert I_{\mathcal{O}_0}\cdot f\Vert_2\le r_m:=\theta^m\delta_n such that \vert\frac{1}{n}\sum_{i=1}^nh_f(O_i) \vert\ge \frac{5\theta L\delta_n}{b}\left\Vert I_{\mathcal{O}_0}\cdot f\right\Vert_2 \ge \frac{5\theta L\delta_n}{b} \frac{r_m}{\theta} =5 \frac{L\delta_n r_m}{b}. Consequently, we have \text{P}\left(\mathcal{E}\cap\mathcal{S}_m\right)\le\text{P}\left[Z_n(r_m)\ge5 \frac{L\delta_n r_m}{b}\right]. Applying Lemma 9 with r=r_m and u=\frac{L\delta_nr_m}{b}, and setting c_1=\left\{8b^2(1+17e)\right\}^{-1}, we obtain that \text{Pr}\left(\mathcal{E}\cap\mathcal{S}_m\right) happens with probability at most 2\exp\Big\{-\frac{nL^2r_m^2\delta_n^2}{8b^2(eL^2r_m^2+16eL^2\delta_nr_m+L^2\delta_nr_m)}\Big\} \le 2\exp\left(-c_1n\delta_n^2\right) where the inequality holds since \delta_n\le r_m. Then \text{P}(\mathcal{E})\le\sum_{m=1}^M\text{P}(\mathcal{E}\cap\mathcal{S}_m)\le 2M\exp(-c_1 n\delta_n^2)=2\{ 1+I\{\delta_n<b/\theta\}\lfloor\{\log(\theta)\}^{-1}\log(b/\delta_n)\rfloor\}\exp\left(-c_1n\delta_n^2\right).

Uses Lemma 9
Lemma

Lemma 11

RHNC

Suppose the assumptions of Lemma 9 hold. Let h_f and Z_n(r) be defined as in Lemma 9 and let \delta_n be any solution to the inequality \mathcal{R}_n\left(star\left(I_{\mathcal{O}_0}\cdot\mathcal{F}\right),\delta\right)\le\frac{\delta^2}{b}. Set c_1=\left\{8b^2(1+17e)\right\}^{-1}. Then,

\big\vert\frac{1}{n} \sum_{i=1}^n h_f(O_i) \big\vert \le \frac{6L\delta_n}{b} \left\{ \left\Vert I_{\mathcal{O}_0}\cdot f \right\Vert_2 + \delta_n \right\} \quad \forall f\in\mathcal{F}

happens with probability at least 1-\{4+2I\{\delta_n<5b/6\}\lfloor\{\log(6/5)\}^{-1}\log(b/\delta_n)\rfloor\}\exp(-c_1n\delta_n^2).

Uses Lemma 9 · Used by Proof, Proof
Proof of Lemma 11

Consider the events \mathcal{E}_0=\{Z_n(\delta_n)\ge5L\delta_n^2/b\} and

\mathcal{E}_1=\Big\{\exists f\in\mathcal{F}:\Vert I_{\mathcal{O}_0}\cdot f\Vert_2\ge\delta_n\text{ and } \Big\vert\frac{1}{n} \sum_{i=1}^n h_f(O_i) \Big\vert\ge \frac{6L\delta_n}{b}\Vert I_{\mathcal{O}_0}\cdot f\Vert_2\Big\}.

If (11) is violated, then either \mathcal{E}_0 or \mathcal{E}_1 must occur. Applying Lemma 9 with r=\delta_n and u=\frac{L\delta_n^2}{b} implies that \mathcal{E}_0 happens with probability at most 2\exp(-c_1n\delta_n^2) where c_1=\{8b^2(1+17e)\}^{-1}. Applying Lemma 10 with \theta = 6/5 we conclude that \mathcal{E}_1 happens with probability at most \{2+2I\{\delta_n<5b/6\}\lfloor\{\log(6/5)\}^{-1}\log(b/\delta_n)\rfloor\}\:\exp(-c_1n\delta_n^2). Thus, (11) is violated with probability at most \{4+2I\{\delta_n<5b/6\}\lfloor\{\log(6/5)\}^{-1}\log(b/\delta_n)\rfloor\}\:\exp(-c_1n\delta_n^2). This concludes the proof.

Lemma

Lemma 12

KEC4

Under the assumptions of Theorem 4 and with the constants defined as in that Theorem, the following event happens for any h\in \mathcal{H}_{B}:

\begin{aligned} \frac{1}{4c^2}\Big\{\big\Vert I_0 \mathcal{T} \big(h_0^\dag-&\tilde h\big)\big\Vert^2_2-\big\Vert I_0\mathcal{T} \big(h_0^\dag-h\big)\big\Vert^2_2\Big\}+\lambda_{\mathcal{H}}\Big(\big\Vert \tilde h\big\Vert_{\mathcal{H}}^2-\big\Vert h\big\Vert_{\mathcal{H}}^2\Big)\\ &\quad\quad\quad\le \frac{1}{2c^2}\big\Vert I_0\mathcal{T} \big(h_0^\dag-h\big)\big\Vert^2_2+c_0\delta_n\big\Vert I_0\mathcal{T}\big(\tilde h-h\big)\big\Vert_2+\lambda_{\mathcal{G}}B_1^2+c_0'\delta^2_n. \end{aligned}
Uses Theorem 4 · Used by Proof, Proof, Proof
Proof of Lemma 12

For any h\in\mathcal{H}_{B}, define g_h:=\frac{1}{2c^2}\mathcal{T}(h_0^\dag-h). Since \Vert h_0^\dag-h\Vert_{\mathcal{H}}\le\Vert h_0^\dag\Vert_{\mathcal{H}}+\Vert h\Vert_{\mathcal{H}}\le 2B, Assumption 5 of the main text implies that g_h\in\mathcal{G}_{B_1} where B_1:=B'/c^2. In addition, for any h\in\mathcal{H}_{B}, we have \sup_{w\in\mathcal{W}}|h(w)|\le k_1 B, so the function class \mathcal{H}_B is uniformly bounded by k_1 B. Similarly, the function class \mathcal{G}_{B_1} is uniformly bounded by k_2B_1.

By part 1 of Proposition 2 and Assumption 5, we have

\frac{1}{4c^2}\big\Vert I_0 \mathcal{T} \big(h_0^\dag-\tilde h\big)\big\Vert^2_2= \mathbb{E}_{P_0}\big\{I_0(X)s(O)g_{\tilde h}(Z)-I_0(X)r(O)\tilde h(W)g_{\tilde h}(Z)-c^2\cdot I_0(X)g_{\tilde h}(Z)^2\big\},

where the expectation in the right hand-side is with respect to O, and \tilde h is regarded as a fixed, non-random function.

Next, we apply Lemma 11 to bound the expectation of each of the three terms in the right hand side of the last display. Specifically, with c_1:=\left\{8b^2(1+17e)\right\}^{-1}, c_2:=6L'/b, b:=k_2B_1, \xi(\delta_n):=\{4+2I\{\delta_n<5b/6\}\lfloor\{\log(6/5)\}^{-1}\log(b/\delta_n)\rfloor\}\:\exp(-c_1n\delta_n^2), and

L'=\max\{\Vert s\Vert_\infty,\:\Vert r\Vert_\infty\cdot k_1B,\: 2c^2\cdot b\},

applying Lemma 11, we can conclude that

  1. I. With probability at least 1-\xi(\delta_n), the following event happens

    \Big\vert \mathbb{E}_n\big[I_o(X)s(O)g(Z) - \mathbb{E}_{P_0}\left\{I_0(X)s(O)g(Z)\right\} \big]\Big \vert \le c_2\delta_n\left\{\left\Vert I_0g\right\Vert_2+\delta_n\right\} \quad \forall {g \in \mathcal{G}_{B_1}}.

    In particular, \mathbb{E}_{P_0}\{I_0(X)s(O)g_{\tilde h}(Z)\}\le \mathbb{E}_n\{I_0(X)s(O)g_{\tilde h}(Z)\}+ c_2\delta_n\{\Vert I_0g_{\tilde h}\Vert_2+\delta_n\} holds with probability at least 1-\xi(\delta_n).

  2. II. With probability at least 1-\xi(\delta_n), for all g \in \mathcal{G}_{B_1}, the following event happens

    \Big\vert \mathbb{E}_n\big[I_0(X)r(O)\tilde h(W)g(Z) - \mathbb{E}_{P_0}\{I_0(X)r(O)\tilde h(W)g(Z)\} \big]\Big \vert \le c_2\delta_n\{\left\Vert I_0g\right\Vert_2+\delta_n\}.

    Notably, -\mathbb{E}_{P_0}\{I_0(X)r(O)\tilde h(W)g_{\tilde h}(Z)\}\le -\mathbb{E}_n\{I_0(X)r(O)\tilde h(W)g_{\tilde h}(Z)\}+ c_2\delta_n\{\Vert I_0g_{\tilde h}\Vert_2+\delta_n\} holds with probability at least 1-\xi(\delta_n).

  3. III. With probability at least 1-\xi(\delta_n), the following event happens

    \Big\vert \mathbb{E}_n\big[c^2 I_0(X)g(Z)^2 - \mathbb{E}_{P_0}\{c^2 I_0(X)g(Z)^2\} \big]\Big \vert \le c_2\delta_n\{\Vert I_0g\Vert_2+\delta_n\} \quad \forall {g \in \mathcal{G}_{B_1}}.

    In particular, -\mathbb{E}_{P_0}\{c^2 I_0(X)g_{\tilde h}(Z)^2\}\le -\mathbb{E}_n\{c^2 I_0(X)g_{\tilde h}(Z)^2\}+ c_2\delta_n\{\Vert I_0g_{\tilde h}\Vert_2+\delta_n\} holds with probability at least 1-\xi(\delta_n).

To apply Lemma 11 to prove (I), (II), and (III), we define convenient function classes \mathcal{F}, maps f \in \mathcal{L}^2(P_{0,O})\mapsto t_f, and sets \mathcal{O}_0. For these objects, we argue that the following conditions hold:

  1. a. \mathcal{F} is a separable metric space with metric denoted by d such that \mathcal{F}\subset\mathcal{L}^2(P_{0,O}) and: (a.1) \mathcal{F} is uniformly bounded by b, (a.2) for each o\in\mathcal{O}, the evaluation functional f\mapsto f(o) is continuous relative to the metric d, and (a.3) f=0 belongs to \mathcal{F}.

  2. b. For each f\in\mathcal{L}^2(P_{0,O}), t_f:\mathcal{O}\mapsto\mathbb{R} is a measurable function.

  3. c. For each o\in\mathcal{O}, the map f\mapsto t_{I_{\mathcal{O}_0}\cdot f}(o) is continuous with respect to d at all f \in \mathcal{F}.

  4. d. The map f\mapsto t_{I_{\mathcal{O}_0}\cdot f} is continuous from (\mathcal{F},d) to \mathcal{L}^1(P_{0,O}).

  5. e. t_f satisfies t_f=0 when f=0.

  6. f. \exists L such that for any f_1,f_2\in\mathcal{L}^2(P_{0,O}), |t_{f_1}(o)-t_{f_2}(o)|\le L\left\vert f_1(o)-f_2(o)\right\vert \forall\:o\in\mathcal{O}.

(i) Application of Lemma 11 to show (I). Define \mathcal{F}:=\mathcal{G}_{B_1}, t_f(o):=s(o)\cdot f(z), and \mathcal{O}_0:=\{o\in\mathcal{O}\: :\:\text{the subvector }x\text{ of }o\text{ is in }\mathcal{X}_0\}. We endow \mathcal{F} with the metric d associated with the norm \Vert\cdot\Vert_{\mathcal{G}} of the RKHS \mathcal{G}. To show that condition (a) holds first note that \mathcal{F}= \mathcal{G}_{B_1}\subset\mathcal{L}^2(P_{0,Z})\subset\mathcal{L}^2(P_{0,O}), and \mathcal{G}_{B_1} is separable because \mathcal{G} is a separable RKHS by Assumption 6 (see Proposition 11.7 of [paulsen2016introduction]). We already argued that (a.1) holds with b=k_2B_1, (a.2) holds because the evaluation functionals are continuous in RKHSs, and (a.3) trivially holds.

Condition (b) is trivially satisfied. Condition (c) holds because |t_{I_{\mathcal{O}_0}\cdot f_1}(o)-t_{I_{\mathcal{O}_0}\cdot f_2}(o)|=|I_{\mathcal{O}_0}(o)s(o)\left\{f_1(z)-f_2(z)\right\}|\le \Vert s\Vert_\infty |f_1(z)-f_2(z)|\le \Vert s\Vert_\infty k_2\Vert f_1-f_2\Vert_{\mathcal{G}}, where the last inequality holds because \Vert f\Vert_\infty\le k_2\Vert f\Vert_\mathcal{F} \forall f\in\mathcal{F}. Condition (d) holds because \mathbb{E}_{P_0}\{|t_{I_{\mathcal{O}_0}\cdot f_1}(O)-t_{I_{\mathcal{O}_0}\cdot f_2}(O)|\}\le\sup_{o\in\mathcal{O}}|t_{I_{\mathcal{O}_0}\cdot f_1}(o)-t_{I_{\mathcal{O}_0}\cdot f_2}(o)|\le \Vert s\Vert_\infty\cdot k_2\cdot \Vert f_1-f_2\Vert_{\mathcal{G}}. Condition (e) trivially holds. Arguing as in the proof of (c), condition (f) holds with L=\Vert s\Vert_{\infty}.

Before applying Lemma 11, note that I_0\cdot\mathcal{G}_{B_1} is star-shaped because \mathcal{G}_{B_1} is star-shaped and multiplication by I_0 preserves this. Hence, its critical radius equals that of its star-shaped hull, so the lemma applies with an upper bound on the critical radius of I_0\cdot\mathcal{G}_{B_1}.

Applying Lemma 11, we obtain that the display (?) happens with probability at least 1-\xi(\delta_n) with c_2=6\cdot \Vert s\Vert_{\infty}/b and b=k_2B_1. Consequently, the display (?) also happens with probability at least 1-\xi(\delta_n) with c_2 now equal to 6L'/b where L' is defined in (?).

(ii) Application of Lemma 11 to show (II). Define \mathcal{F}, the metric d, and \mathcal{O}_0 as in (i). Define t_f(o):=r(o)\cdot \tilde h(w)\cdot f(z). Condition (a) holds as shown in (i). Condition (b) holds trivially. Conditions (c) and (d) hold as shown in (i), using in addition that \sup_{w\in\mathcal{W}}|\tilde h(w)|\le k_1B. Condition (e) is immediate. Arguing as in the proof of condition (c) in (i), (f) holds with L=\Vert r\Vert_{\infty}\cdot k_1B. Applying Lemma 11, we obtain that the display (?) happens with probability at least 1-\xi(\delta_n) with c_2=6 \Vert r\Vert_{\infty} k_1B/b and b=k_2B_2. Then, (?) also happens with probability at least 1-\xi(\delta_n) with c_2 now equal to 6L'/b where L' is defined in (?).

(iii) Application of Lemma 11 to show (III). Define \mathcal{F}, d, and \mathcal{O}_0 as in (i). Define t_f(o):=c^2\cdot f(z)^2. Condition (a) holds as shown in (i), (b) holds trivially, (c) holds because |t_{I_{\mathcal{O}_0}\cdot f_1}(o)-t_{I_{\mathcal{O}_0}\cdot f_2}(o)|=|c^2I_{\mathcal{O}_0}(o)\left\{f_1(z)^2-f_2(z)^2\right\}| \le c^2 \left\vert f_1(z)+f_2(z)\right\vert\cdot\left\vert f_1(z)-f_2(z)\right\vert \le 2c^2 k_2 B_1\left\vert f_1(z)-f_2(z)\right\vert\le 2c^2 k_2^2 B_1\Vert f_1-f_2\Vert_{\mathcal{G}}, and (d) holds because \mathbb{E}_{P_0}\{|t_{I_{\mathcal{O}_0}\cdot f_1}(O)-t_{I_{\mathcal{O}_0}\cdot f_2}(O)|\}\le\sup_{o\in\mathcal{O}}|t_{I_{\mathcal{O}_0}\cdot f_1}(o)-t_{I_{\mathcal{O}_0}\cdot f_2}(o)|\le 2c^2\cdot k_2^2\cdot B_1\cdot \Vert f_1-f_2\Vert_{\mathcal{G}}. Condition (e) is trivial and condition (f) holds with L=2c^2 b arguing as in the proof of (c).

Applying Lemma 11, we obtain that the display (?) happens with probability at least 1-\xi(\delta_n) with c_2=6\cdot 2c^2\cdot b/b and b=k_2B_1. Consequently, the display (?) also happens with probability at least 1-\xi(\delta_n) with c_2 now equal to 6L'/b where L' is defined in (?).

Setting c_3=3c_2=18L'/b with L' defined in (?), with probability at least 1-3\xi(\delta_n), we have \frac{1}{4c^2}\Vert I_0\mathcal{T}(h_0^\dag-\tilde h)\Vert^2_2\le\mathbb{E}_n\{I_0(X)s(O)g_{\tilde h}(Z)-I_0(X)r(O)\tilde h(W)g_{\tilde h}(Z)-c^2 I_0(X)g_{\tilde h}(Z)^2\}+c_3\delta_n\Vert I_0 g_{\tilde h}\Vert_2 +c_3\delta_n^2.

Then, the following event happens with probability at least 1-3\xi(\delta_n), for any h\in\mathcal{H}_B:

\begin{aligned} \frac{1}{4c^2}\big\Vert I_0\mathcal{T}\big(&h_0^\dag-\tilde h\big)\big\Vert^2_2\le\mathbb{E}_n\big\{I_0(X)s(O)g_{\tilde h}(Z)-I_0(X)r(O)\tilde h(W)g_{\tilde h}(Z)-c^2 I_0(X)g_{\tilde h}(Z)^2\big\}\\ &\quad-\lambda_{\mathcal{G}}\Vert g_{\tilde h}\Vert^2_{\mathcal{G}}+\lambda_{\mathcal{G}}\Vert g_{\tilde h}\Vert^2_{\mathcal{G}}+c_3\big(\delta_n\Vert I_0\cdot g_{\tilde h}\Vert_2 +\delta_n^2\big)\\ &\le \sup_{g\in\mathcal{G}}\big[\mathbb{E}_n\big\{I_0(X)s(O)g(Z)-I_0(X)r(O)\tilde h(W)g(Z)-c^2 I_0(X)g(Z)^2\big\}-\lambda_{\mathcal{G}}\Vert g\Vert_{\mathcal{G}}^2\big]\\ &\quad+\lambda_{\mathcal{G}}B_1^2+c_3\big(\delta_n\Vert I_0\cdot g_{\tilde h}\Vert_2 +\delta_n^2\big)\\ &\le\sup_{g\in\mathcal{G}}\big[\mathbb{E}_n\big\{I_0(X)s(O)g(Z)-I_0(X)r(O)h(W)g(Z)-c^2 I_0(X)g(Z)^2\big\}-\lambda_{\mathcal{G}}\Vert g\Vert_{\mathcal{G}}^2\big]\\ &\quad+\lambda_{\mathcal{G}}B_1^2+\lambda_{\mathcal{H}}\Big(\big\Vert h\big\Vert_{\mathcal{H}}^2-\big\Vert \tilde h\big\Vert_{\mathcal{H}}^2\Big)+c_3\big(\delta_n\Vert I_0\cdot g_{\tilde h}\Vert_2 +\delta_n^2\big). \end{aligned}

Note that the second inequality is always true because g_{\tilde h}\in \mathcal{G}_{B_1}, and the third inequality is also true because \tilde h is the empirical risk minimizer defined in (?).

Set c_4:=\max\{c_3[1+c_3/(2c^2)], 5c_2^2/c^2\}. For any h\in\mathcal{H}_B, we define

\begin{aligned} T_n(h)&:=\sup_{g\in\mathcal{G}}\mathbb{E}_{P_0}\big\{I_0(X)s(O)g(Z)-I_0(X)r(O)h(W) g(Z)-\frac{c^2}{2}\cdot I_0(X) g(Z)^2\big\}+c_4\delta_n^2,\\ A_n(h)&:=\sup_{g\in\mathcal{G}_{B_1}}\left[\mathbb{E}_n\big\{I_0(X)s(O)g(Z)-I_0(X)r(O)h(W)g(Z)-c^2 I_0(X)g(Z)^2\big\}-\lambda_{\mathcal{G}}\Vert g\Vert_{\mathcal{G}}^2\right]\\ B_n(h)&:=\sup_{g\in\mathcal{G}\setminus\mathcal{G}_{B_1}}\left[\mathbb{E}_n\big\{I_0(X)s(O)g(Z)-I_0(X)r(O)h(W)g(Z)-c^2 I_0(X)g(Z)^2\big\}-\lambda_{\mathcal{G}}\Vert g\Vert_{\mathcal{G}}^2\right],\\ \text{and}\quad&\\ C_n(h)&:= \sup_{g\in\mathcal{G}}\left[\mathbb{E}_n\big\{I_0(X)s(O)g(Z)-I_0(X)r(O)h(W)g(Z)-c^2\cdot I_0(X)g(Z)^2\big\}-\lambda_{\mathcal{G}}\Vert g\Vert_{\mathcal{G}}^2\right]. \end{aligned}

Below we will show that, with probability at least 1-3\xi(\delta_n), each of the following event happens: A_n(h)\le T_n(h) and B_n(h)\le T_n(h). This will then imply that with probability at least 1-6\xi(\delta_n) for any h\in\mathcal{H}_B:

\begin{aligned} C_n(h)\le T_n(h). \end{aligned}

Assume for now that the last assertion holds. By part 1 of Proposition 2 and Assumption 5, we have that \sup_{g\in\mathcal{G}}\mathbb{E}_{P_0}\{I_0(X)s(O)g(Z)-I_0(X)r(O)h(W)g(Z)-\frac{c^2}{2} I_0(X)g(Z)^2\}=\frac{1}{2c^2}\Vert I_0\mathcal{T}(h_0^\dag-h)\Vert^2_2. Combining this equality with the inequalities in (?) and (?), we then conclude that with probability at least 1-9\xi(\delta_n), for any h\in\mathcal{H}_B:

\begin{aligned} \frac{1}{4c^2}\big\Vert I_0\mathcal{T}\big(h_0^\dag-\tilde h \big)\big\Vert^2_2\le& \frac{1}{2c^2}\big\Vert I_0\mathcal{T}\big(h_0^\dag-h\big)\big\Vert^2_2+\lambda_{\mathcal{H}}\Big(\big\Vert h\big\Vert_{\mathcal{H}}^2-\big\Vert \tilde h\big\Vert_{\mathcal{H}}^2\Big)+\lambda_{\mathcal{G}}B_1^2 \\ &\quad+c_3\cdot\delta_n\Vert I_0\cdot g_{\tilde h}\Vert_2 +\left(c_3+c_4\right)\delta_n^2. \end{aligned}

Furthermore, by the triangle inequality \Vert I_0\mathcal{T}(h_0^\dag-\tilde h)\Vert_2\le \Vert I_0\mathcal{T}(h_0^\dag-h)\Vert_2+\Vert I_0\mathcal{T}(\tilde h-h)\Vert_2 and since \Vert I_0\cdot g_{\tilde h}\Vert_2=\frac{1}{2c^2}\Vert I_0\mathcal{T}(h_0^\dag-\tilde h)\Vert_2, the inequality (?) implies that the following event happens with probability at least 1-9\xi(\delta_n) for any h\in\mathcal{H}_B

\begin{aligned} \frac{1}{4c^2}\big\Vert I_0\mathcal{T}\big(h_0^\dag-\tilde h\big)\big\Vert^2_2\le&\frac{1}{2c^2}\big\Vert I_0\mathcal{T}\big(h_0^\dag-h\big)\big\Vert^2_2+\lambda_{\mathcal{H}}\Big(\big\Vert h\big\Vert_{\mathcal{H}}^2-\big\Vert \tilde h\big\Vert_{\mathcal{H}}^2\Big)+\lambda_{\mathcal{G}}B_1^2\\ &+\frac{c_3}{2c^2}\Big\{\delta_n\big\Vert I_0\mathcal{T}\big(h_0^\dag-h\big)\big\Vert_2+\delta_n\big\Vert I_0\mathcal{T}\big(\tilde h-h\big)\big\Vert_2\Big\}+(c_3+c_4)\delta_n^2. \end{aligned}

By the AM-GM inequality \frac{c_3}{2c^2}\delta_n \Vert I_0\mathcal{T}(h_0^\dag-h)\Vert_2\le\frac{1}{4c^2}\Vert I_0\mathcal{T}(h_0^\dag-h)\Vert_2^2+\frac{c_3^2}{4c^2}\delta_n^2. Hence, the following event happens with probability at least 1-9\xi(\delta_n) for all h\in\mathcal{H}_B:

\begin{aligned} \frac{1}{4c^2}&\Big\{\big\Vert I_0\mathcal{T}\big(h_0^\dag-\tilde h\big)\big\Vert^2_2-\big\Vert I_0\mathcal{T}\big(h_0^\dag-h\big)\big\Vert^2_2\Big\}+\lambda_{\mathcal{H}}\Big(\big\Vert \tilde h\big\Vert_{\mathcal{H}}^2-\big\Vert h\big\Vert_{\mathcal{H}}^2\Big)\\ &\quad\quad\quad\le \frac{1}{2c^2}\big\Vert I_0\mathcal{T}\big(h_0^\dag-h\big)\big\Vert^2_2+\frac{c_3}{2c^2}\delta_n\big\Vert I_0\mathcal{T}\big(\tilde h-h\big)\big\Vert_2+\lambda_{\mathcal{G}}B_1^2+\Big(c_3+c_4+\frac{c_3^2}{4c^2}\Big)\delta^2_n, \end{aligned}

thus, arriving at the desired inequality.

We now turn to prove that A_n(h)\le T_n(h) holds with probability at least 1-3\xi(\delta_n). Fix h\in\mathcal{H}_B and take g\in\mathcal{G}_{B_1}. Applying Lemma 11 as we did to show (I), (II), and (III), and noting that -\lambda_{\mathcal{G}}\Vert g\Vert_{\mathcal{G}}^2\le0, we have that the following event happens with probability at least 1-3\xi(\delta_n),

\begin{aligned} &\mathbb{E}_n\big\{I_0(X)s(O)g(Z)-I_0(X)r(O)h(W)g(Z)-c^2I_0(X)g(Z)^2\big\}-\lambda_{\mathcal{G}}\Vert g\Vert_{\mathcal{G}}^2\\ &\le E_{P_0}\big\{I_0(X)s(O)g(Z)-I_0(X)r(O)h(W)g(Z)-c^2I_0(X) g(Z)^2\big\}+c_3\delta_n\Vert I_0\cdot g\Vert_2+c_3\delta_n^2. \end{aligned}

By the AM-GM inequality, c_3\delta_n\Vert I_0\cdot g\Vert_2\le \frac{c^2}{2}\Vert I_0\cdot g\Vert_2^2+\frac{c_3^2}{2c^2}\delta_n^2. Hence, the right-hand side of (?) is upper bounded by

\begin{aligned} \sup_{g\in\mathcal{G}}\mathbb{E}_{P_0}\Big\{I_0(X)s(O)g(Z)-I_0(X)r(O)h(W)g(Z)-\frac{c^2}{2} I_0(X)g(Z)^2\Big\}+\left(c_3+\frac{c^2_3}{2c^2}\right)\delta_n^2\le T_n(h). \end{aligned}

Since g\in\mathcal{G}_{B_1} was arbitrary, A_n(h)\le T_n(h) happens with probability at least 1-3\xi(\delta_n).

Turn now to prove that B_n(h)\le T_n(h) holds with probability at least 1-3\xi(\delta_n). Fix h\in\mathcal{H}_B and take g\in\mathcal{G}\setminus\mathcal{G}_{B_1}. Then, \Vert g\Vert_{\mathcal{G}}>B_1. Define \tilde g:=B_1\cdot g/\Vert g\Vert_{\mathcal{G}}. Since \mathcal{G} is star-shaped and B_1/\Vert g\Vert_{\mathcal{G}}\in(0,1), \tilde g\in\mathcal{G}. Furthermore, \Vert\tilde g\Vert_{\mathcal{G}}=B_1, and so \tilde g\in\mathcal{G}_{B_1}. Applying Lemma 11 as we did to prove (I) and (II), we obtain that, with probability at least 1-2\xi(\delta_n), the following happens:

\begin{aligned} \mathbb{E}_n\big\{I_0(X)&s(O)g(Z)-I_0(X)r(O)h(W)g(Z)\big\}\\ &=\frac{\Vert g\Vert_\mathcal{G}}{B_1}\cdot \mathbb{E}_n\big\{I_0(X)s(O)\tilde g(Z)-I_0(X)r(O)h(W)\tilde g(Z)\big\}\\ &\le \frac{\Vert g\Vert_\mathcal{G}}{B_1}\cdot \big[\mathbb{E}_{P_0}\big\{I_0(X)s(O)\tilde g(Z)-I_0(X)r(O)h(W)\tilde g(Z)\big\}+2c_2\delta_n\Vert I_0\cdot \tilde g\Vert_2+2c_2\delta_n^2\big]\\ &=\mathbb{E}_{P_0}\big\{I_0(X)s(O)g(Z)-I_0(X)r(O)h(W)g(Z)\big\}+2c_2\delta_n\Vert I_0\cdot g\Vert_2+\frac{2c_2}{B_1}\Vert g\Vert_\mathcal{G}\cdot\delta_n^2, \end{aligned}

where c_2=6L'/b, with L' defined in (?). Applying Lemma 11 as we did to prove (III), with probability at least 1-\xi(\delta_n), the following event happens:

\begin{aligned} - \mathbb{E}_n\big\{c^2 I_0(X) g(Z)^2\big\}&=-\frac{\Vert g\Vert_\mathcal{G}^2}{B_1^2} \mathbb{E}_n\big\{c^2 I_0(X)\tilde g(Z)^2\big\}\\ &\le \frac{\Vert g\Vert_\mathcal{G}^2}{B_1^2} \left[-\mathbb{E}_{P_0}\big\{c^2I_0(X)\tilde g(Z)^2\big\}+12c^2\delta_n\left(\Vert I_0\cdot \tilde g\Vert_2+\delta_n\right)\right]\\ &=-\mathbb{E}_{P_0}\big\{c^2 I_0(X) g(Z)^2\big\}+\frac{12c^2}{B_1}\delta_n\Vert g\Vert_\mathcal{G}\cdot \Vert I_0\cdot g\Vert_2+\frac{12c^2}{ B_1^2}\Vert g\Vert_\mathcal{G}^2\cdot\delta_n^2. \end{aligned}

Hence, with probability at least 1-3\xi(\delta_n), the following event happens:

\begin{aligned} \mathbb{E}_n&\big\{I_0(X)s(O)g(Z)-I_0(X)r(O)h(W)g(Z)-c^2 I_0(X)g(Z)^2\big\}-\lambda_{\mathcal{G}}\Vert g\Vert_{\mathcal{G}}^2\\ &\le \mathbb{E}_P\big\{I_0(X)s(O)g(Z)-I_0(X)r(O)h(W)g(Z)-c^2 I_0(X) g(Z)^2\big\}-\lambda_{\mathcal{G}}\Vert g\Vert_{\mathcal{G}}^2\\ &\quad+2c_2\delta_n\Vert I_0\cdot g\Vert_2+\frac{2c_2}{B_1}\Vert g\Vert_\mathcal{G}\cdot\delta_n^2+\frac{12c^2}{B_1}\delta_n\Vert g\Vert_\mathcal{G}\cdot \Vert I_0\cdot g\Vert_2+\frac{12c^2}{ B_1^2}\Vert g\Vert_\mathcal{G}^2\cdot\delta_n^2. \end{aligned}

Applying the AM-GM inequality, we obtain (1) 2c_2\delta_n\Vert I_0\cdot g\Vert_2\le \frac{c^2}{4}\Vert I_0\cdot g\Vert_2^2+\frac{4c_2^2}{c^2}\delta_n^2, (2) \frac{2c_2}{B_1}\Vert g\Vert_\mathcal{G}\cdot\delta_n^2\le \frac{c_2^2}{c^2}\delta_n^2+\frac{c^2}{B_1^2}\Vert g\Vert_\mathcal{G}^2\cdot\delta_n^2, and (3) \frac{12c^2}{B_1}\delta_n\Vert g\Vert_\mathcal{G}\cdot \Vert I_0\cdot g\Vert_2\le \frac{c^2}{4}\Vert I_0\cdot g\Vert_2^2+\frac{144c^2}{B_1^2}\Vert g\Vert_\mathcal{G}^2\cdot\delta_n^2. Using these inequalities and the fact that \lambda_\mathcal{G}\ge 157c ^2\cdot\delta_n^2/B_1^2, the right-hand side of (?) is upper bounded by E_{P_0}\{I_0(X)s(O)g(Z)-I_0(X)r(O)h(W)g(Z)-\frac{c^2}{2} I_0(X) g(Z)^2\}+\frac{5c_2^2}{c^2}\delta_n^2\le\sup_{g\in\mathcal{G}} E_{P_0}\big\{m(O; g)-I_0(X)r(O)h(W)g(Z)-\frac{c^2}{2} I_0(X) g(Z)^2\big\}+\frac{5c_2^2}{c^2}\delta_n^2\le T_n(h). Thus, B_n(h)\le T_n(h) holds with probability at least 1-3\xi(\delta_n). This completes the proof.

Lemma

Lemma 13

NBCT

Under the assumptions of Theorem 5 and with \lambda_\mathcal{G}\ge 157c^2/B_1^2, with probability at least 1-\{36+18\cdot I\{\delta_n<5b/6\}\lfloor\{\log(6/5)\}^{-1} \log(b/\delta_n)\rfloor\}\exp(-c_1n\delta_n^2), the following event happens for any h\in \mathcal{H}_{B}:

\begin{aligned} \frac{1}{4c^2}\Big\{\big\Vert I_0 \mathcal{T} \big(h_0^\dag-&\tilde h\big)\big\Vert^2_2-\big\Vert I_0\mathcal{T} \big(h_0^\dag-h\big)\big\Vert^2_2\Big\}+\lambda_{\mathcal{H}}\Big(\big\Vert \tilde h\big\Vert_{\mathcal{H}}^2-\big\Vert h\big\Vert_{\mathcal{H}}^2\Big)\\ &\quad\quad\quad\le \frac{1}{2c^2}\big\Vert I_0\mathcal{T} \big(h_0^\dag-h\big)\big\Vert^2_2+c_0\delta_n\big\Vert I_0\mathcal{T}\big(\tilde h-h\big)\big\Vert_2+\lambda_{\mathcal{G}}B_1^2+c_0'\delta^2_n, \end{aligned}

where \tilde h=\arg\min_{h\in\mathcal{H}_{B}}\max_{g\in\mathcal{G}}\{\mathbb{E}_n[\{m(O;g) -I_0(X)r(O)h(W)g(Z)-c^2I_0(X)g(Z)^2\}]-\lambda_{\mathcal{G}}\Vert g\Vert_{\mathcal{G}}^2+\lambda_{\mathcal{H}}\Vert h\Vert_{\mathcal{H}}^2\}, c_0:=3\cdot\max\{3,B_2+2\}\cdot \max\{1\:\Vert r\Vert_\infty\cdot k_1B,\: 2c^2b\}/(c^2 b), b=B_1\max\{k_2,B_3\}, with B_2 and B_3 the constants in Assumption ?, and c_1 and c'_0 are defined as in Lemma 12.

Uses Theorem 5 · Used by Proof
Proof of Lemma 13

The proof proceeds similarly to that of Lemma 12, with minor changes to handle the terms \mathbb{E}_{P_0}\{m(O;g)\} and \mathbb{E}_n\{m(O;g)\}. Here, \xi(\delta_n) is as defined in 12, and the constants c_1 and c_2 are as in Lemma 12, except that L' and b are now L' := \max\left\{1,\; \|r\|_\infty \cdot k_1 B,\; 2c^2 \cdot b \right\} and b:=B_1\max\{k_2,B_3\}, with B_3 being the constant specified in Assumption ?.

To use Lemma 11 to bound \mathbb{E}_{P_0}\{m(O;g_{\tilde h})\}, define \mathcal{F}:=\{f(\cdot) :=m(\cdot;g):\mathcal{O}\mapsto\mathbb{R}\:|\: g\in\mathcal{G}_{B_1}\}, t_f:=f, and \mathcal{O}_0:=\mathcal{O}\text{ (equivalentlyI_O_0=1)}. We verify conditions (a)–(f) listed in the proof of Lemma 12, which are required for Lemma 11. Condition (a) follows immediately from part 2 of Assumption ?. Condition (a.3) holds because g=0 belongs to \mathcal{G}_{B_1} and therefore f_0:=0=m(\cdot,0) belongs to \mathcal{F} by the linearity of m. Next, for any f \in \mathcal{F}, f=m(\:\cdot\:,g_f) for some g\in\mathcal{G}_{B_1}. Then,

\begin{aligned} |f(o)| = |f(o)-f_0(o)|\le B_3\cdot d(f,f_0)= B_3\cdot d(m(o;g_f),m(o,0)) \le B_3\Vert g_f-0\Vert_\mathcal{G}\le B_3B_1, \end{aligned}

where the first and second inequalities follow by part 2 of Assumption ?, and the third holds because g_f \in \mathcal{G}_{B_1}. Thus, \mathcal{F} is uniformly bounded by B_3B_1\le b, and so it satisfies condition (a.1). Condition (a.2) follows from part 2 of Assumption ?, which implies for all o\in\mathcal{O} and f_1,f_2\in\mathcal{F} that |f_1(o)-f_2(o)|\le\sup_{o\in\mathcal{O}}|f_1(o)-f_2(o)|\le B_3\cdot d(f_1,f_2). Condition (b) holds by the assumption on m specified in Section ?. Condition (c) holds because t_{I_{\mathcal{O}_0}\cdot f}=f and therefore (c) is the same as (a.2) that we have already argued to hold. Condition (d) holds because the last display implies \mathbb{E}_{P_0}\{|f_1(O)-f_2(O)|\}\le\sup_{o\in\mathcal{O}}|f_1(o)-f_2(o)|\le B_3\cdot d(f_1,f_2). Condition (e) trivially holds because t_f=f. Likewise, (f) trivially holds with L=1\le L'.

By Lemma 11, \mathbb{E}_{P_0}\{m(O;g_{\tilde h})\}\le \mathbb{E}_n\{m(O;g_{\tilde h})\}+ c_2\delta_n\{\Vert m(O;g_{\tilde h})\Vert_2+\delta_n\} happens with probability at least 1-\xi(\delta_n). Also, \Vert m(O;g_{\tilde h})\Vert_2\le B_2\Vert I_0\cdot\tilde g\Vert_2 by part 1 of Assumption ?. Then, \mathbb{E}_{P_0}\left\{m(O;g_{\tilde h})\right\}\le \mathbb{E}_n\left\{m(O;g_{\tilde h})\right\}+ B_2c_2\cdot\delta_n\left\Vert I_0\cdot g_{\tilde h}\right\Vert_2+c_2\delta^2_n holds with probability at least 1-\xi(\delta_n). Combining the latter with the events in (?) and (?) from the proof of Lemma 12, each of which holds with probability at least 1-\xi(\delta_n), we conclude that, with probability at least 1-3\xi(\delta_n),

\begin{aligned} \frac{1}{4c^2}\big\Vert I_0\mathcal{T}\left(h_0^\dag-\tilde h\right)\big\Vert^2_2&\le\mathbb{E}_n\big\{I_0(X)s(O)g_{\tilde h}(Z)-I_0(X)r(O)\tilde h(W)g_{\tilde h}(Z)-c^2\cdot I_0(X)g_{\tilde h}(Z)^2\big\}\\ &\quad +c_2\delta_n\Vert I_0\cdot g_{\tilde h}\Vert_2 +c_2\delta_n^2, \end{aligned}

where c_3:=\max\{B_2+2,3\}\cdot c_2. The remainder of the proof then follows exactly as in Lemma 12, with this updated constant c_3 replacing the value c_3=3c_2=18L'/b used there.

Source and licence

The text on this page is from arXiv:2508.17181v1 by Antonio Olivas-Martinez and Andrea Rotnitzky, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.

The record of each item is in analysis/objects/2508/2508.17181.jsonl.