The general regularisation scheme applied to conditional density estimation

Recall that b=\mathbb{E}\left[\mathcal{T}_2 k(V)\right] and \hat{b}=1/n \sum_{i=1}^n \mathcal{T}_2 k(v_i). By Assumption (A4), we have \left\|\mathcal{T}_2 k(v)\right\|_\mathcal{H}\leq C for all v\in \mathcal{V} and \mathbb{E}\left[\left\|\mathcal{T}_2 k(V)\right\|_\mathcal{H}^2\right]\leq C^2. All the assumptions of Proposition 2 in [Caponnetto07] are thus verified with L=2C and \sigma=C and the desired result follows.

The space of Hilbert-Schmidt operators between \mathcal{H} and \mathcal{H} can be identified with \mathcal{H}\otimes \mathcal{H}, which is again a separable Hilbert space with norm \left\|\cdot\right\|_{\text{HS}}. Define the operator \zeta_z:\mathcal{H}\to\mathcal{H} by \zeta_z f=\left\langle f,k(\cdot,z)\right\rangle_{\mathcal{H}}k(\cdot,z)=f(z)k(\cdot,z). Observe that \mathcal{L}=\mathbb{E}[\zeta_Z]. To apply Proposition 2 from [Caponnetto07], we have to show that there exists L>0 such that \left\|\zeta_z\right\|_{\text{HS}}\leq L/2 for all z\in\mathcal{Z}. We first consider Setting (S1). Let (e_i)_{i\in\mathbb{N}} be an orthonormal basis of \mathcal{H}.

The Hilbert-Schmidt norm of \zeta_z is

Using that k\leq \kappa^2 and Parseval’s identity, we obtain

Note that this implies \mathbb{E}[\left\|\zeta_Z\right\|^2_{\text{HS}}]<\kappa^4.

All the assumptions of Proposition 2 in [Caponnetto07] are thus verified with L=2\kappa^2 and \sigma=\kappa^2 and the desired result follows since \left\|\cdot\right\|_{\mathcal{H}\to\mathcal{H}}\leq \left\|\cdot\right\|_{\mathrm{HS}}.

Now we consider the proof under Setting (S2). Let (X_i)_{i=1}^n be an i.i.d. sample of X and (U_j)_{j=1}^{n_u} be an i.i.d. sample of U. Let

and

Observe that \left\|{1}/{n}\sum_{i}\zeta_{(x_i,u)}\right\|\leq {1}/{n}\sum_{i}\left\|\zeta_{(x_i,u)}\right\|\leq \kappa^2 for all u\in \mathcal{U} and x_1,\ldots,x_n\in \mathcal{X} by Jensen’s inequality and the first part of the proof and similarly \left\|\mathbb{E}\left[\zeta_{(X,U)}|X=x\right]\right\|\leq \kappa^2 for all x\in \mathcal{X}. By Proposition 2 in [Caponnetto07], we have \mathbb{P}[B_1\geq \frac{6\kappa^2}{\sqrt{n_u}}\log\frac{4}{\eta}|X_1,\ldots,X_n]\leq \eta/2 and \mathbb{P}[B_2\geq \frac{6\kappa^2}{\sqrt{n}}\log\frac{4}{\eta}]\leq \eta/2. Finally, observe that

Now we compute

In this proof, we write \left\|\cdot\right\| instead of \left\|\cdot\right\|_{\mathcal{H}}. Let m=n_z under Setting (S1) and m=n_u under Setting (S2). We have

where r_\lambda(\sigma)=g_\lambda(\sigma)\sigma-1. Assume first that r\leq 1. On the one hand, as \left\|g_\lambda(\hat{\mathcal{L}})\right\|\leq B/\lambda, we have using Lemmas 2.2 and 2.3

where A_1 and A_2 are constants that do not depend on n, m, \eta and \lambda. On the other hand, since \left\|r_\lambda(\hat{\mathcal{L}})\right\|\leq \gamma and \left\|r_\lambda(\hat{\mathcal{L}})\hat{\mathcal{L}}^r\right\|\leq \gamma_r\lambda^r by definition of a regulariser, we have

By Theorem 1 in [Bauer], there exists c_r>0 such that

where we have chosen \lambda>6\kappa^2\left(1/{\sqrt{m}}+1/{\sqrt{n}}\right)\log({4}/{\eta}) to obtain the last inequality. Gathering the above inequalities, we can conclude that

where A_3 does not depend on n, m, \eta and \lambda. Choosing \lambda=(n^{-1/2}+m^{-1/2})^{1/(r+1)}, we obtain the desired bound.

To ensure that \lambda>6\kappa^2\left(1/{\sqrt{m}}+1/{\sqrt{n}}\right)\log({4}/{\eta}), we require that (n^{-1/2}+m^{-1/2})^{-r/(r+1)}>6\kappa^2\log(4/\eta), which can be written as \lambda^{-r}>6\kappa^2\log(4/\eta).

When r>1, Theorem 1 in [Bauer] cannot be applied but we have that \left\|\mathcal{L}^r-\hat{\mathcal{L}}^r\right\|\leq c'_r\left\|\mathcal{L}-\hat{\mathcal{L}}\right\| by Lemma D.5 in [Zhou] for some c'_r>0. Then, we can replace Equation (?) by

and a similar reasoning applies. The proof of the convergence in L^2(Z) follows similar arguments and is based on the identity

A complete proof is provided in [Bauer]. In their notation, it suffices to replace T by \mathcal{L}, T_\mathbf{x} by \hat{\mathcal{L}}, S_\mathbf{x}^*y by \hat{b} and to set \phi(t)=t^r.