Kernel methods provide an elegant framework for developing nonlinear learning algorithms from simple linear methods. Though these methods have superior empirical performance in several real data applications, their usefulness is inhibited by the significant computational burden incurred in large sample situations. Various approximation schemes have been proposed in the literature to alleviate these computational issues, and the approximate kernel machines are shown to retain the empirical performance. However, the theoretical properties of these approximate kernel machines are less well understood. In this work, we theoretically study the trade-off between computational complexity and statistical accuracy in Nyström approximate kernel principal component analysis (KPCA), wherein we show that the Nyström approximate KPCA matches the statistical performance of (non-approximate) KPCA while remaining computationally beneficial. Additionally, we show that Nyström approximate KPCA outperforms the statistical behavior of another popular approximation scheme, the random feature approximation, when applied to KPCA.
(\mathcal{X, B}) is a completely separable space endowed with \sigma-algebra \mathcal{B}. \mathcal{H} is a separable RKHS of \mathbb{R}-valued functions on \mathcal{X} with bounded continuous positive definite kernel satisfying \sup_{x\in\mathcal{X}}k(x,x)=:\kappa<\infty.
The eigenvalues (\lambda_i)_{i\in I} of \Sigma are simple, positive, and w.l.o.g. satisfy a decreasing rearrangement, i.e., \lambda_1>\lambda_2>\ldots
The rank of \widehat{\Sigma} is n-1, eigenvalues (\widehat{\lambda}_i)_{i=1}^{n-1} of \widehat{\Sigma} are simple, positive and w.l.o.g. satisfy a decreasing rearrangement, i.e., \widehat{\lambda}_1>\widehat{\lambda}_2>\ldots
Let (\widehat{\lambda}_i,\widehat{\phi}_i)_i be the eigensystem of \widehat{\Sigma} in (?). Define \mathbf{K}=[k(X_i,X_j)]_{i,j\in[n]}. Then
where \boldsymbol{\gamma}_i=(\gamma_{i,1},\cdots,\gamma_{i,n})=\frac{1}{n(n-1)}\mathbf{H}_n\boldsymbol{\widehat{\alpha}}_i with \boldsymbol{\widehat{\alpha}}_i\notin\emph{null}(\mathbf{H}_n), and (\widehat{\lambda}_{i},\boldsymbol{\widehat{\alpha}}_i)_i are the eigenvalues and eigenvectors of \frac{1}{n(n-1)}\mathbf{KH}_n.
Define the m\times m matrix \mathbf{M}=\mathbf{K}_{mm}^{-1/2}\mathbf{K}_{mn}\mathbf{H}_n\mathbf{K}_{nm}\mathbf{K}_{mm}^{-1/2}. The solution to (?) is given by
where \mathbf{u}_1=(u_{1,1},\ldots,u_{1,m}) is the unit eigenvector of \frac{1}{n(n-1)}\mathbf{M} corresponding to its largest eigenvalue, denoted \Tilde{\lambda}_1 and \tilde{S}_m^*:\mathbb{R}^m\rightarrow\mathcal{H}, \boldsymbol{\alpha}\mapsto\sum_{j=1}^m\boldsymbol{\alpha}_jk(\cdot,X_{r_j}).
The rank of \Tilde{\mathbf{K}}\mathbf{H}_n is m. The eigenvalues (\Tilde{\lambda}_i)_{i=1}^{m-1} of \frac{1}{n(n-1)}\Tilde{\mathbf{K}}\mathbf{H}_n are simple, positive, and w.l.o.g. satisfy a decreasing rearrangement, i.e., \Tilde{\lambda}_1>\Tilde{\lambda}_2>\ldots.
Suppose Assumptions 1-4 hold. For any t>0 define \mathcal{N}_\Sigma(t)=\emph{tr}(\Sigma(\Sigma+tI)^{-1}) and \mathcal{N}_{C,\infty}(t)=\sup_{x\in\mathcal{X}}\left\langle k(\cdot,x),(C+tI)^{-1}k(\cdot,x)\right\rangle_\mathcal{H} for the uncentered covariance operator C=\int_\mathcal{X} k(\cdot,x)\otimes_{\mathcal{H}} k(\cdot,x)d\mathbb{P}(x). Then the following hold:
(i)
(ii) For any 0\le\delta\le\frac{1}{2} satisfying n\ge2\log\frac{2}{\delta} and \frac{140\kappa}{n}\log\frac{16\kappa n}{\delta}\le t\le\left\lVert\Sigma\right\rVert_{\mathcal{L}^\infty(\mathcal{H})},
(iii) For any 0\le\delta\le\frac{1}{2},
provided the following conditions are satisfied:
1. \left(\frac{140\kappa}{n}\log\frac{16\kappa n}{\delta}\vee\frac{9\kappa}{n}\log\frac{n}{\delta}\right)\le t\le \left\lVert\Sigma\right\rVert_{\mathcal{L}^\infty(\mathcal{H})}\wedge\left\lVertC\right\rVert_{\mathcal{L}^\infty(\mathcal{H})},
2. m\ge\left(67\vee5\mathcal{N}_{C,\infty}(t)\right)\log\frac{4\kappa}{t\delta}\vee\frac{140\kappa}{t}\log\frac{8}{t\delta},
3. n\ge2\log\frac{2}{\delta}.
Since \Sigma is trace-class and \lambda_\ell\rightarrow 0 as \ell\rightarrow \infty, it follows that R^\ell(\Sigma)\rightarrow 0 as \ell\rightarrow \infty. The rate of this convergence may be analyzed after making assumptions on the decay rate of the (\lambda_i)_i, which will be presented in the upcoming corollaries. The behavior of the empirical variations depends significantly on t and \mathcal{N}_{\Sigma}(t). \mathcal{N}_\Sigma(t) is referred to as the effective dimension or degrees of freedom [Caponnetto-07], which measures the capacity of the hypothesis space \mathcal{H}. Upon making assumptions regarding the decay rate of (\lambda_i)_i, the size of \mathcal{N}_\Sigma(t) can be quantified and convergence rates for R^\ell(\widehat{\Sigma}) and R_{nys}^\ell(\widehat{\Sigma}) can be obtained. The upper bounds for R^\ell(\widehat{\Sigma}) and R_{nys}^\ell(\widehat{\Sigma}) are equivalent up to constants; however, the conditions imposed on m and t in (iii) will dictate whether this behavior of R_{nys}^\ell(\widehat{\Sigma}) may be achieved with a reduced computational complexity (m<n). We also would like to highlight that the results presented in Theorem 4.1, which are obtained for the U-statistic estimator \widehat{\Sigma} of the centered covariance operator, \Sigma, matches up to constants, the results in Theorem 2 of [Sterge-20], which were derived for the uncentered covariance operator, C.
The following corollary derives convergence rates from the bounds in Theorem 4.1 under polynomial decay assumptions on the eigenvalues of \Sigma.
Suppose \underbar{A}i^{-\alpha}\le\lambda_i\le\bar{A}i^{-\alpha} for some \alpha>1 and 0<\underbar{A}<\bar{A}<\infty. Let \ell=n^{\frac{\theta}{\alpha}}, where 0<\theta\le \alpha. Then the following hold:
(i)
There exists an N\in\mathbb{N} such that for all n>N, the following hold:
(ii)
(iii)
(i) Of course, \alpha>1 is required to ensure that \Sigma is trace class. Observing (ii) and (iii), we see that the convergence rates of R^\ell(\widehat{\Sigma}) and R_{nys}^\ell(\widehat{\Sigma}) rely heavily on the growth of \ell through \theta. Comparing R^\ell(\widehat{\Sigma}) to R^\ell(\Sigma), EKPCA will match the convergence rate of KPCA provided \ell does not grow faster than n^{1/\alpha}. We note that 0<\theta<1 is the only sensible regime both computationally and statistically, as \theta\ge1 increases the computational complexity while the rate plateaus at \left(\log n/n\right)^{1-\frac{1}{\alpha}}. (ii) When \theta<1, the convergence rate of R_{nys}^\ell(\widehat{\Sigma}) is equal to that of R^\ell(\Sigma) and R^\ell(\widehat{\Sigma}), provided m\gtrsim n^\theta\log n, i.e., if \ell grows to infinity not faster than n^{1/\alpha} and the number of subsamples m grows sufficiently fast, then NY-EKPCA and EKPCA enjoy the same statistical behavior. From a computational perspective, the computational complexity of EKPCA using the Lanczos method is O(n^{2+\frac{\theta}{\alpha}}), while the complexity of NY-EKPCA is O(nm^2+m^2\ell)=O(nm^2). Thus, NY-EKPCA will offer a computational advantage with no loss in statistical performance, if \theta<\frac{1}{2}+\frac{\theta}{2\alpha}, i.e., \theta<\frac{\alpha}{2\alpha-1}. This means NY-EKPCA has better computational complexity and same statistical rates for \theta<\frac{\alpha}{2\alpha-1} while it loses the computational edge with no loss in the statistical behavior when \frac{\alpha}{2\alpha-1}\le\theta<1. Note that the first few principal components are often the greatest interest in practice; thus, the case \theta<1 may be more relevant in application.
Suppose Assumptions 1-4 hold. For any t>0 define \mathcal{N}_\Sigma(t)=\emph{tr}(\Sigma(\Sigma+tI)^{-1}). Then the following hold:
(i)
(ii) For any \delta>0 with n\ge2\log\frac{2}{\delta} and \frac{140\kappa}{n}\log\frac{16\kappa n}{\delta}\le t\le\left\lVert\Sigma\right\rVert_{\mathcal{L}^\infty(\mathcal{H})},
(iii) Suppose m random features are sampled i.\,i.\,d. from a probability measure \Lambda. For any \delta>0 with n\ge 2\log\frac{2}{\delta}, m\ge \left(2\vee \frac{1024\kappa^2}{\sum_{i>\ell}\lambda^2_i}\right)\log\frac{2}{\delta} and \frac{140\kappa}{n}\log\frac{16\kappa n}{\delta}\vee\frac{86\kappa}{m}\log\frac{16\kappa m}{\delta}\le t\le\frac{\left\lVert\Sigma\right\rVert_{{\mathcal{L}^\infty(\mathcal{H})}}}{3}, with probability at least 1-12\delta over the joint measure \mathbb{P}^n\times\Lambda^m:
where \mathcal{A}_1(t):=\mathcal{N}_\Sigma(t)+\frac{16\kappa\log\frac{2}{\delta}}{tm}+\sqrt{\frac{8\kappa\mathcal{N}_\Sigma(t)\log\frac{2}{\delta}}{tm}}.
Under the same assumptions as in Theorem 4.1,
provided the following conditions are satisfied:
1. \left(\frac{140\kappa}{n}\log\frac{16\kappa n}{\delta}\vee\frac{9\kappa}{n}\log\frac{n}{\delta}\right)\le t\le \left\lVert\Sigma\right\rVert_{\mathcal{L}^\infty(\mathcal{H})}\wedge\left\lVertC\right\rVert_{\mathcal{L}^\infty(\mathcal{H})},
2. m\ge\left(67\vee5\mathcal{N}_{C,\infty}(t)\right)\log\frac{4\kappa}{t\delta}\vee\frac{140\kappa}{t}\log\frac{8}{t\delta},
3. n\ge2\log\frac{2}{\delta}.
(i) Comparing Theorem 4.4 with Theorem 4.3(ii), we note that the bounds for EKPCA and NY-EKPCA are identical up to constants, similar to the case in Theorem 4.1. Compared to their counterparts in Theorem 4.1, T^\ell(\widehat{\Sigma}) and T_{nys}^\ell(\widehat{\Sigma}) have similar dependence on the effective dimension, but a squared dependence on \lambda_{\ell+1} and t—in contrast to a linear dependence in Theorem 4.1. This is a byproduct of working with the {L^2(\mathbb{P})}-norm, which is weaker than the RKHS norm, and therefore will result in faster convergence rates, as will be evident in Corollary 4.5. Additionally, T^\ell(\Sigma) will decay as \ell\rightarrow\infty more rapidly than R^\ell(\Sigma), as it depends on the sum of squared eigenvalues. The error in estimating the mean element is not improved in the move to {L^2(\mathbb{P})}-norm; it is bounded as n^{-1} in all of the empirical varieties regardless of norm.
(ii) An immediate difference between NY-EKPCA and RF-EKPCA is the dependence on m. This difference can be seen in both the upper bounds of T_{rf}^\ell(\widehat{\Sigma}_m) and T_{nys}^\ell(\widehat{\Sigma}), as well as the size requirements on m. This is primarily due to the approximation error incurred by RF-EKPCA, which approximates \mathcal{H} with an m-dimensional RKHS. Of course, this dependence on m is crucial in analyzing the computational vs. statistical trade-off between the two methods.
Under the same assumptions as in Corollary 4.2, the following hold:
(i)
There exists an N\in\mathbb{N} such that for all n>N, the following hold:
(ii)
(iii)
(iv)
where m=n^\gamma for 0<\gamma\le1.
Results (i), (ii), and (iv) are quoted from [sriperumbudur-20, Corollary 3].
(i) We first highlight the difference between the {L^2(\mathbb{P})} and \mathcal{H} norms through the comparison of Corollaries 4.2 and 4.5. T^\ell(\Sigma) decays at a rate of n^{-2\theta(1-\frac{1}{2\alpha})}, which is faster than that of its analog in \mathcal{H}-norm, i.e., R^\ell(\Sigma). While R^\ell(\widehat{\Sigma}) and R_{nys}^\ell(\widehat{\Sigma}) recover the optimal convergence rate (compared to KPCA) in the range \theta<1, T^\ell(\widehat{\Sigma}) and T_{nys}^\ell(\widehat{\Sigma}) are only able to recover the optimal rate for \theta<\frac{\alpha}{2\alpha-1}. The \frac{1}{n} term, which arises due to the empirical recentering, is never dominant in R^\ell(\widehat{\Sigma}) and R_{nys}^\ell(\widehat{\Sigma}); however, it can dominate in T^\ell(\widehat{\Sigma}) and T_{nys}^\ell(\widehat{\Sigma}) depending on the range of \theta.
(ii) Observing T^\ell(\widehat{\Sigma}) and T_{nys}^\ell(\widehat{\Sigma}) we see that, as in Corollary 4.2, NY-EKPCA and EKPCA have similar convergence behavior, provided m is large enough. Therefore, it follows from Remark 2, that in both \mathcal{H} and {L^2(\mathbb{P})}, NY-EKPCA will provide less computational cost with no loss in statistical performance compared to EKPCA. However in {L^2(\mathbb{P})}, unlike in \mathcal{H}, NY-EKPCA is computationally advantageous than EKPCA regardless of the size of \theta.
(iii) When \theta<\frac{\alpha}{2\alpha-1}, both RF-EKPCA and NY-EKPCA achieve the optimal convergence rate of n^{-2\theta(1-\frac{1}{2\alpha})}, but with NY-EKPCA being more computationally efficient than RF-EKPCA. This is because, in this regime, RF-EKPCA scales as O(n^{1+2\gamma}) and NY-EKPCA as O(n^{1+2\theta}) with \gamma\ge\theta(2-\frac{1}{\alpha})>\theta. In the range \theta\ge\frac{\alpha}{2\alpha-1}, NY-EKPCA achieves the optimal convergence rate of \frac{1}{n}, while RF-EKPCA converges as n^{-\gamma} with \gamma<1. Further, in this range of \theta, NY-EKPCA will offer less computational cost for \gamma\ge\frac{\alpha}{2\alpha-1}, as RF-EKPCA scales as O(n^{1+2\gamma}) and NY-EKPCA as O(n^{1+\frac{2\alpha}{2\alpha-1}}).
Let H be a separable Hilbert space and X a separable topological space. Define
where \nu:X\rightarrow H is a Bochner measurable function with \sup_{x\in X}\left\lVert\nu(x)\right\rVert=\kappa. For \{X_i\}_{i=1}^r\stackrel{i.\,i.\,d.}{\sim}P with r\ge2, define
Then the following hold for any 0\le\delta\le\frac{1}{2} and \frac{140\kappa}{r}\log\frac{16\kappa r}{\delta}\le t\le\left\lVert\mathfrak{C}\right\rVert_{\mathcal{L}^\infty(H)}:
(i) P^r\left\{\left\lVert(\mathfrak{C}+tI)^{-1/2}(\widehat{\mathfrak{C}}-\mathfrak{C})(\frak{C}+tI)^{-1/2}\right\rVert_{\mathcal{L}^\infty(H)}\le\frac{1}{2}\right\}\ge1-2\delta;
(ii) P^r\left\{\sqrt{\frac{2}{3}}\le\left\lVert(\frak{C}+tI)^{1/2}(\widehat{\frak{C}}+tI)^{-1/2}\right\rVert_{\mathcal{L}^\infty(H)}\le\sqrt{2}\right\}\ge1-2\delta;
(iii) P^r\left\{\left\lVert(\frak{C}+tI)^{-1/2}(\widehat{\frak{C}}+tI)^{1/2}\right\rVert_{\mathcal{L}^\infty(H)}\le\sqrt{\frac{3}{2}}\right\}\ge1-2\delta;
(iv) P^r\left\{\lambda_k(\widehat{\frak{C}})+t\le\frac{3}{2}(\lambda_k(\frak{C})+t)\right\}\ge1-2\delta for all k\ge1;
(v) P^r\left\{\lambda_k(\frak{C})+t\le2(\lambda_k(\widehat{\frak{C}})+t)\right\}\ge1-2\delta for all k\ge1.
Suppose Assumption 1 holds and \frac{9\kappa}{n}\log\frac{n}{\delta}\le t\le\left\lVertC\right\rVert_{\mathcal{L}^\infty(\mathcal{H})} for any 0<\delta<1. Then
where C=\int k(\cdot,x)\otimes_\mathcal{H} k(\cdot,x)\,d\mathbb{P}(x).
Suppose Assumption 1 holds and n\ge2\log\frac{2}{\delta} for any 0<\delta<1. Then
Suppose Assumption 1 holds, and for some m<n, the set of indices \{i_1,\cdots,i_m\} is drawn uniformly without replacement from [n]. For some t>0, define \mathcal{N}_{C,\infty}(t)=\sup_{x\in\mathcal{X}}\left\langle k(\cdot,x),(C+tI)^{-1}k(\cdot,x)\right\rangle_\mathcal{H}, where C=\int_\mathcal{X} k(\cdot,x)\otimes_{\mathcal{H}} k(\cdot,x)\,d\mathbb{P}(x) is the uncentered covariance operator. Then, for any \delta>0 and m\ge(67\vee5\mathcal{N}_{C,\infty}(t))\log\frac{4\kappa}{t\delta}, we have
where P_m is the orthogonal projector onto \emph{span}\{k(\cdot,X_{i_j})|j\in[m]\}.
Suppose Assumption 1 holds, and for some m<n, the set of indices \{i_j\}_{j=1}^{m} is drawn uniformly from the set of all partitions of size m of \{1,2,...,n\}. Then, for 0\le\delta\le\frac{1}{2}, 0<t\le\left\lVert\Sigma\right\rVert_{\mathcal{L}^\infty(H)} and m\ge \frac{140\kappa}{t}\log\frac{8}{t\delta}, we have
where \bar{P}_m is the orthogonal projector onto \bar{\mathcal{H}}_m as defined in Section ?.
Define \widehat{\Sigma}_m:=\frac{1}{2m(m-1)}\sum_{j\ne l}^m(k(\cdot,X_{i_j})-k(\cdot,X_{i_l}))\otimes_{\mathcal{H}}(k(\cdot,X_{i_j})-k(\cdot,X_{i_l})), which means \widehat{\Sigma}_m=\frac{1}{2m(m-1)}\tilde{S}_m^*\mathbf{H}_m\tilde{S}_m=\frac{1}{2(m-1)}\tilde{S}_m^*\mathbf{C}_m^2\tilde{S}_m=Z^*Z, where Z^*=\frac{1}{\sqrt{2(m-1)}}\tilde{S}_m^*\mathbf{C}_m. Note that Z^* has range \bar{\mathcal{H}}_m, and so \text{ran}(\bar{P}_m)=\text{ran}(Z^*). Therefore, by Proposition 3 of [Rudi-15], we have
The proof is completed by applying Lemma A.1.
For any orthogonal projector P:\mathcal{H}\rightarrow\mathcal{H}, the following holds:
By denoting \bar{k}(\cdot,X):=k(\cdot,X)-m_\mathbb{P}, we have
where we used (I-P)^2=(I-P). Since k is bounded, and thus Bochner integrable, it follows that
The proof is completed by using \left\lVert(I-P)\Sigma^{1/2}\right\rVert_{\mathcal{L}^2(\mathcal{H})}^2=\text{tr}\left(\Sigma^{1/2}(I-P)^2\Sigma^{1/2}\right) in (Proof).
The sampling operator, S_n:\mathcal{H}\rightarrow\mathbb{R}^n,\,f\mapsto\left(f(X_1),f(X_2),\ldots,f(X_n)\right)^\top, and approximate sampling operator, \Tilde{S}_m:\mathcal{H}\rightarrow\mathbb{R}^m,\,f\mapsto\left(f(X_{i_1}),f(X_{i_2}),\ldots,f(X_{i_m})\right)^\top, have the following properties:
(i) S_n^*:\mathbb{R}^n\rightarrow\mathcal{H},\,\alpha\mapsto\sum_{i=1}^n\alpha_ik(\cdot,X_i);
(ii) \frac{1}{n(n-1)}S_n^*\mathbf{H}_nS_n=\widehat{\Sigma};
(iii) S_nS_n^*=\mathbf{K} and \Tilde{S}_m\Tilde{S}_m^*=\mathbf{K}_{mm};
(iv) \frac{1}{n}S_n^*S_n=\frac{1}{n}\sum_{i=1}^n k(\cdot,X_i)\otimes_{\mathcal{H}} k(\cdot,X_i)=C_n;
(v) \mathbf{K}_{nm}=S_n\Tilde{S}_m^*.
We refer the reader to [sriperumbudur-20, Proposition B.1] for the proofs of (i), (ii), and (iii). The proofs of (iv) and (v) are provided in [Rudi-15, Section B].
The text on this page is from arXiv:2105.08875v1 by Nicholas Sterge and Bharath Sriperumbudur, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.
The record of each item is in analysis/objects/2105/2105.08875.jsonl.