Computational-Statistical Trade-off in Kernel Two-Sample Testing with Random Fourier Features

Let R \in \mathbb N be a fixed number and let \boldsymbol{\omega}_R=\{\omega_r\}^R_{r=1} be a sequence of real-valued i.i.d. random vectors from a probability distribution on \mathbb R^d which is absolutely continuous with respect to the Lebesgue measure. For arbitrary \epsilon \in (0,1), there exists an uncountable set \mathcal A_\epsilon of mutually distinct probability distributions on \mathbb R^d such that for any distinct pair P_X,P_Y \in \mathcal A_\epsilon and their corresponding random vectors X and Y, it holds that \mathbb{P}_{\boldsymbol{\omega}_R}(\frac{1}{R}\sum^R_{r=1}|\phi_X(\omega_r)-\phi_Y(\omega_r)|^2=0)\geq1-\epsilon.

Let k,R \in \mathbb N be fixed numbers and let \boldsymbol{\omega}_R=\{\omega_r\}^R_{r=1} be a sequence of real-valued i.i.d. random vectors from a probability distribution on \mathbb R^d which is absolutely continuous with respect to the Lebesgue measure. For arbitrary \epsilon \in (0,1), there exists an uncountable set \mathcal A_{k,\epsilon} of mutually distinct probability distributions on \mathbb R^d such that for any distinct pair P_X,P_Y \in \mathcal A_{k,\epsilon} and their corresponding random vectors X and Y, it holds that \mathbb{P}_{\boldsymbol{\omega}_R}(\boldsymbol{\omega}_R\in \mathcal{E}_k)\geq1-\epsilon.

Let k(x,y)=\kappa(x-y) be a bounded continuous positive definite kernel whose inverse Fourier transform is absolutely continuous with respect to the Lebesgue measure. Then, given any \epsilon>0, for the test \Delta_{{n_1},{n_2},R}^{\alpha} defined in Equation (?) with a fixed number R \geq 1 and the limiting sample-ratio p:=\lim_{{n_1},{n_2}\rightarrow\infty}\frac{{n_1}}{{n_1}+{n_2}} \in (0,1), there exist uncountably many pairs of distinct probability distributions P_X and P_Y on \mathbb R^d that satisfy

Consider the same setting in Theorem 3. Given any \epsilon >0, for the test \Delta_{{n_1},{n_2},R}^{\alpha,u} defined in Equation (?) with a fixed number R and the limiting sample-ratio p, there exist uncountably many pairs of distinct probability distributions P_X and P_Y on \mathbb R^d that satisfy

Consider an arbitrary sequence \{R_{n}\}_{{n} \geq 1} that increases as \lim_{{n}\rightarrow\infty}R_{n}=\infty and assume that the kernel k(\cdot,\cdot) in Theorem 3 is characteristic. Then, against any fixed alternative (P_X,P_Y)\in \mathcal P_1, the permutation test \Delta_{{n_1},{n_2},R}^{\alpha} defined in Equation (?) with R=R_{n} and n := \min \{n_1, n_2\} satisfies

This result also holds for the permutation test \Delta_{{n_1},{n_2},R}^{\alpha,u} defined in Equation (?).

Consider the test \Delta_{{n_1},{n_2},R}^{\alpha,\lambda} and \Delta_{{n_1},{n_2},R}^{\alpha,u,\lambda} with \lambda_i = {n}^{-2/(4s+d)}, i=1,\ldots,d. Then, there exist some positive constants C_R(M_3,\beta,d) and C_{L_2}(M_1,M_2, M_3, \alpha,\beta,d,s) such that the uniform separation of \Delta_{{n_1},{n_2},R}^{\alpha,\lambda} with the choice of R \geq C_R(M_3,\beta,d){n}^{2d/(4s+d)} satisfies

The same guarantee also holds for \Delta_{{n_1},{n_2},R}^{\alpha,u,\lambda}. Moreover, the computational cost of the corresponding test statistics \text{r}\widehat {\mathrm{MMD}}_b^2 and \text{r}\widehat {\mathrm{MMD}}_u^2 is O(Nn^{\frac{2d}{4s+d}}d).

Consider the tests \Delta_{{n_1},{n_2},R}^{\alpha} and \Delta_{{n_1},{n_2},R}^{\alpha,u} equipped with a characteristic kernel k satisfying the assumptions of Theorem 3, and suppose further that 0\leq k(x,y)\leq K for all x,y\in\mathbb R^d. Then, the test \Delta_{{n_1},{n_2},R}^{\alpha} with R={n} = \min\{{n_1,n_2}\} achieves the minimax separation rate, satisfying

for some positive constant C_{\mathrm{MMD}}(\alpha,\beta,K). The same guarantee also holds for \Delta_{{n_1},{n_2},R}^{\alpha,u}. Moreover, the computational cost of the corresponding test statistics \text{r}\widehat {\mathrm{MMD}}_b^2 and \text{r}\widehat {\mathrm{MMD}}_u^2 is O(N{n}d).

For the class of distribution pairs \mathcal C_{N,\Sigma} and the Gaussian kernel k_\lambda(x,y) with any fixed bandwidth \lambda=(\lambda_1,\ldots,\lambda_d)^\top \in (0,\infty)^d, there exist some positive constants C_1(\beta,d,\lambda, \Sigma) and C_2(\alpha,\beta,d, \lambda) such that \Delta_{{n_1,n_2},R}^{\alpha,\lambda} with the choice of R \geq C_1(\beta,d,\lambda, \Sigma) satisfies

and the computational cost of the corresponding estimator \text{r}\widehat {\mathrm{MMD}}_b^2 is O(Nd). This result also holds for the test \Delta_{{n_1,n_2},R}^{\alpha,u,\lambda} with the same choice of R and its corresponding estimator \text{r}\widehat {\mathrm{MMD}}_u^2.

A translation-invariant bounded continuous kernel k(x,y)=\kappa(x-y) on \mathbb R^d is positive definite if and only if there exists a finite non-negative Borel measure \Lambda on \mathbb R^d such that

Let p,q,r\in[1,\infty] satisfy 1/p + 1/q = 1+1/r. Then, for any functions f \in L_p(\mathbb R^d) and g \in L_q(\mathbb R^d),

Suppose X^n=(X_1,\ldots,X_n) has distribution P_n in \mathcal X_n, and \boldsymbol{G}_n is a finite group of transformations g of \mathcal X_n onto itself. Also, let G_n be a random variable that is uniform on \boldsymbol{G}_n. Assume X^n and G_n are mutually independent. For a d-dimensional test statistic B_n=B_n(X^n), let \hat R_n^B denote the randomization distributions of a d-dimensional random vector B_n, defined by

Suppose, under P_n,

for a constant b\in \mathbb R^d. Then under P_n,

where \delta_c denotes the distribution function corresponding to the point mass function at c\in \mathbb R^d.

Let B_n and T_n be sequences of d-dimensional random variables satisfying Equation (11) and

where T and T' are independent, each with common d-variate cumulative distribution function R^T(\cdot). Let \hat R^{T+B}_n(t) denote the randomization distribution of T_n + B_n, defined in Equation (11) with B replaced by T+B. Then, \hat R^{T+B}_n(t) converges to the cumulative distribution function of T+b in probability. In other words,

if R^{T+b} is continuous at t\in \mathbb R^d, where R^{T+b}(\cdot) denotes the corresponding d-variate cumulative distribution function of T+b.

Suppose the randomization distribution of a test statistic T_n converges to T in probability. In other words,

if R^T is continuous at t\in \mathbb R^d, where R^T(\cdot) denotes the corresponding cumulative distribution function of T. Let h be a measurable map from \mathbb R^d to \mathbb R^s. Let C be the set of points in \mathbb R^d for which h is continuous. If \mathbb P(T\in C)=1, then the randomization distribution of h(T_n) converges to h(T) in probability.

Suppose that X_1,\ldots,X_{n_1} are {n_1} i.i.d. random samples from the d-dimensional distribution P_X, where X_i=(X_{i,1},\ldots,X_{i,d})^\top for i=1,\ldots,n_1 with mean vector \mu and covariance matrix \Sigma_X , and independently, Y_1,\ldots,Y_{n_2} be {n_2} i.i.d. random samples from the d-dimensional distribution P_Y, where Y_i=(Y_{i,1},\ldots,Y_{i,d})^\top for i=1,\ldots,n_2 with the common mean vector \mu and covariance matrix \Sigma_Y. Let N={n_1}+{n_2} and write Z=(Z_1,\ldots,Z_N)=(X_1,\ldots,X_{n_1},Y_1,\ldots,Y_{n_2}). Consider a test statistic T_{{n_1},{n_2}}(Z_1,\ldots,Z_N)={n_1}^{-1/2}\big[\sum_{i=1}^{n_1} X_i-\frac {n_1} {n_2} \sum_{j=1}^{n_2} Y_j\big] and its permutation distribution

where \boldsymbol{G}_N denotes the N! permutations of \{1,2,\ldots,N\} and t\in\mathbb R^{d}. Assume 0<\operatorname{Var}(X_{i,k})<\infty and 0<\operatorname{Var}(Y_{j,k})<\infty for all i=1,\ldots,{n_1}, j=1,\ldots,{n_2}, and k=1,\ldots,d. Let {n_1},{n_2}\rightarrow\infty, p=\lim_{{n_1},{n_2}\rightarrow\infty} \frac{{n_1}}{{n_1}+{n_2}} and assume that \bar \Sigma = \frac p {1-p}\Sigma_X+\Sigma_Y is positive definite. Then,

where G denotes the d-variate normal distribution with mean \boldsymbol{0} and variance \bar \Sigma.

Suppose P_X=N(\mu_X,\Sigma) and P_Y=N(\mu_Y,\Sigma). The squared MMD between P_X and P_Y using a Gaussian kernel \kappa_\lambda(x-y)=\prod^{d}_{i=1}\frac{1}{\sqrt{\pi}\lambda_i}\exp{\bigl\{-\frac{(x_i-y_i)^2}{\lambda_i^2}\bigr\}} has the following explicit form:

where D(\lambda^2/4)=\mathrm{diag}(\lambda_1^2/4,\ldots,\lambda_d^2/4).

Let X',X'' and Y',Y'' be the independent copies of X and Y, respectively. Then, the following two equations hold:

For the class of distribution pairs, \mathcal C_{N,\Sigma}, in Equation (?) and the Gaussian kernel k_\lambda(x,y) with any fixed bandwidth \lambda=(\lambda_1,\ldots,\lambda_d)^\top \in (0,\infty)^d, the inequality in Equation (?) holds with c= 2. Specifically, there exists a constant C=C(d,\lambda,\Sigma)>0 such that