Anticorrelated Noise Injection for Improved Generalization

First, we observe that Eq. (?) implies – in the considered case of Bernoulli perturbations \xi_n with (\xi_i^n)^2 = \sigma^2 a.s. – that

Since we assumed that L has \beta-Lipschitz continuous third-order partial derivatives, Theorem 2.1.5 from [nesterov2018book] implies that \tilde L is \beta-smooth. For some tensor B which captures all summands from the mean-zero perturbation of Eq. (Proof) we have

Taking the expectation, most of the terms cancel out and we get

Note that, under the assumption of positive \eta, \left(\eta-\frac{\beta\eta^2}{2}\right)>0 implies 0<\eta\le \frac{2}{\beta}. If we further require \left(\eta-\frac{\beta\eta^2}{2}\right)\ge\frac{\eta}{2}, we get the condition 0<\eta\le \frac{1}{\beta}.

Under this condition, the following upper bound holds(Outlined in blue color below is the updated version of the proof of the theorem.):

Rearranging the terms we get

After a division by \eta/2>0 we arrive at

Summing all terms, we arrive at the following:

After dividing by N we get

Note that \tilde L(z_{N})\ge\tilde L^* = \min_z \tilde L(z) > -\infty. This is because L is lower bounded and also tr(\nabla^2 L) is lower bounded since we assumed L has Lipschitz gradients. Therefore, \mathbb{E}[\tilde L(z_{0})] - \mathbb{E}[\tilde L(z_{N})] = O(1) and we conclude that

If we set \eta = O(\epsilon/\sigma^2), then O(\eta\sigma^2) = O(\epsilon). Further, under the choice \eta = \Theta(\epsilon/\sigma^2), we also have O(\eta^{-1} N^{-1}) = O(\sigma^2 \epsilon^{-1}N^{-1}). By setting N = \Omega(\sigma^2\epsilon^{-2}), we ensure that O(\eta^{-1} N^{-1}) = O(\epsilon).

All in all, we get \mathbb{E}\left[\frac{1}{N}\sum_{n=0}^{N-1}\|\nabla \tilde L(z_{n})\|^2\right] \le O(\epsilon) + O(\sigma^3).

For k=1 we get w_1 = A_0 w_0 + \varepsilon_0. The induction step yields

This completes the proof of the variations of constants formula.

We have, by direct computation:

Using independence of the \{\xi_k\} family, we obtain for the anticorrelated case:

where we used the fact that the \xi_k are not correlated. The case \varepsilon_k = \xi_k is similar and therefore left to the reader.

Simple application of the formula for geometric series. Numerical verification in Figure ?.

The proof is based on an induction argument, starting from the equation in Proposition 3.4:

First, note that by assumption on \rho_k the first term vanishes as k\to\infty. We just have to deal with the second term. Specifically, we want to show that for whatever sequence \rho_k\in(0,1) we have

A fundamental observation, is that the term can be written in a recursive form. Indeed,

where the second equality follows from the fact that, as previously noted, our notation implies \prod_{j=k+1}^k \rho_k^2 = 1, for all k. Let us now proceed again by induction to show that v_k\in(0,1) for all k\ge 0. Note that trivially v_0=0. Let’s proceed with the inductive step:

This quantity is less then one if and only if

Note that this is satisfied since \nu_{k-1}+1\le2, and \rho_k^2 \le \rho_k since \rho_k\in(0,1). The result follows.

As above, we denote the perturbations by \varepsilon_k; i.e., \varepsilon_k = \xi_k for PGD and \varepsilon_k = \xi_{k+1} - \xi_k for Anti-PGD. Let us start by inspecting the equation

where \varepsilon_{k}^u\in\mathbb{R}^d is the projection of the noise \varepsilon_{k} to the first d coordinates. It is clear that the optimal strategy of making \|w\| small is to increase |v|, so to sample nearby points and pick up the gradient. The greater v is in norm, the better. We can increase the norm of v by heavy noise injection (second equation). However, too much noise also increases \varepsilon_{k}^u, which acts adversarially to the decrease of \|w\| (error accumulation increases the Euclidean norm in expectation).

1.5ex plus 0.5ex minus .2ex

< - 1.5ex plus 0.5ex minus .2ex-1emparagraph41.5ex plus 0.5ex minus .2ex-1emChoice of stepsize and operating region. We start by motivating the choice of stepsize \eta =\frac{\alpha}{2D}. Starting from the point (u_0,0) with \|u_0\|^2=D>0, we consider the operating landscape region \alpha D<\|u_k\|^2<D/\alpha, with \alpha\in(0,1). We want to show that while standard noise injection makes the process exit the region from the right (D/\alpha side, see Figure ?), anticorrelated noise injection makes the process exit the region from the left (\alpha D side). In this region, named \mathcal{D}_\alpha, the maximal allowed learning rate is \eta\le\frac{2}{\max_{\mathcal{D}_\alpha}\{v^2,\|u\|^2\}}. Since v stays small (we are going to check this later in great detail), we select the stepsize \eta\le\frac{1}{2\max_{\mathcal{D}_\alpha}\|u\|^2}=\frac{\alpha}{2D} — which guarantees stability in expectaction, i.e. without noise injection (even allowing for some slack).

1.5ex plus 0.5ex minus .2ex

< - 1.5ex plus 0.5ex minus .2ex-1emparagraph41.5ex plus 0.5ex minus .2ex-1em[]Lower bound for uncorrelated noise (\boldsymbol{\varepsilon_k =\xi_k}). For this case, we have to show that standard noise injection cannot possibly work for reaching \alpha D, therefore we have to put ourselves in the best case scenario for PGD: that is, we have to provide an uniform upper bound for v_k under the second equation (i.e. the equation for v) and show that this is not enough for a substantial decrease in \|u\|. In the next paragraph (anticorrelated noise), we instead have to put ourselves in the worst case scenario — i.e. a lower bound for |v|— and show that this is still enough for anticorrelated noise to yield a substantial decrease in \|u\|.

To start, let us then look at the second equation:

where \varepsilon^v_k is the (d+1)-th component of \varepsilon_k. Since we start from v_0=0, the equation is completely dominated by noise, and is strongly mean reverting (i.e. v is effectively bounded). Indeed, since \eta = \frac{\alpha}{2D} and \|u_k\|^2\in\left(\alpha D,D/\alpha\right) by assumption, we have

where we used the fact that |\varepsilon^v_k|=\sigma and that \alpha^2\in(0,1). By induction, the last inequality yields that, starting from v_0=0, we have

Hence, we found the “best case scenario” for the w equation: |v_k| = \frac{2\sigma}{\alpha^2}, for all k. This gives w the best decrease rate possible.(Note that noise injection in u is independent of v, therefore to minimize \|u\| we need the shrinking factor to be as large as possible. We note that using this bound is precise: with probability one we are in the best scenario (we are finding a lower bound).) However, we need to check this value v_{\max} is such that the equation for w is indeed stable (we promised this to the reader in the last paragraph). We recall that this equation is w_{k+1} = (1 - \eta v_k^2) \cdot w_k + \varepsilon_{k}^u. Let us require (1 - \eta v_k^2)\in(0,1), for this we need 1/v_{\max}^2>\eta = \frac{\alpha}{2D}. Therefore, we need

This is guaranteed by assumption. To proceed, we substitute v_{\max} into the first equation to get

Let us call \rho := \left(\frac{\alpha^3 D- 2\sigma^2}{\alpha^3 D}\right)\in(0,1) the (best case) shrinking factor. Since \varepsilon_{i}^u is zero-mean, computing the expected value of \|u_{k}\|^2 leads to the following limit by Corollary ?:

where we assumed \sigma^2 strictly positive. Note that this limit is a monotonically increasing function of \sigma^2\in(0,D \alpha^3/2). Hence, we get

Last, we need to show that this lower bound on \lim_{k\to\infty} \mathbb{E}[\|u_{k}\|^2] coincides with (or is bigger than) the right boundary of the operating region in Figure ?. To do this we set:

This concludes the proof of Eq. (?).

1.5ex plus 0.5ex minus .2ex

< - 1.5ex plus 0.5ex minus .2ex-1emparagraph41.5ex plus 0.5ex minus .2ex-1em[]Upper bound for anticorrelated noise (\boldsymbol{\varepsilon_k =\xi_k-\xi_{k-1}}). We consider anticorrelated noise injection (\xi_k)_i \in \{-\sigma,\sigma\}, and \varepsilon_k = (\varepsilon_k^u,\varepsilon_k^v) =\xi_k-\xi_{k-1}. Again, let us first look at the second equation:

Since by hypothesis \eta = \frac{\alpha}{2D} and \alpha D \le \|u_k\|^2\le D/\alpha, we have (1 - \eta \|u_k\|^2)\in\left(1-\frac{\alpha^2}{2}, \frac{1}{2}\right). Clearly, we have that, for any k\ge 1 v_{k}^2, is non-zero with a non-vanishing probability. For  (noiseless) stability, we also need an upper bound on |v_k|. An easy (yet absolutely not tight) upper bound is the following:

where we simply used the absolute value subadditivity and the fact that |\varepsilon^v_k|\le |\xi_k|+|\xi_{k-1}| =2\sigma. Note that the equation directly yields by induction |v_k|\le 4\sigma for all k\ge 0.

Let us now deal with the equation for w.

For this equation, we would want all the coefficients \rho_k :=1 - \eta v_k^2 to be between 0 and 1 — i.e. we need to check that v is indeed not too big. Since |v_k|\le 4\sigma for all k\ge 0, we have the requirement 1 - \frac{\alpha}{2D} 16\sigma^2>0, which implies \sigma^2\le \frac{D}{8\alpha} — that satisfies our hypothesis.

So, to sum it up, we are in operating regime of Proposition 3.7: anticorrelated noise, \rho_k<1 with non-vanishing probability and \rho_k always between 0 and 1. Hence, we get that

Hence, for \sigma^2 small enough, the value \alpha D is reached. This directly implies the missing Eq. (?). The proof is thereby complete.

Recall from Eq. (?) that \operatorname{Tr}(\nabla^2 L(u,v)) = d v^2 + \|u\|^2.

For the first two inequalities in the corollary, recall from Eq. (?) of the proof of Theorem 3.1 that |v_n|\le 4\sigma for all n, almost surely. The first two inequalities in the Corollary follow now by Eq. (?).

For the last two inequalities in the Corollary, we lower bound the trace by \|u\|^2 and make use of Eq. (?).