Accelerating Stochastic Recursive and Semi-stochastic Gradient Methods with Adaptive Barzilai-Borwein Step Sizes

Deferred to the Appendix B.

By employing Lemma 1 and Lemma Proof, we are adequate to provide the theoretical analysis of inner loops in Theorem Proof.

Deferred to the Appendix C.

Theorem Proof shows sublinear convergence rates of the inner loops, i.e., the inner \{\left\|\nabla P\left(w_k^s\right)\right\|^2\} converges sublinearly in expectation with increasing m. Indeed, we’re sufficient to fix s=1 to dispose of the outer epoch, Algorithm ? degenerates to MB-SARAH-IN-RHBB/RHBB+ (see [nguyen2] for reference). To obtain an \varepsilon-accurate solution in MB-SARAH-IN-RHBB, the number of iterations, m, is put up so that \mathbb{E}\left[\|\nabla P(w_m)\|^2\right] \leq \varepsilon, which suggests that

Assume that P\left(w_0\right)-P\left(w_*\right)=\sigma, (?) implies m_{RH} = \lceil \frac{2\overline{b}\mu \sigma\kappa}{\varepsilon \gamma\left(\hat{\alpha} \kappa+1-\tilde{\alpha}\right)}-1\rceil. Compare with m_{R} = \lceil \frac{2\overline{b}\mu \sigma}{\varepsilon \gamma}-1\rceil in [yang], we have \left(m_{RH}<m_{R}\right) due to L > \mu. In MB-SARAH-RHBB+, that’s m_{RH+} = \lceil \frac{2\overline{b}\mu_q \sigma\kappa^+}{\epsilon \gamma(\hat{\alpha} \kappa^+ +1-\tilde{\alpha})}-1\rceil to achieve the same \varepsilon-accuracy. Regardless of rounding errors, it’s very likely that m_{RH+}\leq m_{RH}.

Ineq. (?) indicates that we can locally manipulate \hat{\alpha}, \tilde{\alpha} in the early epochs (e.g., by temporarily using a different h) to address the issue of a poor initial w_0^0 (or w_0^s) with an unexpectedly large \sigma. To our best know, this issue can not be effectively resolved in many existing methods, e.g., [yang] [yang3] [nguyen] [nguyen2] [yangy] [liu3].

For a class of ill-conditioned objective functions P(\cdot) under L \gg \mu, (?) implies m_{RH+}+1\leq m_{RH}+1\approx \frac{1}{\hat{\alpha}}\left(m_{R}+1\right), suggesting the inner speedup is approximately proportional to \mathcal{O}\left(\frac{1}{\hat{\alpha}}\right). In ill-conditioning, we tolerate towards the functional form and the decay rate of h, but remain focus on \hat{\alpha}.

Next, we evaluate the workload in terms of incremental first order oracle (IFO) complexity model in Agarwal et al.[agarwal]. In [ghadimi], it’s SFO under stochastic settings. MB-SARAH-RHBB/RHBB+ are IFO algorithms that are specified through calls to an IFO, regardless of P(\cdot). Each epoch invokes SFO at most 2bm times in the recursive gradient evaluations (?), corresponding to an overall cost of \mathcal{O}\left(n+2bm\right) SFOs. Since the adaptor h is mentor-specified, we can force \hat{\alpha} and \tilde{\alpha} to be arbitrarily large and small. By setting L+\frac{1-\tilde{\alpha}}{\hat{\alpha}}\mu=\mathcal{O}\left(L\right) and L_q+\frac{1-\tilde{\alpha}}{\hat{\alpha}}\mu_q=\mathcal{O}\left(L_q\right), it’s sufficient to have m =\mathcal{O}\left(\frac{\overline{b}\mu}{\varepsilon \gamma\hat{\alpha}}\right) and m =\mathcal{O}\left(\frac{\overline{b}\mu_q}{\varepsilon \gamma\hat{\alpha}}\right). Therefore, we obtain the following conclusions for the complexity bounds.

On the basis of Theorem Proof, we can proceed to the analysis of multiple outer steps, and we establish the convergence of MB-SARAH-RHBB and MB-SARAH-RHBB+ in subsequent Theorem ?.

Deferred to the Appendix D.

Theorem ? indicates that the outer \{\left\|\nabla P\left(\widetilde{w}_s\right)\right\|^2\} converges linearly in expectation. Assume that \left\|\nabla P\left(\widetilde{w}_0\right)\right\|^2=\zeta, to obtain \mathbb{E}\left[\left\|\nabla P\left(\widetilde{w}_s\right)\right\|^2\right] <\varepsilon in MB-SARAH-RHBB, the number of outer epoch s must satisfy

It infers s_{RH}=\lceil \frac{log(\zeta)-log(\varepsilon)}{log(\hat{\alpha}\kappa+1-\tilde{\alpha})-log(\kappa)+log(\gamma(m+1))-log(\overline{b})}\rceil. Compared to [yang] with s_{R}=\lceil\frac{log(\zeta)-log(\epsilon)}{log(\gamma(m+1))-log(\overline{b})}\rceil, the overhead of the outer epoch decreases. In MB-SARAH-RHBB+, both \mu_r\leq1, \kappa^+\geq\kappa implies s_{RH+}\leq s_{RH}, indicating the iterative cost can be further reduced via the effective sampling. In ill-conditioning, we then have s_{RH^+}\leq s_{RH}\approx\lceil\frac{log(\zeta)-log(\varepsilon)}{log(\hat{\alpha})+log(\gamma(m+1))-log(\overline{b})}\rceil.

Furthermore, our analysis can be refined to obtain smaller rate constants in some gradient dominated scenarios (see in Polyak et al.[polyak], Reddi et al. [reddi]). If P(\cdot) is \delta-gradient dominated with \delta<\frac{1}{2\mu}, we derive the rate constants

for MB-SARAH-RHBB and MB-SARAH-RHBB+, respectively. The theoretical convergence speed further increases in virtue of 2\mu\delta<1.

Compared with MB-SARAH [nguyen2], MB-SARAH-RBB [yang], MB-SARAH-HD [yang5], iSARAH-BB [yangy], Corollary ? indicates that MB-SARAH-RHBB/RHBB+ have lower complexity when using an appropriate adaptor h(\cdot) and a proper \overline{b}.

Deferred to the Appendix F.