A Dynamic Sampling Adaptive-SGD Method for Machine Learning

Achraf Bahamou and Donald Goldfarb

arXiv: 1912.13357v2 · PDF · 3 March 2020
Categories: cs.LG, math.OC, stat.ML
Licence: CC BY 4.0 The site shows the text.

Abstract

We propose a stochastic optimization method for minimizing loss functions, expressed as an expected value, that adaptively controls the batch size used in the computation of gradient approximations and the step size used to move along such directions, eliminating the need for the user to tune the learning rate. The proposed method exploits local curvature information and ensures that search directions are descent directions with high probability using an acute-angle test and can be used as a method that has a global linear rate of convergence on self-concordant functions with high probability. Numerical experiments show that this method is able to choose the best learning rates and compares favorably to fine-tuned SGD for training logistic regression and DNNs. We also propose an adaptive version of ADAM that eliminates the need to tune the base learning rate and compares favorably to fine-tuned ADAM on training DNNs. In our DNN experiments, we rarely encountered negative curvature at the current point along the step direction in DNNs.

Items 14

Definition

Definition 1

CINE

A convex function f : \mathbb{R}^{n} \rightarrow \mathbb{R} is self-concordant if there exists a constant \kappa such that for every x \in \mathbb{R}^{n} and every h \in \mathbb{R}^{n}, we have :

\left|\nabla^{3} f(x)[h, h, h]\right| \leq \kappa\left(\nabla^{2} f(x)[h, h]\right)^{3 / 2},

f is standard self-concordant if the above is satisfied for \kappa = 2.

Lemma

Lemma 1

AYXF

(Lemma 4.1, [gaogoldself])

For F standard self-concordant, for all 0 \leq t<\frac{1}{\delta_k}:

\quad F(x_k+t d_k) \leq F(x_k)+t g(x_k)^{T} d_k-\delta_k t-\log (1-\delta_k t).

Used by Proof

Lemma

Lemma 2

XWM9

(Lemma 2 in [Roosta-Khorasani2019] Under Assumptions 1. and 2., given any 0<\varepsilon, p<1 \text { and } x, d \in \mathbb{R}^{n}, if |X| \geq 16 \kappa^{2} \ln (2 n / p) / \varepsilon^{2}, we have:

\mathbb{P}\left(\left\| \nabla^{2} F_X(x)-\nabla^{2} F(\mathbf{x})\right\| \leq \varepsilon m\right) \geq 1-p,

where \|.\| is the operator norm induced by the Euclidean norm.

Used by Proof

Lemma

Lemma 3

ZGXP

(Lemma 3 in [Roosta-Khorasani2019] If for any x \in \mathbb{R}^{n} and any 0<\nu \left\|\nabla F_{S}(\mathbf{x})\right\| , \delta <1 and |S| \geq\left[\frac{G(\mathrm{x})}{\left.\left\|\nabla F_{S}(\mathrm{x})\right\| \nu\right)}\right]^{2}(1+\sqrt{8 \ln \frac{1}{\delta}})^{2}, then

\mathbb{P}\left(\left\|\nabla F(\mathbf{x})-\nabla F_{S}(x)\right\| \leq \nu\left\|\nabla F_{S}(\mathbf{x})\right\|\right) \geq 1-\delta,

where \left\|\nabla F_{i}(\mathrm{x})\right\| \leq G(\mathrm{x}), \quad \forall i=1, \ldots, n

Used by Theorem 1, Theorem 1, Proof

Theorem

Theorem 1

4HOM

Let 0 <\nu\gamma, \varepsilon, p, \delta < 1. Suppose that F satisfies the Assumptions 1-4 and that we have an efficient way to compute G(x) where \left\|\nabla F_{i}(\mathrm{x})\right\| \leq G(\mathrm{x}), \quad \forall i=1, \ldots, n. Let \{x_k, X_k, S_k\} be the set of iterates, and sample Hessian and gradient batches generated by taking the step t_{k, \varepsilon} at iteration k starting from any x_0, where |S_{k}|, |X_{k}| are chosen such that Lemma 3 and Lemma 2 are satisfied at each iteration. Then, for any k, with probability (1-p)(1-\delta) we have :

(F(x_{k+1}) - F(x^*)) \leq \rho (F(x_k) - F(x^*)),

where \rho= 1 - \frac{m(1-\nu)^2(1-\varepsilon)}{2(1+\nu^2) M (1+\frac{\gamma}{\sqrt{m}})}.

Uses Lemma 3 · Used by Proof

Lemma

Lemma 4

0OHU

Suppose that F satisfies Assumptions 1-4, Let \{x_k\} be the iterates generated by the idealized Algorithm ? where |S_{k}| is chosen such that the (exact) acute-angle test (?) and Hessian sub-sampling test (?) are satisfied at each iteration for any given constants 0 <\nu, \varepsilon, p < 1. Then, for all k, starting from any x_0,

\begin{aligned} \frac{(\nabla F_{S_k}(x_k)^{T} \nabla F(x_k))^2}{ \left\lVert\nabla F_{S_k}(x_k)\right\rVert^2} \geq (1-\nu^2) \left\lVert\nabla F(x_k)\right\rVert^2. \end{aligned}

with probability at least 1-p. Moreover, with probability (1-p)^2, we have

F(x_{k+1}) \leq F(x_{k}) - \frac{\alpha}{2} \|\nabla F(x_k)\|^2,

where \alpha = \frac{(1-\nu^2)(1-\varepsilon)}{M (1+\frac{\gamma}{\sqrt{m}})} > 0.

Theorem

Theorem 2

DXLO

F(x_k) - F(x^*) \leq \rho^k (F(x_0) - F(x^*)),

where \rho= 1 -m \alpha = 1 - \frac{m(1-\nu^2)(1-\varepsilon)}{M (1+\frac{\gamma}{\sqrt{m}})}.

Notation

GABM

Theorem

Theorem 1

B5N4

(F(x_{k+1}) - F(x^*)) \leq \rho (F(x_k) - F(x^*)),

where

\rho= 1 - \frac{m(1-\nu)^2(1-\varepsilon)}{2(1+\nu^2) M (1+\frac{\gamma}{\sqrt{m}})}.

Uses Lemma 3

Proof of Theorem 1

By Assumption 3, choosing \nu \leq \frac{1}{\gamma} insures that the conditions for Lemma 3 are satisfied. Therefore, we have with probability 1-\delta:

\begin{aligned} \text{Lemma } \ref{gkprob} &\implies \left\|g_{k}\right\|^{2}-g_{k}^{T} g=g_{k}^{T}\left(g_{k}-g\right) \leq\left\|g_{k}\right\|\left\|g-g_{k}\right\| \leq \nu\left\|g_{k}\right\|^{2}\\ &\implies g_{k}^{T} g \geq(1-\nu)\left\|g_{k}\right\|^{2}. \end{aligned}

Also:

\begin{aligned} \eqref{gkprob} \implies \|g\|^{2}=\left\|g-g_{k}+g_{k}\right\|^{2} \leq 2\left\|g-g_{k}\right\|^{2}+2\left\|g_{k}\right\|^{2} \leq 2\left(1+\nu^{2}\right)\left\|g_{k}\right\|^{2}. \end{aligned}

Hence, with probability 1-\delta:

\begin{aligned} \frac{\left(g_{k}^{T} g\right)^{2}}{\left\|g_{k}\right\|^{2}} \geq(1-\nu)^{2}\left\|g_{k}\right\|^{2} \geq \frac{(1-\nu)^{2}}{2\left(1+\nu^{2}\right)}\|g\|^{2} . \end{aligned}

Let \Delta (\delta_k, t) = (\delta_k + \rho_k)t + log(1-\delta_k t), using Lemma (1), we have:

\quad F(x_k+t d_k) \leq F(x_k)-\Delta (\delta_k, t).

If we sample the Hessian at the rate described in Lemma 2, the following inequality is satisfied with probability (1-p):

\inf_{d} \frac{|d^T(\nabla^{2} F_S(x)-\nabla^{2} F(\mathbf{x})) d|}{\|d\|^2} = \left\| \nabla^{2} F_S(x)-\nabla^{2} F(\mathbf{x})\right\| \leq \varepsilon m \leq \varepsilon \inf_{d} \frac{d^T \nabla^{2} F(\mathbf{x}) d}{\|d\|^2}.

Taking d = g_k results into the following inequalities being satisfied with probability (1-p):

\frac{\hat{\delta_k}}{\sqrt{1+\varepsilon}} \leq \delta_k \leq \frac{\hat{\delta_k}}{\sqrt{1-\varepsilon}} = \hat{\delta}_{k, \varepsilon}.

We have, for t<\frac{1}{\delta}:

\frac{\partial \Delta}{\partial \delta}(\delta, t) = t-\frac{t}{1-\delta t} = \frac{\delta t^2}{1-\delta t} < 0.

Using the monotonicity of \Delta (., t), (14) and the choice of step size as t_{k, \varepsilon} = \frac{\rho_{k}}{\left(\rho_{k}+\hat{\delta}_{k, \varepsilon}\right) \hat{\delta}_{k, \varepsilon}}, we have:

\quad F(x_k+t_{k, \varepsilon} d_k) \leq F(x_k)-\Delta(\hat{\delta}_{k, \varepsilon}, t_{k, \varepsilon}).

The choice of t_{k, \varepsilon} is possible because we now prove that t_{k, \varepsilon} < \frac{1}{\delta_k}. Let t(\delta) = \frac{\rho_k}{\delta (\delta+\rho_k)} clearly t(.) is decreasing as a function of \delta. Using the inequality (14) we have

t_{k, \varepsilon} = t(\hat{\delta}_{k, \varepsilon}) \leq t(\delta_k) = \frac{1}{\delta_k}-\frac{1}{\delta_k + \rho_k} \leq \frac{1}{\delta_k}.

Now we obtain a decrease guarantee of the function with probability (1-p)

: \quad F(x_k+t_{k, \varepsilon} d_k) \leq F(x_k)-\Delta(\hat{\delta}_{k, \varepsilon}, t_{k, \varepsilon}) = F(x_k)- \omega (\eta_{k, \varepsilon} ).

Now, using the Cauchy-Schwarz inequality, inequalities (14) and Assumption 1-3 we have:

|\eta_{k, \varepsilon}| = |\frac{\rho_k}{\frac{\hat{\delta}_{k}}{\sqrt{1-\varepsilon}}}| \leq \frac{\|g\| \|g_k\|}{\delta_k} \leq \frac{\|g\| \|g_k\|}{\sqrt{g_k^T \nabla^2 F(x_k) g_k}} \leq \frac{\|g\|}{\sqrt{m}} \leq \frac{\gamma}{\sqrt{m}}.

By observing that \omega(z) = z - log(1+z) satisfies \omega(z) \geq \frac{1}{2}(1+\Gamma)^{-1} z^{2} for all z \in[0, \Gamma], with probability (1-p), we have:

F(x_k+t_{k, \varepsilon}d_k) \leq F(x_k) - \frac{1-\varepsilon}{2(1+\frac{\gamma}{\sqrt{m}})} \hat{\eta}_{k}^2,

with \hat{\eta}_{k}=\frac{g_{k}^{T} g}{\hat{\delta}_{k}}. Using Assumption 2, we have \hat{\delta}_{k}^2 \leq M \|g_k\| ^2. Therefore, we have:

F(x_k+t_{k, \varepsilon} d_k) \leq F(x_k) - \frac{1-\varepsilon}{2 M (1+\frac{\gamma}{\sqrt{m}})} \frac{(g^T g_k)^2}{\|g_k\|^2}.

Using (?), we have with probability (1-p)(1-\delta):

F(x_k+t_{k, \varepsilon} d_k) \leq F(x_k) - \frac{(1-\nu)^2(1-\varepsilon)}{4(1+\nu^2) M (1+\frac{\gamma}{\sqrt{m}})} \|\nabla F(x_k)\|^2 .

It is well known [bertsekas2003convex] that for strongly convex functions:

\|\nabla F(x_k)\|^2 \geq 2m [F(x_k) - F(x^*)].

Substituting this in (?) and subtracting F(x^*) from both sides, we obtain with probability (1-p)(1-\delta):

\begin{aligned} F(x_{k+1}) - F(x^*) &\leq \rho (F(x_k) - F(x^*)), \end{aligned}

with :

\rho= 1 - \frac{m(1-\nu)^2(1-\varepsilon)}{2(1+\nu^2) M (1+\frac{\gamma}{\sqrt{m}})},

which is the desired result.

Uses Lemma 3, Lemma 1, Lemma 2, Theorem 2

Lemma

Lemma 4

6GA3

\begin{aligned} \frac{(\nabla F_{S_k}(x_k)^{T} \nabla F(x_k))^2}{ \left\lVert\nabla F_{S_k}(x_k)\right\rVert^2} \geq (1-\nu^2) \left\lVert\nabla F(x_k)\right\rVert^2. \end{aligned}

With probability at least 1-p. Moreover, with probability (1-p)^2, we have

F(x_{k+1}) \leq F(x_{k}) - \frac{\alpha}{2} \|\nabla F(x_k)\|^2.

where

\begin{aligned} \alpha = \frac{(1-\nu^2)(1-\varepsilon)}{M (1+\frac{\gamma}{\sqrt{m}})} > 0. \end{aligned}

Proof of Lemma 4

Since the (exact) acute-angle test (?) is satisfied, using Markov’s inequality on the positive random variable U_k := \| \frac{\nabla F_{S_{k}}\left(x_{k}\right)}{\|\nabla F_{S_{k}}\left(x_{k}\right)\|} - \frac{\nabla F_{S_{k}}\left(x_{k}\right)^{T} \nabla F\left(x_{k}\right)}{\|\nabla F_{S_{k}}\left(x_{k}\right)\|\left\|\nabla F\left(x_{k}\right)\right\|^{2}} \nabla F\left(x_{k}\right)\|^{2}, we have \eqref{eq:exactest} \implies \mathbb{E}(U_k) \leq p \nu^2 and:

\begin{aligned} \mathbb{P}(U_k \leq \nu ^2) = 1-\mathbb{P}(U_k > \nu ^2) \geq 1 - \frac{\mathbb{E}(U_k)}{\nu ^2} \geq 1- \frac{p \nu ^2}{\nu ^2} = 1-p. \end{aligned}

Therefore, the following inequality is satisfied with probability at least 1-p:

\begin{aligned} \| \frac{\nabla F_{S_{k}}\left(x_{k}\right)}{\|\nabla F_{S_{k}}\left(x_{k}\right)\|} - \frac{\nabla F_{S_{k}}\left(x_{k}\right)^{T} \nabla F\left(x_{k}\right)}{\|\nabla F_{S_{k}}\left(x_{k}\right)\|\left\|\nabla F\left(x_{k}\right)\right\|^{2}} \nabla F\left(x_{k}\right)\|^{2} \leq \nu^2 \end{aligned}

\begin{aligned} (\eqref{theorem2g}) &\implies 1 - 2\frac{(\nabla F_{S_k}(x_k)^{T} \nabla F(x_k))^2}{ \left\lVert\nabla F_{S_k}(x_k)\right\rVert^2 \left\lVert\nabla F(x_k)\right\rVert^2} + \frac{(\nabla F_{S_k}(x_k)^{T} \nabla F(x_k))^2}{ \left\lVert\nabla F_{S_k}(x_k)\right\rVert^2 \left\lVert\nabla F(x_k)\right\rVert^2} \leq \nu^2 \\ &\implies 1 - \frac{(\nabla F_{S_k}(x_k)^{T} \nabla F(x_k))^2}{ \left\lVert\nabla F_{S_k}(x_k)\right\rVert^2 \left\lVert\nabla F(x_k)\right\rVert^2} \leq \nu^2 \\ &\implies \frac{(\nabla F_{S_k}(x_k)^{T} \nabla F(x_k))^2}{ \left\lVert\nabla F_{S_k}(x_k)\right\rVert^2} \geq (1-\nu^2) \left\lVert\nabla F(x_k)\right\rVert^2. \end{aligned}

Therefore with probability at least 1-p:

\frac{\left(g_{k}^{T} g\right)^{2}}{\left\|g_{k}\right\|^{2}} \geq(1-\nu^{2})\|g\|^{2}.

Since the Hessian sub-sampling test (?) is satisfied at each iteration, using Markov’s inequality on the positive random variable V_k := (\hat{\delta_k}^2 - \delta_k^2)^2, we have \eqref{dhdtest} \implies \mathbb{E}(V_k) \leq p\varepsilon^2 \delta_k^4 and:

\begin{aligned} \mathbb{P}(V_k \leq \varepsilon^2 \delta_k^4) = 1-\mathbb{P}(V_k > \varepsilon^2 \delta_k^4) \geq 1 - \frac{\mathbb{E}(V_k)}{\varepsilon^2 \delta_k^4} \geq 1- \frac{p \varepsilon^2 \delta_k^4}{\varepsilon^2 \delta_k^4} = 1-p. \end{aligned}

Therefore, the following inequality is satisfied with probability at least 1-p:

\frac{\hat{\delta_k}}{\sqrt{1+\varepsilon}} \leq \delta_k \leq \frac{\hat{\delta_k}}{\sqrt{1-\varepsilon}} = \hat{\delta}_{k, \varepsilon}.

Following similar arguments to those used in the proof of Theorem 1, we have with probability (1-p)^2:

F(x_{k+1}) \leq F(x_{k}) - \frac{\alpha}{2} \|\nabla F(x_k)\|^2,

where

\begin{aligned} \alpha = \frac{(1-\nu^2)(1-\varepsilon)}{M (1+\Gamma)} > 0. \end{aligned}

Which is the desired result.

Uses Theorem 1

Theorem

Theorem 2

MZRO

F(x_k) - F(x^*) \leq \rho^k (F(x_0) - F(x^*)),

where

\rho= 1 -m \alpha = 1 - \frac{m(1-\nu^2)(1-\varepsilon)}{M (1+\frac{\gamma}{\sqrt{m}})}.

Used by Proof

Proof of Theorem 2

It is well known [bertsekas2003convex] that for strongly convex functions

\|\nabla F(x_k)\|^2 \geq 2m [F(x_k) - F(x^*)].

Substituting this into (?) and subtracting F(x^*) from both sides, we obtain with probability (1-p)^2:

\begin{aligned} F(x_{k+1}) - F(x^*) &\leq \rho (F(x_k) - F(x^*)), \end{aligned}

with :

\rho= 1 -m \alpha = 1 - \frac{m(1-\nu^2)(1-\varepsilon)}{M (1+\frac{\gamma}{\sqrt{m}})},

from which the theorem follows by induction on k.

Uses Proof

Source and licence

The text on this page is from arXiv:1912.13357v2 by Achraf Bahamou and Donald Goldfarb, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.

The record of each item is in analysis/objects/1912/1912.13357.jsonl.