Descending Everest with First-Order Oracles: Oracle Complexity of Convex Optimization under (L_0,L_1)-Smoothness

• Necessity in (a). The gradient of a convex differentiable function is continuous [rockafellar1970convex, Theorem 25.5], so on every ball it is bounded by some number M. If the segment [x,y] lies in such a ball, then by (2.1)

  \|\nabla f(x)-\nabla f(y)\|\le\ell(M)\|x-y\| .

  Hence, the map t\mapsto\nabla f(x+tv) is Lipschitz on every segment, and therefore locally absolutely continuous and differentiable for almost all t. At such a point t, apply (2.1) to the points x+tv and x+(t+h)v, divide by |h| and let h tend to zero. By the continuity of \nabla f and \ell we obtain (?).

• Sufficiency in (a). Let v=y-x. By the fundamental theorem of calculus and the monotonicity of \ell

  \begin{aligned} \|\nabla f(y)-\nabla f(x)\| &\le\int_0^1\ell\bigl(\|\nabla f(x+tv)\|\bigr)\|v\|\,dt\\ &\le\ell\Bigl(\sup_{u\in[x,y]}\|\nabla f(u)\|\Bigr)\|x-y\| . \end{aligned}

• Statement (b). For f\in C^2 the derivative of the map t\mapsto\nabla f(x+tv) equals \nabla^2f(x+tv)v and is continuous in t. Therefore, condition (?) for almost all t is equivalent to the inequality

  \|\nabla^2f(x)v\|\le\ell\bigl(\|\nabla f(x)\|\bigr)\|v\|

  for all x and v. The Hessian of a convex C^2 function is a symmetric positive semidefinite matrix, and for it this inequality is equivalent to (?). It remains to apply (a).

• Decrease of the function. Since L_0+L_1\|\nabla f(y)\|\ge L_1\|\nabla f(y)\|, the step length

  \|y^+-y\|=\frac{\|\nabla f(y)\|}{4\bigl(L_0+L_1\|\nabla f(y)\|\bigr)}

  is at most 1/(4L_1), and the factor e^{L_1\|y^+-y\|} in (?) is at most e^{1/4}. Therefore,

  \begin{aligned} f(y^+) &\mathrel{\underset{\eqref{eq:local-descent}}{\le}}f(y)-\frac{\|\nabla f(y)\|^2}{4\bigl(L_0+L_1\|\nabla f(y)\|\bigr)}+\frac{e^{1/4}\|\nabla f(y)\|^2}{32\bigl(L_0+L_1\|\nabla f(y)\|\bigr)}\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:local-descent}$}]{}}{=}}f(y)-\Bigl(\frac14-\frac{e^{1/4}}{32}\Bigr)\frac{\|\nabla f(y)\|^2}{L_0+L_1\|\nabla f(y)\|}\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:local-descent}$}]{(\mathrm a)}}{\le}}f(y)-\frac{5\|\nabla f(y)\|^2}{24\bigl(L_0+L_1\|\nabla f(y)\|\bigr)} . \end{aligned}

  Here step (a) uses the inequality e^{1/4}<4/3, which follows from e^t\le1/(1-t) for 0\le t<1.

• Decrease of the distance to the minimizer. We have

  \begin{aligned} \langle \nabla f(y),y-x^\ast\rangle &\mathrel{\underset{(\mathrm b)}{\ge}}f(y)-f^\ast \mathrel{\underset{(\mathrm c)}{\ge}}f(y)-f(y^+)\\ &\mathrel{\underset{(\mathrm d)}{\ge}}\frac{5\|\nabla f(y)\|^2}{24\bigl(L_0+L_1\|\nabla f(y)\|\bigr)} , \end{aligned}

  where step (b) uses convexity, step (c) uses the inequality f(y^+)\ge f^\ast, and step (d) uses the decrease of the function proved above. Expanding the squared norm and substituting this bound, we obtain

  \begin{aligned} \|y^+-x^\ast\|^2 &=\|y-x^\ast\|^2-\frac{\langle \nabla f(y),y-x^\ast\rangle}{2\bigl(L_0+L_1\|\nabla f(y)\|\bigr)}+\frac{\|\nabla f(y)\|^2}{16\bigl(L_0+L_1\|\nabla f(y)\|\bigr)^2}\\ &\le\|y-x^\ast\|^2-\frac{5\|\nabla f(y)\|^2}{48\bigl(L_0+L_1\|\nabla f(y)\|\bigr)^2}+\frac{3\|\nabla f(y)\|^2}{48\bigl(L_0+L_1\|\nabla f(y)\|\bigr)^2} .\qedhere \end{aligned}

For \nabla f(y)=0 the statement coincides with the convexity inequality. Let \nabla f(y)\ne0. We compare the affine approximation at y with the growth bound of Lemma A.2 at the points of the ray

where t\ge0.

• Bound for fixed t. Since \langle \nabla f(y),z_t-x^\ast\rangle=t\|\nabla f(y)\|,

  \begin{aligned} f(y)+\langle \nabla f(y),x^\ast-y\rangle+t\|\nabla f(y)\| &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle(\mathrm a)$}]{}}{=}}f(y)+\langle \nabla f(y),z_t-y\rangle\\ &\mathrel{\underset{(\mathrm a)}{\le}}f(z_t) \mathrel{\underset{(\mathrm b)}{\le}}f^\ast+\frac{L_0}{L_1^2}\bigl(e^{L_1t}-1-L_1t\bigr), \end{aligned}

  where step (a) uses convexity and step (b) uses Lemma A.2.

• Choice of t. Set

  t=\frac1{L_1}\log\Bigl(1+\frac{L_1\|\nabla f(y)\|}{L_0}\Bigr).

  Then

  e^{L_1t}=1+\frac{L_1\|\nabla f(y)\|}{L_0},

  and a direct computation using formula (?) gives

  t\|\nabla f(y)\|-\frac{L_0}{L_1^2}\bigl(e^{L_1t}-1-L_1t\bigr)=\Phi\bigl(\|\nabla f(y)\|\bigr).

  Substituting this into the previous bound, we obtain the statement of the lemma.

Denote by z(t)=v+t(x-v) the points of the segment [v,x]. The function t\mapsto f(z(t)) is convex on [0,1], and its derivative \langle \nabla f(z(t)),x-v\rangle is nondecreasing. For the point y=z(t)

• Stopping criterion. If it was triggered at a point z\in[v,x], then z lies in the ball B(x^\ast,r), and by convexity

  f(z)-f^\ast\le\langle \nabla f(z),z-x^\ast\rangle\le r\|\nabla f(z)\|\le\delta .

  From now on we assume that the stopping criterion was not triggered.

• Return at the endpoints of the segment. If \langle \nabla f(v),x-v\rangle\ge0, then the derivative is nonnegative on [0,1]. Hence, f(v)\le f(x), and for y=v the left-hand side of the second inequality of (?) is zero. If \langle \nabla f(x),x-v\rangle\le0, then for y=x the first inequality of (?) is obvious, and the left-hand side of the second one is nonpositive.

• Bisection. The algorithm maintains the inequalities

  \langle \nabla f(z(t_-)),x-v\rangle<0<\langle \nabla f(z(t_+)),x-v\rangle .

  If the derivative at the midpoint is zero, then this midpoint minimizes the function on the segment, and both inequalities (?) hold for it. Otherwise, after m steps t_+-t_-=2^{-m}.

• Decrease for y=z(t_+). The derivative at the point t_+ is positive, so the function is nondecreasing on [t_+,1], and f(y)\le f(x).

• Error for y=z(t_+). The distance between the points z(t_+) and z(t_-) equals 2^{-m}\|x-v\|, and \|x-v\|\le2r. By condition (3.3)

  L_1\|z(t_+)-z(t_-)\|\le2^{1-m}L_1r\le\frac{2L_1r}{128\bigl(1+L_1r\bigr)^2}\le\frac1{32} .

  By Lemma A.1(b) with base point z(t_+) and the inequality e^s-1\le se^s

  \|\nabla f(z(t_+))-\nabla f(z(t_-))\|\le2\bigl(L_0+L_1\|\nabla f(y)\|\bigr)\|z(t_+)-z(t_-)\| .

  Since \langle \nabla f(z(t_-)),x-v\rangle<0 and t_+\le1,

  \begin{aligned} \langle \nabla f(y),y-v\rangle &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:search-m}$}]{}}{\le}}t_+\langle \nabla f(z(t_+))-\nabla f(z(t_-)),x-v\rangle\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:search-m}$}]{(\mathrm a)}}{\le}}2r\|\nabla f(z(t_+))-\nabla f(z(t_-))\|\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:search-m}$}]{}}{\le}}2^{3-m}r^2\bigl(L_0+L_1\|\nabla f(y)\|\bigr)\\ &\mathrel{\underset{\eqref{eq:search-m}}{\le}}\frac{r^2\bigl(L_0+L_1\|\nabla f(y)\|\bigr)}{16\bigl(1+L_0r^2/\delta+L_1r\bigr)^2} , \end{aligned}

  where step (a) uses the inequalities t_+\le1 and \|x-v\|\le2r.

• Relation to the gradient. The stopping criterion does not hold at y, that is, r\|\nabla f(y)\|>\delta. Therefore,

  \begin{aligned} \frac{r\bigl(L_0+L_1\|\nabla f(y)\|\bigr)}{\|\nabla f(y)\|} &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle(\mathrm b)$}]{}}{=}}\frac{L_0r}{\|\nabla f(y)\|}+L_1r\\ &\mathrel{\underset{(\mathrm b)}{<}}\frac{L_0r^2}\delta+L_1r <1+\frac{L_0r^2}\delta+L_1r , \end{aligned}

  where step (b) uses the inequality r\|\nabla f(y)\|>\delta. Squaring and dividing by 16(L_0+L_1\|\nabla f(y)\|), we obtain

  \frac{r^2\bigl(L_0+L_1\|\nabla f(y)\|\bigr)}{16\bigl(1+L_0r^2/\delta+L_1r\bigr)^2}<\frac{\|\nabla f(y)\|^2}{16\bigl(L_0+L_1\|\nabla f(y)\|\bigr)} ,

  and the second inequality of (?) holds.

• Number of calls. The algorithm makes two calls at the endpoints of the segment and at most m calls at midpoints.

Induction on k. For k=0, inequality (3.4) coincides with the condition \|x^0-x^\ast\|\le R. Suppose the statement holds for k-1.

• The iteration is well defined. By the induction hypothesis, the points v^{k-1} and x^{k-1} lie in the ball B(x^\ast,R). The ball is convex, so the segment [v^{k-1},x^{k-1}] and the point y^k lie in it, and Lemma 3.3 applies. The stopping criterion was not triggered, so inequalities (?) hold. Together with Lemma 3.1 they give x^k\in B(x^\ast,R) and

  \begin{aligned} &f(y^k)\le f(x^{k-1}),\\ &\langle \nabla f(y^k),y^k-v^{k-1}\rangle\le\frac{\|\nabla f(y^k)\|^2}{16\bigl(L_0+L_1\|\nabla f(y^k)\|\bigr)},\\ &f(x^k)\le f(y^k)-\frac{5\|\nabla f(y^k)\|^2}{24\bigl(L_0+L_1\|\nabla f(y^k)\|\bigr)} . \end{aligned}

• Distance. Since v^k=v^{k-1}-a_k\nabla f(y^k),

  \begin{aligned} \MoveEqLeft\frac12\|v^k-x^\ast\|^2-\frac12\|v^{k-1}-x^\ast\|^2\\ &=a_k\langle \nabla f(y^k),y^k-v^{k-1}\rangle-a_k\langle \nabla f(y^k),y^k-x^\ast\rangle+\frac12a_k^2\|\nabla f(y^k)\|^2 . \end{aligned}

  We bound the first two terms on the right:

  \begin{aligned} a_k\langle \nabla f(y^k),y^k-v^{k-1}\rangle &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\text{Lemma~\ref{lem:anchored}}$}]{\eqref{eq:energy-step}}}{\le}}\frac{a_k\|\nabla f(y^k)\|^2}{16\bigl(L_0+L_1\|\nabla f(y^k)\|\bigr)},\\ -a_k\langle \nabla f(y^k),y^k-x^\ast\rangle &\mathrel{\underset{\text{Lemma~\ref{lem:anchored}}}{\le}}-a_k\bigl(f(y^k)-f^\ast\bigr)-a_k\Phi\bigl(\|\nabla f(y^k)\|\bigr). \end{aligned}

• Optimality gap. Since A_k=A_{k-1}+a_k,

  \begin{aligned} \MoveEqLeft A_k\bigl(f(x^k)-f^\ast\bigr)-A_{k-1}\bigl(f(x^{k-1})-f^\ast\bigr)\\ &\mathrel{\underset{\eqref{eq:energy-step}}{\le}}A_k\bigl(f(y^k)-f^\ast\bigr)-\frac{5A_k\|\nabla f(y^k)\|^2}{24\bigl(L_0+L_1\|\nabla f(y^k)\|\bigr)}-A_{k-1}\bigl(f(y^k)-f^\ast\bigr)\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:energy-step}$}]{}}{=}}a_k\bigl(f(y^k)-f^\ast\bigr)-\frac{5A_k\|\nabla f(y^k)\|^2}{24\bigl(L_0+L_1\|\nabla f(y^k)\|\bigr)} . \end{aligned}

• Sum. Multiplying equation (?) by \|\nabla f(y^k)\|^2, we obtain

  a_k\Phi\bigl(\|\nabla f(y^k)\|\bigr)=a_k^2\|\nabla f(y^k)\|^2-\frac{A_k\|\nabla f(y^k)\|^2}{7\bigl(L_0+L_1\|\nabla f(y^k)\|\bigr)} .

  We add (Proof), the two bounds on its terms and the bound on the optimality gap, and then substitute this equality:

  \begin{aligned} \MoveEqLeft\Bigl[A_k\bigl(f(x^k)-f^\ast\bigr)+\frac12\|v^k-x^\ast\|^2\Bigr]-\Bigl[A_{k-1}\bigl(f(x^{k-1})-f^\ast\bigr)+\frac12\|v^{k-1}-x^\ast\|^2\Bigr]\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:coef-basic}$}]{}}{\le}}\Bigl(\frac{a_k}{16}-\frac{5A_k}{24}\Bigr)\frac{\|\nabla f(y^k)\|^2}{L_0+L_1\|\nabla f(y^k)\|}-a_k\Phi\bigl(\|\nabla f(y^k)\|\bigr)+\frac12a_k^2\|\nabla f(y^k)\|^2\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:coef-basic}$}]{}}{=}}\Bigl(\frac{a_k}{16}-\frac{5A_k}{24}+\frac{A_k}7\Bigr)\frac{\|\nabla f(y^k)\|^2}{L_0+L_1\|\nabla f(y^k)\|}-\frac12a_k^2\|\nabla f(y^k)\|^2\\ &\mathrel{\underset{\eqref{eq:coef-basic}}{\le}}-\frac{A_k\|\nabla f(y^k)\|^2}{336\bigl(L_0+L_1\|\nabla f(y^k)\|\bigr)}-\frac12a_k^2\|\nabla f(y^k)\|^2\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:coef-basic}$}]{}}{\le}}-\frac12a_k^2\|\nabla f(y^k)\|^2 . \end{aligned}

  Adding this inequality to inequality (3.4) for k-1, we obtain (3.4) for k. It implies \|v^k-x^\ast\|\le R, which completes the induction.

• Two sums. By Lemma 3.4 and the condition on the optimality gap

  \begin{aligned} &L_0A_K<64L_1^2R^2,\\ &\sum_{i=1}^Ka_i^2\|\nabla f(y^i)\|^2\le R^2 . \end{aligned}

• Bounds for one iteration. From inequalities (?)

  \begin{gathered} \begin{aligned} \sqrt{L_0A_i}-\sqrt{L_0A_{i-1}}=\frac{\sqrt{L_0}\,a_i}{\sqrt{A_i}+\sqrt{A_{i-1}}} &\mathrel{\underset{A_{i-1}\le A_i}{\ge}}\frac{\sqrt{L_0}\,a_i}{2\sqrt{A_i}}\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyleA_{i-1}\le A_i$}]{\eqref{eq:coef-basic}}}{\ge}}\frac1{2\sqrt7}\sqrt{\frac{L_0}{L_0+L_1\|\nabla f(y^i)\|}}, \end{aligned}\\ \begin{aligned} L_0A_i=\frac{L_0A_i^2}{A_i} &\mathrel{\underset{\eqref{eq:coef-basic}}{\ge}}\frac{L_0a_i^2}{A_i}\\ &\mathrel{\underset{\eqref{eq:coef-basic}}{\ge}}\frac{L_0}{7\bigl(L_0+L_1\|\nabla f(y^i)\|\bigr)} . \end{aligned} \end{gathered}

  Divide equation (?) by a_i and multiply by L_1\|\nabla f(y^i)\|:

  L_1a_i\|\nabla f(y^i)\| =\frac{L_1\Phi(\|\nabla f(y^i)\|)}{\|\nabla f(y^i)\|} +\frac{A_i}{7a_i}\cdot\frac{L_1\|\nabla f(y^i)\|}{L_0+L_1\|\nabla f(y^i)\|} .

  Both terms on the right are nonnegative. By definition (?)

  \begin{aligned} \frac{L_1\Phi(\|\nabla f(y^i)\|)}{\|\nabla f(y^i)\|} &=\Bigl(1+\frac{L_0}{L_1\|\nabla f(y^i)\|}\Bigr)\log\Bigl(1+\frac{L_1\|\nabla f(y^i)\|}{L_0}\Bigr)-1\\ &\ge\log\Bigl(1+\frac{L_1\|\nabla f(y^i)\|}{L_0}\Bigr)-1 . \end{aligned}

  Multiplying (Proof) by 7a_i/A_i and adding the first inequality of (Proof), we obtain

  \begin{aligned} \MoveEqLeft2\sqrt7\Bigl(\sqrt{L_0A_i}-\sqrt{L_0A_{i-1}}\Bigr)+\Bigl(\frac{7L_1a_i^2\|\nabla f(y^i)\|}{A_i}\Bigr)^{2/3}\\ &\ge\Bigl(\frac{L_0}{L_0+L_1\|\nabla f(y^i)\|}\Bigr)^{1/2}+\Bigl(\frac{L_1\|\nabla f(y^i)\|}{L_0+L_1\|\nabla f(y^i)\|}\Bigr)^{2/3}\\ &\mathrel{\underset{(\mathrm a)}{\ge}}\frac{L_0}{L_0+L_1\|\nabla f(y^i)\|}+\frac{L_1\|\nabla f(y^i)\|}{L_0+L_1\|\nabla f(y^i)\|}=1 . \end{aligned}

  Here step (a) uses the inequalities t^{1/2}\ge t and t^{2/3}\ge t for 0\le t\le1.

• Iterations with a large product L_1a_i\|\nabla f(y^i)\|. Let J be the set of indices i\le K with L_1a_i\|\nabla f(y^i)\|\ge\sqrt{L_1R}. By the second inequality of (Proof)

  |J|\,L_1R\le\sum_{i\in J}L_1^2a_i^2\|\nabla f(y^i)\|^2\le L_1^2R^2 ,

  that is, |J|\le L_1R.

• Remaining iterations. Let I be the set of the remaining indices. If it is empty, then K=|J|\le L_1R. Otherwise, for i\in I, by (Proof) and (Proof)

  \log\Bigl(1+\frac{L_1\|\nabla f(y^i)\|}{L_0}\Bigr)-1\le L_1a_i\|\nabla f(y^i)\|<\sqrt{L_1R} ,

  and the second inequality of (Proof) gives

  L_0A_i\ge\frac17\Bigl(1+\frac{L_1\|\nabla f(y^i)\|}{L_0}\Bigr)^{-1}>\frac17e^{-\sqrt{L_1R}-1} .

  Let j be the smallest index in I. The weights A_i increase, so for i>j we have A_{i-1}>0 and

  \frac{a_i}{A_i}\le-\log\Bigl(1-\frac{a_i}{A_i}\Bigr)=\log\frac{A_i}{A_{i-1}} .

  Together with the inequality a_j\le A_j this gives

  \begin{aligned} \sum_{i\in I}\frac{a_i}{A_i} &\le1+\log\frac{A_K}{A_j} \mathrel{\underset{\eqref{eq:count-anchor-budget}}{<}}1+\log\bigl(448L_1^2R^2e^{\sqrt{L_1R}+1}\bigr)\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:count-anchor-budget}$}]{}}{=}}\sqrt{L_1R}+2+\log448+2\log(L_1R)\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:count-anchor-budget}$}]{(\mathrm b)}}{<}}5\sqrt{L_1R}+\frac92 \mathrel{\underset{L_1R\ge1}{\le}}\frac{19}2\sqrt{L_1R} . \end{aligned}

  Here step (b) uses the inequalities \log448<13/2 and \log t\le2\sqrt t-2 for t\ge1. Sum inequalities (Proof) over i\in I. The sum of the first terms is at most 2\sqrt7\sqrt{L_0A_K}<16\sqrt7L_1R. To the second terms we apply Hölder’s inequality with exponents 3 and 3/2:

  \begin{aligned} |I| &<16\sqrt7L_1R+7^{2/3}\sum_{i\in I}\bigl(L_1a_i\|\nabla f(y^i)\|\bigr)^{2/3}\Bigl(\frac{a_i}{A_i}\Bigr)^{2/3}\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:count-anchor-budget}$}]{}}{\le}}16\sqrt7L_1R+7^{2/3}\Bigl(\sum_{i\in I}L_1^2a_i^2\|\nabla f(y^i)\|^2\Bigr)^{1/3}\Bigl(\sum_{i\in I}\frac{a_i}{A_i}\Bigr)^{2/3}\\ &\mathrel{\underset{\eqref{eq:count-anchor-budget}}{<}}16\sqrt7L_1R+7^{2/3}(L_1R)^{2/3}\Bigl(\frac{19}2\sqrt{L_1R}\Bigr)^{2/3}\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:count-anchor-budget}$}]{}}{=}}\Bigl(16\sqrt7+\Bigl(\frac{133}2\Bigr)^{2/3}\Bigr)L_1R\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:count-anchor-budget}$}]{(\mathrm c)}}{<}}59L_1R . \end{aligned}

  Here step (c) uses the inequalities 16\sqrt7<42.4 and (133/2)^{2/3}<16.5, which are equivalent to 1792<42.4^2 and 133^2<4\cdot16.5^3.

• Conclusion. Every iteration belongs to exactly one of the sets J and I, therefore

  K=|J|+|I|<L_1R+59L_1R=60L_1R .\qedhere

This is Theorem 3.2 of [tyurin2026accelerated] for \ell(s)=L_0+L_1s with the accuracy parameter equal to twice the right-hand side of the first inequality of (?). We verify its assumptions.

• The author of [tyurin2026accelerated] states the result for twice differentiable functions, but notes (ibid., after Lemma 2.2) that instead of the Hessian bound the inequality

  \|\nabla f(z)-\nabla f(y)\|\le\bigl(L_0+L_1\|\nabla f(y)\|\bigr)\frac{e^{L_1\|z-y\|}-1}{L_1}

  suffices. For our class this is Lemma A.1(b).

• The function in the assumptions of Theorem 3.2 is strictly increasing on [0,\infty), since for t>0

  \frac{d}{dt}\,\frac{t^2}{2(L_0+4L_1t)}=\frac{t(L_0+2L_1t)}{(L_0+4L_1t)^2}>0 .

• The condition of Theorem 3.2 on the accuracy parameter holds with equality:

  \ell\Bigl(8\sqrt{\ell(0)\,\frac{L_0}{64L_1^2}}\Bigr)=L_0+8L_1\sqrt{\frac{L_0^2}{64L_1^2}}=2L_0=2\ell(0) .

• In Theorem 3.2 the starting point of the accelerated method is obtained by gradient descent, but the proof (ibid., Appendix E.1, formula (32)) uses only its two properties (?). For the point x' they hold by assumption.

• The number of preliminary iterations in Theorem 3.2 is zero:

  \frac{\Gamma_0}{8L_0}=\frac1{512L_1^2R^2}\le\frac49 ,

  therefore

  \max\Bigl\{1+\frac12\log_{3/2}\frac{\Gamma_0}{8L_0},\ 0\Bigr\}=0 .

Theorem 3.2 gives the bound in the statement of the proposition. With N chosen as in (3.6), its right-hand side for k=N is at most \varepsilon. The algorithm computes gradients at the points y^0,\dots,y^N, so the number N+1 of its oracle calls satisfies

• Reduction to the case L_1R\ge1. If L_1R<1, then the class \mathcal F_{L_0,L_1}^d is contained in the class \mathcal F_{L_0,1/R}^d. We then replace L_1 by 1/R in all parameters of both stages. After the replacement K=60 and m=22, so the first stage makes at most 60\cdot24=240\log_264 oracle calls, which does not exceed 240(1+L_1R)\log_2\bigl(64(1+L_1R)\bigr). From now on L_1R\ge1.

• First stage. Run Algorithm ?. Its number m satisfies condition (3.3) with r=R and \delta=L_0/(128L_1^2). Let x' be the returned point. We prove that it satisfies inequalities (?). If the stopping criterion was triggered at iteration k, then the first of them is given by Lemma 3.3. Moreover, x' lies on the segment [v^{k-1},x^{k-1}], whose endpoints lie in the ball B(x^\ast,R) by Lemma 3.4. Otherwise all K\ge60L_1R iterations are performed, the first inequality is given by Lemma 3.5, and the second one by Lemma 3.4.

• Number of first-stage calls. It is at most K(m+2). For L_1R\ge1

  \begin{aligned} &K\le60L_1R+1\le60(1+L_1R),\\ &m+2\mathrel{\underset{(\mathrm a)}{\le}}\log_2\bigl(2^{21}L_1^2R^2(1+L_1R)^2\bigr)+3\le4\log_2\bigl(64(1+L_1R)\bigr) . \end{aligned}

  Here step (a) uses the inequality 1+128L_1^2R^2+L_1R\le128L_1R(1+L_1R). Hence, the first stage makes at most 240(1+L_1R)\log_2\bigl(64(1+L_1R)\bigr) oracle calls.

• Second stage. If \varepsilon is not smaller than the right-hand side of the first inequality of (?), we return x'. Otherwise we run Algorithm ? from x' with the number of iterations (3.6). By Proposition 3.6, it finds a point with optimality gap at most \varepsilon, and its number of oracle calls is bounded as in this proposition.

Adding the numbers of calls at the two stages, we obtain the statement of the theorem. Both algorithms use only gradients.

• Continuity. The function r is convex and r(0)=0, so on [0,1]

  0\le r(p)\le p\,r'(p)\le p\,r'(1) .

  Hence, the second term of H is at most Z_nr'(1) and tends to zero together with Z_n. At the other points of K continuity is obvious.

• Second derivative along a segment. Let z,z'\in K be distinct, u=z'-z and w=z+tu, t\in[0,1]. Denote W_n=w_1+\dots+w_n and U_n=u_1+\dots+u_n, so that W_n depends on t and its derivative in t equals U_n. Set

  p=\frac w{W_n} .

  Coordinates that vanish at both points vanish on the whole segment, and the others are positive for 0<t<1. Therefore, on (0,1) the function t\mapsto H(w) is twice continuously differentiable. The second term of H is the perspective of the function r(p_1)+\dots+r(p_n). Along the segment

  \begin{aligned} \frac{dp}{dt}&=\frac{u-pU_n}{W_n},\\ \Phi''(W_n)&\mathrel{\underset{\eqref{eq:Phi-derivatives}}{=}}\frac1{L_0+L_1W_n}, \end{aligned}

  and differentiation gives

  \frac{d^2}{dt^2}H(w)=\frac{2U_n^2}{L_0+L_1W_n}+\sum_{i=1}^n\frac{r''(p_i)}{W_n}\,(u_i-p_iU_n)^2 .

  The vector p lies in the simplex, so each of its coordinates is at most its norm and

  \frac1{\sqrt n}\le\|p\|\le1 .

  Moreover, \|w\|=W_n\|p\|. We bound the coefficients in (Proof):

  \begin{aligned} \frac{r''(p_i)}{W_n} &\mathrel{\underset{\eqref{eq:r-derivatives}}{=}}\frac{4\sqrt n}{L_1W_n\,(1+\sqrt n\,p_i)} \mathrel{\underset{\eqref{eq:simplex-p}}{\ge}}\frac2{L_1W_n\|p\|}\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:r-derivatives}$}]{}}{=}}\frac2{L_1\|w\|} \ge\frac2{L_0+L_1\|w\|},\\ \frac2{L_0+L_1W_n} &\mathrel{\underset{\|p\|\le1}{\ge}}\frac{2\|p\|^2}{L_0+L_1W_n\|p\|} =\frac{2\|p\|^2}{L_0+L_1\|w\|} . \end{aligned}

  Substituting them into (Proof), we obtain

  \begin{aligned} \frac{d^2}{dt^2}H(w) &\ge\frac2{L_0+L_1\|w\|}\Bigl(\|p\|^2U_n^2+\|u-pU_n\|^2\Bigr)\\ &\mathrel{\underset{(\mathrm a)}{\ge}}\frac{\|u\|^2}{L_0+L_1\|w\|} \mathrel{\underset{(\mathrm b)}{\ge}}\frac{\|u\|^2}{L_0+L_1\max\{\|z\|,\|z'\|\}} . \end{aligned}

  Here step (a) uses the inequality \|u\|^2\le2\|pU_n\|^2+2\|u-pU_n\|^2, and step (b) uses the convexity of the norm.

• Conclusion. The restriction of H to the segment minus the quadratic term from Lemma 4.2 has a nonnegative second derivative on (0,1) and is continuous on [0,1]. Hence, it is convex, and Lemma 4.2 applies.

Let y\in\mathcal K_{j-1}, that is, y_i=0 for i\ge j, and let z=\nabla\varphi(y). We need to prove that z_l=0 for l>j. If z=0, there is nothing to prove. Otherwise Z_n>0. Let p be the vector z divided by Z_n.

• Optimality conditions. In a neighborhood of z the function H is differentiable and \partial_iH(z)=\nu+r'(p_i), where

  \nu=2\Phi'(Z_n)+\sum_{k=1}^n\bigl[r(p_k)-p_kr'(p_k)\bigr].

  The point z maximizes a concave function over the polytope K. By the Karush–Kuhn–Tucker theorem, there exists a multiplier \lambda\ge0 of the linear constraint defining K such that

  y_i+\sigma_i-\frac\lambda{G_i}\ \begin{cases}=\nu+r'(p_i), & z_i>0,\\ \le\nu, & z_i=0.\end{cases}

• Upper bound on \nu. Since r(p)\le p\,r'(p), we have \nu\le2\Phi'(Z_n). Further, Z_n\le G_1, since z\in K and all slopes are at most G_1. The function \Phi' is increasing, therefore

  \nu\le2\Phi'(G_1)\mathrel{\underset{\eqref{eq:Phi-derivatives}}{=}}\frac2{L_1}\log\Bigl(1+\frac{L_1G_1}{L_0}\Bigr).

• Lower bound on \nu. Since 0\le r'(p_j)\le r'(1) and y_j=0, in both cases (Proof) gives

  -\lambda\le G_j\nu-G_j\sigma_j+G_jr'(1) .

• Coordinate l is zero. Since y_l=0 and G_j\ge G_l,

  \begin{aligned} G_l\Bigl(\sigma_l-\frac\lambda{G_l}-\nu\Bigr) &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:ord-nu-upper}$}]{\eqref{eq:ord-nu-lower}}}{\le}}G_l\sigma_l-G_j\sigma_j+G_jr'(1)+(G_j-G_l)\nu\\ &\mathrel{\underset{\eqref{eq:ord-nu-upper}}{\le}}G_l\sigma_l-G_j\sigma_j+G_jr'(1)+(G_j-G_l)\,\frac2{L_1}\log\Bigl(1+\frac{L_1G_1}{L_0}\Bigr)\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:ord-nu-upper}$}]{\eqref{eq:shift-gap}}}{\le}}0 . \end{aligned}

  If z_l>0, then by (Proof) the left-hand side would equal G_lr'(p_l)>0. Hence, z_l=0.

• Statement (a). The point z=0 is feasible, so \varphi\ge-H(0)=0. On the other hand, H\ge0, and \varphi(-\sigma)=\max_K\{-H\}=0.

• Statement (b). Substitute the vertex G_ne_n into (?). The function \Phi is convex, \Phi(0)=0 and \Phi' is increasing, so \Phi(G_n)\le G_n\Phi'(G_n)\le G_n\Phi'(G_1). Moreover, r(1)\le r'(1). Hence,

  \begin{aligned} \varphi(y) &\ge G_n\sigma_n-2\Phi(G_n)-G_nr(1)\\ &\ge G_n\sigma_n-\frac{2G_n}{L_1}\log\Bigl(1+\frac{L_1G_1}{L_0}\Bigr)-G_n\,\frac4{L_1}\log\bigl(1+\sqrt n\bigr) \mathrel{\underset{\eqref{eq:slopes}}{=}}\frac{2G_n}{L_1}=2\varepsilon .\qedhere \end{aligned}

We bound the three quantities that make up the shifts \sigma_i:

Here step (a) uses the inequality \log(1+xe^v)\le\log(1+x)+v for x,v\ge0, step (b) uses the inequality (1+\sqrt n)^2\le4n, and step (c) uses the sum of a geometric progression together with the inequality e^{-x}(1+x)\le1. Adding these bounds, we obtain for every coordinate

All \sigma_i are positive, so \|\sigma\|\le\sqrt n\max_i\sigma_i and

The ratio of \log(4n) to n decreases and is therefore at most \log4<2, while n\log(4n)\ge\log4>1. Hence, the expression in brackets is less than 4+2\sqrt2+2<9.

The function \varphi is a convex chain from the class \mathcal F_{L_0,L_1}^n (Lemmas 4.4 and 4.5). Its minimum is zero and is attained at the point -\sigma, and for y_n=0 its value is at least 2\varepsilon (Lemma 4.6). Lemma 4.7 and condition (4.8) imply that the norm of \sigma is at most R. It remains to apply Proposition 4.3(b) with \mathrm{gap}=2\varepsilon.

Set

If n=0, the statement is obvious. Let n\ge1. We verify condition (4.8) by bounding its two terms.

• Second term. The number 4n is less than 2+L_1R, so \log(4n)\le\log(2+L_1R) and

  9n\sqrt{\log(4n)}\le9n\sqrt{\log(2+L_1R)}\mathrel{\underset{\text{def. of }n}{\le}}\frac38L_1R .

• First term. The ratio of \log^{3/2}(2+x) to x decreases for x\ge16 and is less than three eighths at the point 16. Therefore,

  \begin{aligned} 2\sqrt n\log\Bigl(1+\frac{L_1^2\varepsilon}{L_0}\Bigr) &\mathrel{\underset{L_0R^2\ge\varepsilon}{\le}}2\sqrt n\log\bigl(1+(L_1R)^2\bigr) \mathrel{\underset{1+t^2\le(2+t)^2}{\le}}4\sqrt n\log(2+L_1R)\\ &\mathrel{\underset{\text{def. of }n}{\le}}\frac4{\sqrt{24}}\,L_1R\,\sqrt{\frac{\log^{3/2}(2+L_1R)}{L_1R}}\\ &\mathrel{\underset{L_1R\ge16}{<}}\frac4{\sqrt{24}}\sqrt{\frac38}\,L_1R=\frac{L_1R}2 . \end{aligned}

The sum of the terms is less than L_1R, and Theorem 4.8 gives

We bound each of the three terms in brackets by 1+\mathcal N(\mathcal P_{\rm R}).

• First. Obviously, 1\le1+\mathcal N(\mathcal P_{\rm R}).

• Second. Functions with L_0-Lipschitz gradient belong to the class. The classical bound [nesterov2018lectures, Theorem 2.1.7] together with Proposition 4.3 gives

  \sqrt{\frac{L_0R^2}\varepsilon}\le4\bigl(1+\mathcal N(\mathcal P_{\rm R})\bigr).

• Third. For L_1R\ge16, by Lemma 4.9 it is at most 24(1+\mathcal N(\mathcal P_{\rm R})). For L_1R<16 it is less than 10, since it increases in L_1R and

  \frac{16}{\sqrt{\log18}}\mathrel{\underset{\log18>4\log2}{<}}\frac{16}{\sqrt{4\log2}}\mathrel{\underset{\log2>2/3}{<}}\sqrt{96}<10 .

Adding up, we obtain that the expression in brackets in (4.10) is at most 29(1+\mathcal N(\mathcal P_{\rm R})).

We track the gradient at the points y+t(z-y), t\in[0,1].

• Integral inequality. By Lemma 2.2(a), the map t\mapsto\nabla f(y+t(z-y)) is absolutely continuous, and for almost all t the norm of its derivative is at most (L_0+L_1\|\nabla f(y+t(z-y))\|)\|z-y\|. Therefore, for all t\in[0,1]

  \|\nabla f\bigl(y+t(z-y)\bigr)-\nabla f(y)\|\le\|z-y\|\int_0^t\Bigl(L_0+L_1\|\nabla f\bigl(y+\tau(z-y)\bigr)\|\Bigr)d\tau .

  By the triangle inequality this implies

  \begin{aligned} \MoveEqLeft L_0+L_1\|\nabla f\bigl(y+t(z-y)\bigr)\|\\ &\le L_0+L_1\|\nabla f(y)\|+L_1\|z-y\|\int_0^t\Bigl(L_0+L_1\|\nabla f\bigl(y+\tau(z-y)\bigr)\|\Bigr)d\tau . \end{aligned}

• Grönwall’s inequality. The left-hand side of the last inequality is continuous in t, and by Grönwall’s inequality in integral form

  L_0+L_1\|\nabla f\bigl(y+t(z-y)\bigr)\|\le\bigl(L_0+L_1\|\nabla f(y)\|\bigr)e^{L_1\|z-y\|t} .

  For t=1 this is statement (a).

• Increment of the gradient. Substitute this bound into (Proof):

  \begin{aligned} \|\nabla f\bigl(y+t(z-y)\bigr)-\nabla f(y)\| &\le\bigl(L_0+L_1\|\nabla f(y)\|\bigr)\|z-y\|\int_0^te^{L_1\|z-y\|\tau}d\tau\\ &=\bigl(L_0+L_1\|\nabla f(y)\|\bigr)\frac{e^{L_1\|z-y\|t}-1}{L_1} . \end{aligned}

  For t=1 this is statement (b).

• Increment of the function. The left inequality in (c) is the convexity of f. For the right one

  \begin{aligned} f(z)-f(y)-\langle \nabla f(y),z-y\rangle &=\int_0^1\langle \nabla f\bigl(y+t(z-y)\bigr)-\nabla f(y),z-y\rangle\,dt\\ &\mathrel{\underset{(\text{b})}{\le}}\bigl(L_0+L_1\|\nabla f(y)\|\bigr)\|z-y\|\int_0^1\frac{e^{L_1\|z-y\|t}-1}{L_1}\,dt\\ &=\bigl(L_0+L_1\|\nabla f(y)\|\bigr)\frac{e^{L_1\|z-y\|}-1-L_1\|z-y\|}{L_1^2} . \end{aligned}

  The second step uses the Cauchy–Schwarz inequality and statement (b).

Both bounds are statements (b) and (c) of Lemma A.1 with y=x^\ast and z=x, since \nabla f(x^\ast)=0. The right-hand side of the second bound increases in \|x-x^\ast\|, which gives (A.2).

• Gradient. The strong convexity of r gives the strong convexity of H on K with a constant modulus. Therefore, the maximizer is unique and, as in the proof of Lemma 4.2, \nabla\psi(y)=z(y) and the gradient is continuous. This is statement (e).

• Optimality conditions. Set \theta_0=0 and \theta_i=y_i+\sigma_i for i\ge1. The maximization problem in (?) is equivalent to maximizing

  \sum_{i=0}^n\theta_ip_i-\sum_{i=0}^nr(p_i)

  over the simplex. Moving weight between two positive coordinates and from a positive coordinate to a zero one gives a number \nu such that for 0\le i\le n

  \theta_i\ \begin{cases}=\nu+r'(p_i), & p_i>0,\\ \le\nu, & p_i=0.\end{cases}

  These conditions are also sufficient by the convexity of r. Since 0\le r'(p_i)\le b, we always have \nu\ge\theta_i-b.

• Statement (a). The point z=0 is feasible, so \psi\ge0. Equality holds exactly when the vector p=e_0 is optimal. By (Proof) this is equivalent to \nu=-b and \theta_i\le-b for i\ge1. The formulas for the set of minimizers and the minimizer closest to zero follow from (?).

• Statement (b). For y=0, conditions (Proof) hold for p=e_1 and \nu=\sigma_1-b=(2n-2)b: all other \theta_i, including \theta_0=0, are at most \nu. Therefore, \psi(0)=G\sigma_1=(4-2/n)\varepsilon.

• Statement (c). Substituting z=Ge_n, we obtain \psi(y)\ge G\sigma_n=2\varepsilon.

• Statement (d). If y_i=0 for i\ge j, then for l>j

  \theta_l=\sigma_l\le\sigma_j-b=\theta_j-b\le\nu .

  A positive p_l would contradict (Proof), since r'(p_l)>0. Hence, the l-th coordinate of the gradient Gp_l is zero.

Let z,z'\in K be distinct and u=z'-z. On the interval 0<t<1 the coordinates p_i of the point z+tu are either positive or identically zero. Therefore,

where U_n=u_1+\dots+u_n. The terms with identically zero coordinates are omitted. If p_0\equiv0, then U_n=0 and the last term is absent as well. The second step uses the inequality Gp_i=(z+tu)_i\le\|z+tu\|, and the third one the convexity of the norm. Integration and continuity at the endpoints give the assumption of Lemma 4.2, which completes the proof.

• Worst-case functions. If condition (?) holds, take the quadratic function r, otherwise the logarithmic one (?). In both cases \psi\in\mathcal F_{L_0,L_1}^n by Lemma B.2, and Lemma B.1 gives the remaining assumptions of Proposition 4.3(b) with \mathrm{gap}=2\varepsilon and \Delta=4\varepsilon. By this proposition, for every method there exists a matrix V with orthonormal columns such that the function f(x)=\psi(V^\top x) lies in \mathcal P_{4\varepsilon} and f(\hat x)\ge2\varepsilon. The remaining properties of f are given by statements (a) and (b) of Lemma B.1.

• Case L_1^2\varepsilon\ge L_0. For each of the conditions (?) and (?), take as n the integer part of the largest admissible right-hand side. If it is positive, the first part of the theorem gives \mathcal N(\mathcal P_{4\varepsilon})\ge n, and if it is zero, this inequality is obvious. Together with the inequality 1+\lfloor x\rfloor\ge x we obtain

  \begin{aligned} 1+\mathcal N(\mathcal P_{4\varepsilon})&\ge1,\\ 1+\mathcal N(\mathcal P_{4\varepsilon})&\ge\frac14\sqrt{\frac{L_0R^2}\varepsilon},\\ 1+\mathcal N(\mathcal P_{4\varepsilon})&\ge\Bigl(\frac{L_1R}{8\log(2+L_1^2\varepsilon/L_0)}\Bigr)^{2/3}\\ &=\frac14\Bigl(\frac{L_1R}{\log(2+L_1^2\varepsilon/L_0)}\Bigr)^{2/3} . \end{aligned}

  Adding these inequalities with the factors 1, 4 and 4, we obtain that the expression in brackets in (B.3) is at most 9\bigl(1+\mathcal N(\mathcal P_{4\varepsilon})\bigr).

• Case L_1^2\varepsilon<L_0. Then (L_1R)^2<L_0R^2/\varepsilon, and for L_0R^2\ge\varepsilon

  \begin{aligned} \Bigl(\frac{L_1R}{\log(2+L_1^2\varepsilon/L_0)}\Bigr)^{2/3} &\mathrel{\underset{(\mathrm a)}{\le}}(\log2)^{-2/3}(L_1R)^{2/3} \mathrel{\underset{(\mathrm b)}{<}}\frac32\Bigl(\frac{L_0R^2}\varepsilon\Bigr)^{1/3}\\ &\mathrel{\underset{(\mathrm c)}{\le}}\frac32\sqrt{\frac{L_0R^2}\varepsilon} , \end{aligned}

  where step (a) uses the inequality \log(2+t)\ge\log2, step (b) uses the inequalities (\log2)^{-2/3}<3/2 and (L_1R)^2<L_0R^2/\varepsilon, and step (c) uses the condition L_0R^2\ge\varepsilon. Hence, the expression in brackets in (B.3) is at most

  \begin{aligned} 1+\frac52\sqrt{\frac{L_0R^2}\varepsilon} &\le\bigl(1+\mathcal N(\mathcal P_{4\varepsilon})\bigr)+10\bigl(1+\mathcal N(\mathcal P_{4\varepsilon})\bigr)\\ &=11\bigl(1+\mathcal N(\mathcal P_{4\varepsilon})\bigr). \end{aligned}

  Here we used the first two inequalities from the previous case: they were obtained without any condition on the sign of L_1^2\varepsilon-L_0.

In both cases the constant 1/12 is valid.

The proof repeats the proof of Lemma 3.4. For k=0, inequality (B.4) follows from the conditions on x^0. The induction step differs in three places.

• Replacements. Instead of Lemma 3.2, the convexity inequality \langle \nabla f(y^k),y^k-x^\ast\rangle\ge f(y^k)-f^\ast is used, and Lemma 3.3 is used in the same way. Inequalities (B.4) follow from it and Lemma 3.1.

• Increment. With these replacements, the bounds on the distance and on the optimality gap from the proof of Lemma 3.4 give

  \begin{aligned} \MoveEqLeft\Bigl[A_k\bigl(f(x^k)-f^\ast\bigr)+\frac12\|v^k-x^\ast\|^2\Bigr]-\Bigl[A_{k-1}\bigl(f(x^{k-1})-f^\ast\bigr)+\frac12\|v^{k-1}-x^\ast\|^2\Bigr]\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:coef-gap-rule}$}]{}}{\le}}\Bigl(\frac{a_k}{16}-\frac{5A_k}{24}\Bigr)\frac{\|\nabla f(y^k)\|^2}{L_0+L_1\|\nabla f(y^k)\|}+\frac12a_k^2\|\nabla f(y^k)\|^2\\ &\mathrel{\underset{\eqref{eq:coef-gap-rule}}{=}}\Bigl(\frac{a_k}{16}-\frac{5A_k}{24}+\frac{A_k}{32}\Bigr)\frac{\|\nabla f(y^k)\|^2}{L_0+L_1\|\nabla f(y^k)\|}-\frac12a_k^2\|\nabla f(y^k)\|^2\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:coef-gap-rule}$}]{(\mathrm a)}}{\le}}-\frac12a_k^2\|\nabla f(y^k)\|^2 , \end{aligned}

  where step (a) uses the inequalities a_k\le A_k and 1/16-5/24+1/32<0.

• Ball of radius r. By inequality (B.4) and the assumption of the lemma

  \|v^k-x^\ast\|^2\le R_0^2+2a_0\Delta_0\le r^2 ,

  and the point x^k lies in the ball by Lemma 3.1.

All sums in the proof are over i=k+1,\dots,k+N.

• Optimality gap through the weights. We have

  \begin{aligned} f(x^{k+N})-f^\ast &\mathrel{\underset{\eqref{eq:energy-gap}}{\le}}\frac{R^2}{2A_{k+N}} \le\frac{R^2}{2\bigl(\sqrt{A_{k+N}}-\sqrt{A_k}\bigr)^2}\\ &\mathrel{\underset{\eqref{eq:coef-gap}}{\le}}\frac{64R^2}{\Bigl(\sum_i\bigl(L_0+L_1\|\nabla f(y^i)\|\bigr)^{-1/2}\Bigr)^2} . \end{aligned}

• Hölder’s inequality. With exponents 3/2 and 3

  \begin{aligned} N&=\sum_i\bigl(L_0+L_1\|\nabla f(y^i)\|\bigr)^{-1/3}\bigl(L_0+L_1\|\nabla f(y^i)\|\bigr)^{1/3}\\ &\le\Bigl(\sum_i\bigl(L_0+L_1\|\nabla f(y^i)\|\bigr)^{-1/2}\Bigr)^{2/3}\Bigl(\sum_i\bigl(L_0+L_1\|\nabla f(y^i)\|\bigr)\Bigr)^{1/3} . \end{aligned}

  Substituting into the previous bound, we obtain

  \begin{aligned} f(x^{k+N})-f^\ast &\le\frac{64R^2}{N^3}\sum_i\bigl(L_0+L_1\|\nabla f(y^i)\|\bigr)\\ &=\frac{64L_0R^2}{N^2}+\frac{64L_1R^2}{N^3}\sum_i\|\nabla f(y^i)\| . \end{aligned}

• The sum of gradient norms is small when the optimality gap is small. Summing inequalities (B.4) over the block, we obtain

  \sum_i\frac{\|\nabla f(y^i)\|^2}{L_0+L_1\|\nabla f(y^i)\|}\le8\bigl(f(x^k)-f^\ast\bigr).

  By the Cauchy–Schwarz inequality

  \begin{aligned} \Bigl(\sum_i\|\nabla f(y^i)\|\Bigr)^2 &\le\sum_i\frac{\|\nabla f(y^i)\|^2}{L_0+L_1\|\nabla f(y^i)\|}\cdot\sum_i\bigl(L_0+L_1\|\nabla f(y^i)\|\bigr)\\ &\le8\bigl(f(x^k)-f^\ast\bigr)\Bigl(NL_0+L_1\sum_i\|\nabla f(y^i)\|\Bigr). \end{aligned}

  If the sum \sum_i\|\nabla f(y^i)\| exceeded \sqrt{8NL_0(f(x^k)-f^\ast)}+8L_1(f(x^k)-f^\ast), then its square would exceed this quantity times the sum itself, which is at least the right-hand side of the last inequality. Therefore,

  \sum_i\|\nabla f(y^i)\|\le\sqrt{8NL_0\bigl(f(x^k)-f^\ast\bigr)}+8L_1\bigl(f(x^k)-f^\ast\bigr).

• Substitution. Substitute this bound into (Proof). The second term in (Proof) becomes at most

  \frac{64L_1R^2}{N^3}\sqrt{8NL_0\bigl(f(x^k)-f^\ast\bigr)}+\frac{512\,L_1^2R^2\bigl(f(x^k)-f^\ast\bigr)}{N^3} .

  By the inequality between the arithmetic and geometric means, the first of these terms is at most

  \frac{32L_0R^2}{N^2}+\frac{256\,L_1^2R^2\bigl(f(x^k)-f^\ast\bigr)}{N^3} .

  Therefore,

  f(x^{k+N})-f^\ast\le\frac{96L_0R^2}{N^2}+\frac{768\,L_1^2R^2\bigl(f(x^k)-f^\ast\bigr)}{N^3} .

  Since 768\le3072, we obtain (B.5).

The right-hand side of (B.4) is at most R^2/2+R^2/2=R^2. Since f(x^K)-f^\ast>\varepsilon, this gives

We split the iterations into two groups: with a small gradient, L_1\|\nabla f(y^i)\|\le L_0, and with a large one, L_1\|\nabla f(y^i)\|>L_0.

• Iterations with a small gradient. For them L_0+L_1\|\nabla f(y^i)\|\le2L_0, and by (?) each such iteration increases \sqrt{L_0A_i} by at least 1/16. Hence, their number is at most

  16\sqrt{L_0A_K}\mathrel{\underset{\eqref{eq:count-weight}}{<}}16\sqrt{\frac{L_0R^2}\varepsilon} .

• Iterations with a large gradient. For them L_0+L_1\|\nabla f(y^i)\|<2L_1\|\nabla f(y^i)\|, therefore

  \begin{aligned} a_i\|\nabla f(y^i)\|&\mathrel{\underset{\eqref{eq:coef-gap}}{=}}\frac{A_i\|\nabla f(y^i)\|}{32\bigl(L_0+L_1\|\nabla f(y^i)\|\bigr)a_i}\\ &\mathrel{\underset{\mathmakebox[\widthof{$\scriptstyle\eqref{eq:coef-gap}$}]{}}{>}}\frac{A_i}{64L_1a_i} . \end{aligned}

  This and (Proof) give

  \sum_{L_1\|\nabla f(y^i)\|>L_0}\Bigl(\frac{A_i}{a_i}\Bigr)^2<(64L_1)^2\cdot2R^2=8192\,L_1^2R^2 .

  Since A_{i-1}\ge a_0>0, by (?)

  \begin{aligned} \sum_{L_1\|\nabla f(y^i)\|>L_0}\frac{a_i}{A_i} &\le\log\frac{A_K}{a_0} \mathrel{\underset{\eqref{eq:count-weight},\,\eqref{eq:weight-params}}{<}}\log\frac{2\Delta}\varepsilon . \end{aligned}

  By Hölder’s inequality with exponents 3 and 3/2, the number of iterations with a large gradient equals

  \begin{aligned} \sum_{L_1\|\nabla f(y^i)\|>L_0}\Bigl(\frac{A_i}{a_i}\Bigr)^{2/3}\Bigl(\frac{a_i}{A_i}\Bigr)^{2/3} &\le\Bigl(\sum\Bigl(\frac{A_i}{a_i}\Bigr)^2\Bigr)^{1/3}\Bigl(\sum\frac{a_i}{A_i}\Bigr)^{2/3}\\ &<(8192\,L_1^2R^2)^{1/3}\log^{2/3}\frac{2\Delta}\varepsilon\\ &<21\,(L_1R)^{2/3}\log^{2/3}\frac{2\Delta}\varepsilon .\qedhere \end{aligned}

• Part (a). If the stopping criterion was triggered, the output is correct by Lemma 3.3. Otherwise bound (B.5) with k=0 and f(x^0)-f^\ast\le4\varepsilon gives

  f(x^K)-f^\ast\le\frac{96L_0R^2}{K^2}+\frac{3072\,L_1^2R^2\cdot4\varepsilon}{K^3}\mathrel{\underset{(\mathrm a)}{\le}}\frac\varepsilon2+\frac\varepsilon2 ,

  where step (a) uses the choice of K. The bound on K follows from the inequalities \sqrt{192}<14 and 24576^{1/3}<30.

• Part (b). For \Delta\le\varepsilon we return x^0. Otherwise, by Lemma B.6, after K iterations either the stopping criterion was triggered and the output is correct by Lemma 3.3, or f(x^K)-f^\ast\le\varepsilon.

In both parts the number of calls is at most K(m+2).