When is a better LLM worth paying for? Stochastic Optimization with Switching Mixed-Fidelity Oracles

The claim follows by induction. At stage s, the selected oracle \lambda_s is admissible, so Definition 1 guarantees \mathbb{E}\|y^s-x^*\|^2\le R_{s-1}^2/2=R_s^2.

To prove the claim, we need to control the state-dependent variance, and also bound the error introduced by the persistent oracle bias. We begin with the SGE recursion stated in Proposition 4, whose proof follows from a similar analysis of SGE as in [ilandarideva2025state] (see Appendix ?). With this recursion, we can handle the state-dependent variance and systematic bias transparently. Our parameter choices ensure that the state-dependent variance terms can be absorbed into the converging function-value terms in this recursion. Taking the telescoping sum over the recursions yields

where A_k and P_k grow quadratically in k, and \Gamma_k contains the initial-error and residual-variance terms. To bound the accumulated bias, observe that the same inequality controls the distance of each z_t from x^\ast. Thus, the distances appearing on the right-hand side can themselves be bounded in terms of the accumulated error. Applying Young’s inequality provides a bound of the form 2\Gamma_k+O(k^2B^2/\mu), and dividing both sides of the inequality by A_k yields the stated function-value bound. By further using the strong convexity property, we can obtain the distance-contraction guarantee. The full proof is deferred to Appendix ?.

The comparison with the best fixed oracle follows directly from minimizing the cost at each stage: the sum of the stagewise minima is no larger than the minimum of the sums. It remains to show how large this improvement can be.

We first construct two oracles. The first has a smaller coarse-scale cost but a larger residual-variance term, whereas the second has a constant stage cost of M throughout the optimization horizon. The parameters are chosen so that the first oracle is cheaper until the final m=\lceil\log_2 K\rceil stages. Thus, the adaptive policy incurs a total cost of \Theta(M\log K), while the best fixed oracle incurs a cost of MK.

To show that this order cannot be improved, let d_s denote the minimum cost at stage s, and consider an oracle attaining d_{K-m}. Its cost over all K stages is at most O(Kd_{K-m}). On the other hand, because d_s is nondecreasing, the adaptive policy pays at least (m+1)d_{K-m} over the final m+1 stages. Combining these bounds gives an improvement of at most O(K/\log K). The construction with tight examples, and the complete bounds are presented in Appendix ?.

Let u_s:=y^s, thus \|u_s-x^*\|^2\le R_s^2. For k=s,\ldots,t-2 define

By Definition 1 and the contraction-admissibility conditions in (?), a simple induction implies

Therefore, the input u_{t-1} to the terminal projected-gradient reduction satisfies the mean-square warm-start condition at radius R_{t-1}. Definition 2 and the terminal PG-admissibility condition then imply (2). By applying Markov’s inequality, we have

which proves (2).

By Lemma 2, every independent block trial succeeds with probability at least 3/4. Hence the probability that the first r trials all fail is at most 4^{-r}. Therefore

The expectation for the total number of trials for a successful certification is bounded by

which completes the proof.

Consider a block-admissible oracle. By Lemma 2, one independent block trial fails to certify the full target with probability at most 1/4. Therefore all \mathcal R independent trials fail with probability at most 4^{-\mathcal R}\le\tfrac{\delta}{K+\ell}. Every fully successful block advances the certified level by at least one, so there can be at most K such blocks. Every unsuccessful block permanently removes one oracle, so there can be at most \ell unsuccessful blocks. A union bound over at most K+\ell block decisions shows that, with probability at least 1-\delta, no block-admissible oracle is incorrectly removed.

Every advancement of the certified level, whether a full-block success or the partial progress, is justified by the exact projected-gradient certificate in Lemma 1. Thus the certified state always satisfies \|y^s-x^*\|^2\le R_s^2. Under the stated availability condition, an unfinished level cannot exhaust all block-admissible oracles on the above high-probability event. Thus the certified level eventually reaches K, where R_K^2\le\epsilon.

Finally, each certification round queries the trusted certification module at most \mathcal{R} times. Multiplying \mathcal{R} by K + \ell yields the stated trusted-gradient complexity.

From (?), the stage cost of SGE has the form C_k(\lambda)=\max\{\alpha_\lambda,\beta_\lambda2^k\}. For any two oracles \lambda and \nu, their strict ordering can change at most once as k increases.

Thus one unordered stochastic oracle pair can trigger at most one strict economic switch. Since removed oracles never return to the pool, we have Q\le\binom{\ell}{2}.

First, note that the fully successful blocks cover disjoint certified level intervals. By (?), every contraction stage in such blocks costs no more than the corresponding stage of our benchmark, while the cost of the PG stage is at most \zeta times its SGE-equivalent stage cost by (?). Therefore, the single-trial stochastic-training cost of all fully successful blocks is at most \zeta C_{\rm sw}^{\rm PG}.(Here, we do not carefully distinguish the cost of the contraction stage and PG stage, as \zeta is only a logarithmic factor of the condition number.) Considering the sequential implementation rule with at most \mathcal{R} trials, we have

For a failed block, by the budget constraint rule in (?), the uncertified training cost of all contraction stages is at most B, as shown in Ineq. (?). Meanwhile, in case B is very large or even \infty, we can always bound the block’s contraction cost by C_{\rm sw}^{\rm PG} due to (?). Therefore, the total cost of the contraction stages in all failed blocks is bounded by

It remains to account for the terminal PG costs of failed blocks, i.e.,

where \lambda_b and k_b are the oracle and terminal stage of failed block b. On the high-probability event under consideration, a removed oracle cannot be PG-admissible at its terminal stage: for the SGE/RR+SGE implementation, terminal PG-admissibility makes the whole block block-admissible, and Lemma 2 together with Proposition 2 rules out such a false rejection on this event. Thus k_b\ge\kappa_{\lambda_b}. Moreover, because \lambda_b is selected at k_b while the current pool contains an oracle attaining C_{k_b}^\star, we have C_{k_b}(\lambda_b)\le C_{k_b}^\star by (?). Therefore the terminal PG cost of each failed oracle is bounded by its corresponding summand in (?). Since each oracle can fail at most once, we have

Finally, we need to compute the total certification cost. To do so, we need to count the successful certification blocks. If a fully successful block is terminated by the budget condition and its terminal stage is t-1, then

The corresponding successful block intervals are disjoint, and their total (single-trial) contraction subroutine cost is at most C_{\rm sw}^{\rm PG} by (?). Therefore, there are at most \lceil C_{\rm sw}^{\rm PG}/B\rceil budget-triggered successful blocks. In addition there are Q economic-switch blocks and at most one final block, while the total number of successful blocks is at most K. Thus, the total number of successful blocks is bounded by \min\left\{ K,\, \left\lceil\tfrac{C_{\rm sw}^{\rm PG}}{B}\right\rceil+Q+1 \right\}, and the total number of certification blocks is at most \min\left\{ K,\, \left\lceil\tfrac{C_{\rm sw}^{\rm PG}}{B}\right\rceil+Q+1 \right\} +M, indicating a total certification cost of

Combining Ineqs. (Proof), (?), (?), and (?) proves Ineq. (3).

For B\le C_{\rm sw}^{\rm PG}, we have \bar B=B and \lceil C_{\rm sw}^{\rm PG}/B\rceil \le C_{\rm sw}^{\rm PG}/B+1. Also recall the upper bound M\leq \ell and Q\leq \binom{\ell}{2} in Lemma 5. Applying these inequalities in (3) yields (2). By minimizing the two B-dependent terms, we can show B^* and \mathrm{Cost}_{B^*}.