Two Sample Sizes in Test-Time LLM Ensembling: Buffered ERM and Minimax Theory

For t\ge0, define the clean population and empirical threshold risks

In particular, \widetilde R_0 is the ordinary empirical risk that would be available if the clean answer distributions were known. It contains the randomness from sampling Q questions, but not the finite-N generation noise present in \widehat R_{\varepsilon_N}. Write \mathfrak v_Q=\mathfrak v_Q(\delta) for brevity. The result follows from a chain of six inequalities. By Lemma A.2, with probability at least 1-\delta/4 over the sampled questions, |\widetilde R_t(w)-R_t(w)|\le\mathfrak v_Q simultaneously for all w\inDelta_K and t\in\{0,2\varepsilon_N\}. In particular,

Separately, Lemma A.1 shows, conditionally on the sampled questions, that event (?) holds with probability at least 1-\delta/4 over the generations. On this event, the buffered sandwich (?) holds for every w\inDelta_K and j\in[Q]. Summing over j and dividing by Q gives

The middle inequality (c) follows from the minimality of \widehat w_{\varepsilon_N} for the buffered empirical risk. Finally, Assumption 2.1 gives

By a union bound, the two preceding events hold simultaneously with probability at least 1-\delta. Combining (a)–(f) in alphabetic order and using R=R_0 proves (2.2).

We first bound the error of the deployment vote on questions with margin greater than r. The enlarged training buffer will then control the population mass of the remaining questions.

Fix w and q with M_q(w)>r. For each wrong answer a\ne a^\star(q), define the vote differences

Conditional on q, these variables are independent, take values in [-1,1], and have mean p_w(a^\star(q)\mid q)-p_w(a\mid q)\ge M_q(w)>r. Hoeffding’s inequality gives

The gold answer fails to be the unique empirical plurality only if this event occurs for some wrong answer. A union bound over at most A-1 wrong answers therefore gives the conditional error bound (A-1)e^{-Tr^2/2}. On questions with M_q(w)\le r, we bound the conditional error probability by one. Averaging over questions gives

This bound holds for every w. Conditional on the training data, the learned weight is fixed, and the fresh deployment question and generations have the law used above.

The rest of the argument follows the proof of Theorem 2.2, and we provide a brief sketch below. Let \widetilde R_s be the clean empirical threshold risk defined there. Apply Lemma A.2 at the fixed thresholds r and 2\varepsilon_N+r. With probability at least 1-\delta/4 over the questions, |\widetilde R_s(w)-R_s(w)|\le\mathfrak v_Q(\delta) simultaneously for all w and both thresholds. Separately, Lemma A.1 bounds the uniform training-margin error by \varepsilon_N with probability at least 1-\delta/4, conditional on the questions. On this latter event, the buffered sandwich (?), with thresholds shifted by r, holds for every j\in[Q] and w\inDelta_K:

Averaging over the training questions gives

The two events hold jointly with probability at least 1-\delta. The same comparison argument now applies at these shifted thresholds: the sandwich and empirical optimality compare the clean empirical risks at \widehat w_{\varepsilon_N+r} and w^\star, while uniform convergence adds \mathfrak v_Q(\delta) at each end. Consequently,

Since 2\varepsilon_N+r\le t_0, the one-sided margin condition gives

Combining this inequality with (?) proves (2.4).

For (?), take r_T=\sqrt{2/T}\log(eAT). The last term in (2.4) becomes (A-1)\exp\{-\log^2(eAT)\}, which, for fixed A, decreases faster than any fixed power of T^{-1}. Also,

Here one can take C_beta=3^beta for every beta>0, since 2\varepsilon_N+r_T\le3\max\{\varepsilon_N,r_T\}. Substitution of (?) and (?) now gives (?).

The stated risks for the three decisions t>1/2 (the + decision), t<1/2 (the - decision), and t=1/2 follow from (?). The decision t=1/2 is weakly dominated. Moreover,

where the mass at zero cancels. Thus, when theta(G)>0, the + decision is optimal and choosing - incurs excess risk theta(G)=|theta(G)|. The case theta(G)<0 is similar, while the two decisions have equal risk when theta(G)=0.

Equality of K_nG and K_nH is equivalent to equality of the integrals of b_{k,n}((1+z)/2) for k=0,\dots,n. The Bernstein polynomials form a basis for the polynomials of degree at most n in p, and composition with the invertible affine map p=(1+z)/2 gives a basis for the polynomials of degree at most n in z. Monomials z^j, j=0,\dots,n form another basis.

By Lemma B.1, the set

is compact and convex, and both coordinates are continuous affine functions of G. Its indirect modulus is exactly (?). Applying Theorem B.2 with \mathcal X=\mathcal X_n, followed by Lemma B.3, gives

for a universal constant C. This proves the rightmost inequality in (4.1); the pointwise reduction (?) gives the middle inequality.

It remains to prove the lower bound for decision regret. Fix \varepsilon=(2\sqrt Q)^{-1} and take any G,H\in\mathcal G_{beta,L} satisfying h(K_nG,K_nH)\le\varepsilon. After interchanging G and H if necessary, write

The original targets may have the same sign, in which case the same side is optimal under both distributions and the pair need not be difficult for the decision problem. Reflection symmetrization produces a pair with opposite optimal decisions. Specifically, define

By Lemma B.1, G_+,G_-\in\mathcal G_{beta,L} and

Writing P_+=K_nG_+ and P_-=K_nG_-, joint convexity of squared Hellinger distance and invariance under reversal of the count labels give

For probability measures P and R, write

for their Hellinger affinity. Affinity factorizes over product measures, so

The last inequality is Bernoulli’s inequality. Under the present Hellinger convention, Cauchy–Schwarz also gives

Combining (?)–(?) with \varepsilon=(2\sqrt Q)^{-1} yields

Now fix any, possibly randomized, decision rule \widehat\sigma. Because a wrong decision costs \delta/2 under either G_+ or G_- (recall (?)),

The penultimate inequality is Le Cam’s two-point inequality. Since (?) holds for every admissible pair G,H and every decision rule, taking first the infimum over \widehat\sigma and then the supremum over all pairs in the definition of the modulus gives

Finally, Lemma B.3 gives

which proves the lower bound in (4.1).

Suppose K_nG=K_nH. By Lemma 3.2, G and H integrate every polynomial of degree at most n equally. Fix s=beta+1 and let P_n be the corresponding polynomial from Lemma E.1. Then

by Lemma E.2. Taking the supremum over all pairs with K_nG=K_nH proves the upper bound on omega_{n,beta,L}(0).

Let H_{n,+},H_{n,-} be the adjusted degree-n alternatives supplied by Lemma E.4. Their targets are opposite with

By Lemma 3.2, K_nH_{n,+}=K_nH_{n,-}, so this pair proves the lower bound on omega_{n,beta,L}(0).

Take the two distributions G_{\sigma,\eta} displayed after the proposition. Conditional on Z=+1, the count is deterministically S=n, while conditional on Z=-1, it is S=0. Hence K_nG_{+1,\eta} assigns masses (1-\eta)/2 and (1+\eta)/2 to 0 and n, respectively, while K_nG_{-1,\eta} interchanges these masses. Therefore

For 0<\varepsilon\le1, take \eta=\varepsilon/2. The two channel laws are then within Hellinger distance \varepsilon, whereas

Both measures belong to \mathcal G_{beta,L} for every beta>0 and L\ge1, proving (4.3). The minimax conclusion (4.3) follows from the lower bound in Theorem 4.1.

The upper inequality is (?). We prove the lower inequality.

We first record the Bayes reduction that guides the construction. Suppose \nu_+ and \nu_- are finitely supported priors on \mathfrak Z_{beta,L} such that, for some b>0, theta_Z=\pm b for every Z\in\operatorname{supp}(\nu_\pm), respectively. Let \overline{\mathbb P}_+ and \overline{\mathbb P}_- denote the corresponding prior-averaged laws of the full observed sample. Then every decision rule \widehat\sigma satisfies

The first inequality bounds the maximum risk below by its average under the prior that first chooses the sign uniformly and then draws Z from the corresponding \nu_\pm; the second is the usual two-point testing inequality. It therefore suffices to construct two such priors with b comparable to the modulus and whose prior-averaged data laws have total variation bounded away from one.

Fix \varepsilon_Q:=(64\sqrt Q)^{-1} and take any 0<Delta<omega_{n,beta,L}(\varepsilon_Q). By the reflection symmetrization used in the proof of Theorem 4.1, there exist G_+,G_-\in\mathcal G_{beta,L} such that

where \dagger denotes reflection under z\mapsto-z.

Apply Lemma D.1 to G_+ and reflect the result to obtain equal-weight measures G^{(M)}_+ and G^{(M)}_-. Write

Because the targets and channel laws converge, and because M may be taken arbitrarily large, choose M so that

Reflection gives the analogous channel bound for the negative hypothesis and theta(G^{(M)}_-)=-theta(G^{(M)}_+). The triangle inequality gives

Consequently, the product-Hellinger and total-variation inequalities (?) and (?) give

Each discretized alternative is now lifted to a family of deterministic question-to-advantage maps. For each sign, let \nu_\pm be the uniform permutation prior from Lemma D.2, constructed using G^{(M)}_\pm. Every map in the support of \nu_\pm belongs to \mathfrak Z_{beta,L} and has target \pm a_M. Denote the corresponding prior-averaged full-data laws by \overline{\mathbb P}_+ and \overline{\mathbb P}_-. The permutation lemma gives

Since adjoining the common factor \lambda^{\otimes Q} does not change total variation, the triangle inequality and (?) yield

Applying (?) with b=a_M and then taking the infimum over \widehat\sigma gives

Since this holds for every 0<Delta<omega_{n,beta,L}(\varepsilon_Q), this proves (4.4) with the stated universal constants.

In this submodel the clean margin of w is

Put p_+:=\mathbb P\{Z>0\} and p_-:=\mathbb P\{Z<0\}. The margin condition implies that Z has no mass at zero, so p_++p_-=1 and theta_Z=p_+-p_-. Accordingly, the risk of w is p_- when w_1>w_2, p_+ when w_1<w_2, and one when w_1=w_2. Hence \inf_{w\inDelta_K}R_Z(w)=(1-|theta_Z|)/2, attained at e_1 when theta_Z>0 and at e_2 when theta_Z<0. At either choice,

Thus the embedded environments satisfy the one-sided margin condition at w^\star used in the general theorem.

Given any estimator \widehat w, define the induced scalar decision \widehat\sigma to be +1 when \widehat w_1\ge\widehat w_2 and -1 otherwise. If \widehat w_1\ne\widehat w_2, its clean excess risk equals |theta_Z| when \widehat\sigma chooses the wrong side and is zero otherwise. If \widehat w_1=\widehat w_2, the ensemble margin is zero on every question, so R_Z(\widehat w)=1 and its excess risk is (1+|theta_Z|)/2\ge|theta_Z|. Therefore, pointwise in the data,

For models 1 and 2, S_j is sufficient for their raw generations; the generations of models i\ge3 have a law independent of Z. Thus the induced \widehat\sigma is statistically equivalent to a possibly randomized rule based on ((q_j,S_j))_{j=1}^Q, and such randomization is already allowed in (?). Taking expectations, suprema, and infima proves the first inequality; the second follows from Corollary 4.5.

Let (Z,S,\widehat\xi) denote a generic copy of (Z_j,S_j,\widehat\xi_j), where \widehat\xi=\mathbf 1\{S>n/2\}-\mathbf 1\{S<n/2\}. Then \mathbb E_G\widehattheta_{\mathrm{plug}}=\mathbb E_G\widehat\xi, and the bias–fluctuation decomposition gives

Because the \widehat\xi_j are iid and bounded by one, the fluctuation term satisfies

It remains to control the bias. The margin condition implies G(\{0\})=0, so the value of \operatorname{sgn} at zero is immaterial. For Z\ne0, the pointwise inequality |\widehat\xi-\operatorname{sgn}(Z)|\le2\mathbf 1\{\widehat\xi\ne\operatorname{sgn}(Z)\} gives

For z\ne0, conditional Hoeffding bounds the probability of an incorrect sign or a tie by

Consequently,

To integrate this bound, put a=n^{-1/2} and split the range of |Z| into

The margin condition, together with the trivial probability bound once 2^{j+1}a>1, gives

On A_j, one has e^{-nZ^2/2}\le e^{-2^{2j-1}}. Therefore

because the series converges. Combining this estimate with (?) and the fluctuation bound proves (5.1). Finally, (?) gives the same upper bound for the regret of \widehat\sigma_{\mathrm{plug}}.

The functions b_{k,n}((1+z)/2), k=0,\dots,n, form a basis of \Pi_n: the Bernstein polynomials form a basis in p, and p=(1+z)/2 is an invertible affine change of variable.

The conditional identity (?) gives \mathbb E_G\widehattheta_P = \int P(z)\,\,\mathrm d G(z), and hence

Moreover, a_{S_j}(P) is supported on an interval of length A(P), so \operatorname{Var}_G(a_{S_j}(P))\le A(P)^2/4. Independence across questions and Jensen’s inequality give

Combining the bias and fluctuation bounds proves (5.4).

Expand P(z)=c_0+\sum_{j=1}^dc_jT_j(z) in Chebyshev polynomials. The cosine coefficient formula and \norm P_infinity\le1 give |c_0|\le1 and |c_j|\le2 for j\ge1. Let L_j be the sum of the absolute values of the monomial coefficients of T_j. The recurrence T_{j+1}(z)=2zT_j(z)-T_{j-1}(z) gives L_{j+1}\le2L_j+L_{j-1}; since L_0=L_1=1, it follows that L_j\le C(1+\sqrt2)^j. After substituting z=2p-1, each monomial z^\ell has absolute coefficient sum 3^\ell, so the corresponding sum for T_j(2p-1) is at most C\{3(1+\sqrt2)\}^j.

Write P(2p-1)=\sum_{j=0}^d r_jp^j, so the r_j are its coefficients in the ordinary power basis. The preceding bounds and a geometric sum give \sum_{j=0}^d|r_j|\le C_0B_0^d, with B_0=7.5>3(1+\sqrt2).

For any n\ge d, the identity p^j=\sum_{k=j}^n\frac{\binom kj}{\binom nj}b_{k,n}(p) gives the degree-n Bernstein coefficients directly:

Every ratio in this sum lies in [0,1], so |a_k(P)|\le\sum_{j=0}^d|r_j|\le C_0B_0^d.

The operator D_n=T_nT_n^* is self-adjoint and positive. It also preserves each polynomial space \Pi_j. Indeed, beta integration gives

which is a degree-j polynomial in k. Averaging this polynomial under k\sim\operatorname{Bin}(n,p) shows that D_nu^j\in\Pi_j and that its leading coefficient is

Here we use the fact that the expectation of a monic degree-j polynomial in a \operatorname{Bin}(n,p) variable has leading coefficient n!/(n-j)! as a polynomial in p.

For j=0, the claim follows directly from D_n1=1. For j\ge1, because L_0,\dots,L_j are obtained by orthogonalizing 1,u,\dots,u^j,

Now D_nL_j\in\Pi_j by degree preservation. For every q\in\Pi_{j-1}, self-adjointness and degree preservation give

because D_nq\in\Pi_{j-1} and L_j\perp\Pi_{j-1}. Hence D_nL_j lies in the one-dimensional space \Pi_j\cap\Pi_{j-1}^\perp=\operatorname{span}\{L_j\}. Comparing leading coefficients proves (5.6).

Define v_j=\lambda_{j,n}^{-1/2}T_n^*L_j. Then

and T_nv_j=\sqrt{\lambda_{j,n}}L_j. There are n+1 orthonormal vectors v_j in the (n+1)-dimensional space \mathsf H_n, so they form a basis. Thus (v_j,L_j,\sqrt{\lambda_{j,n}}) is a singular system and (5.6) follows.

If P\in\Pi_d, write \widetilde P=\sum_{j=0}^dc_jL_j. Since T_na(P)=\widetilde P, comparison of the Legendre coefficients in (5.6) gives

Consequently,

Parseval’s identity in \mathsf H_n now gives

which proves (5.7). Since j\mapsto\lambda_{j,n} is decreasing, Parseval in L_2[0,1] also gives

Finally,

Applying the same bound to the maximum and minimum proves (5.7).

Let h_d, G_{d,\pm}, and H_{d,\pm} be, respectively, the degree-d dual certificate, raw alternatives, and margin-adjusted alternatives supplied by Lemma E.4. By Lemma E.4, Lemma E.3,

We now bound the Hellinger distance between their count laws. Write p=(1+z)/2. Under this transformation, \mu_beta has density

with respect to Lebesgue measure. Put

and define

Then

If r_0:=K_nH_0 and m_0:=\rho_Lq_0+(1-\rho_L)r_0, the final pair has probability mass functions

Because m_0\ge\rho_Lq_0 coordinatewise,

For 0<beta\le1, w_beta(p)\gebeta for almost every p\in(0,1), and \int_0^1b_{k,n}(p)\,\,\mathrm d p=\frac1{n+1}. Thus

Combining (?) and (?) gives

The cancellation in (E.4) removes the first d singular modes. Indeed, L_j((1+z)/2) is a polynomial of degree j in z, so

By Lemma G.1 and the monotonicity of j\mapsto\lambda_{j,n},

Since \|g_d\|_1\le1 and the orthonormal shifted Legendre polynomial satisfies \|L_j\|_infinity=\sqrt{2j+1},

Therefore

Equations (?) and (?) imply

Together with (Proof), this proves (6.1).

If beta>1/2, then g_d\in L_2[0,1] and \|g_d\|_2^2 \le \int_0^1w_beta(p)^2\,\,\mathrm d p = \frac{beta^2}{2beta-1}. Bessel’s inequality replaces (?) by

Substitution into (?) proves (6.1).

For j\le n/2, the lower half of (?) gives

Let

where A_{beta,L} is large enough to absorb the constant in (6.1), and choose

Because y_{Q,n}\ge1 and A_{beta,L} is fixed, \Lambda_{Q,n}\asymp_{beta,L}y_{Q,n}. If the constant c_0 in Proposition 6.2(a) is sufficiently small and n is sufficiently large, then d+1\le n/2 and

Consequently, the Hellinger radius in (6.1) is at most Q^{-1/2}. Monotonicity of the modulus, Theorem 4.1, and d\asymp\sqrt{n\Lambda_{Q,n}} yield

The comparison \Lambda_{Q,n}\asymp_{beta,L}y_{Q,n} proves (?). When beta>1/2, apply (6.1) instead and set \Lambda_{Q}:=\log(A_{beta,L}\sqrt Q). Then \Lambda_Q\asymp_{beta,L}y_Q^{(2)}, and the same argument proves the refined assertion.

For d\le n/2, the upper half of (?) gives \log\kappa_{d,n}\le C_\kappa d(d+1)/n. Choose

with a>0 small enough that C_\kappa d(d+1)/n\le x_{Q,n}/4. Reducing c_0 if necessary ensures d\le n/2, while x_{Q,n}\ge1 gives d\asymp\sqrt{nx_{Q,n}}. The spectral upper bound (?) therefore yields a bias term of order (nx_{Q,n})^{-beta/2} and a stochastic term bounded by

Since e^{-x/4}\le C_beta x^{-beta/2} for x\ge1, the stochastic term has the same stated order, proving (?).

Under the lower inequality in (6.3),

The upper inequality in (6.3), with c_\eta sufficiently small, places both logarithmic scales in the ranges required by Proposition 6.2. Combining its lower and upper assertions proves (6.3).

For each (j,i,a), Hoeffding’s inequality gives

A union bound over at most KAQ triples makes the probability of any violation at most \delta/4. On the complementary event,

for every w\inDelta_K. Taking minima over wrong answers preserves this bound and gives (A.1); the indicator sandwich follows.

By (?), the supremum in (A.2) is \sup_{f\in\mathcal F_{\mathrm{buf}}}|P_Qf-Pf|. Apply Lemma A.5 to \mathcal F_{\mathrm{buf}}, take u=\mathfrak v_Q(\delta) and use (A.4) with m=2Q. Taking the universal constant C_0 in (?) sufficiently large makes the resulting probability at most \delta/4.

Constant affine functions may be discarded because they do not increase the number of patterns. The hyperplanes \{g_h=0\} partition \mathbb R^p into relatively open faces of all dimensions. The weak sign vector is constant on each face, so it suffices to bound the total number of faces.

Let F(H,p) be the largest possible number of such faces. We prove the bound by induction on H. After arranging the first H-1 hyperplanes, add the Hth. The preceding hyperplanes induce an arrangement of at most H-1 hyperplanes on the new hyperplane, and hence at most F(H-1,p-1) faces there. Each induced face can split one old face into two while itself becoming a new face, producing a net increase of at most two. Therefore

with F(0,p)=F(H,0)=1. The quantity B(H,p) obeys the same recurrence by Pascal’s identity, so induction proves F(H,p)\le B(H,p). Finally, when H\ge p,

If H<p, then \binom Hr=0 for r>H, and the binomial theorem gives B(H,p)=\sum_{r=0}^H2^r\binom Hr=3^H. The stated logarithmic bound follows from these two cases.

Fix q_1,\dots,q_m and t. For every wrong answer a\ne a^\star(q_j), consider the affine function

on the (K-1)-dimensional affine hull of Delta_K. Since

the label vector (f_{w,t}(q_1),\dots,f_{w,t}(q_m)) is determined by the weak sign pattern of at most m(A-1) affine functions. Lemma A.3 proves (A.4).

The restrictions realized by a union of two classes are contained in the union of their two sets of restrictions. Hence \Pi_{\mathcal F_{\mathrm{buf}}}(m)\le \Pi_{\mathcal F_{s_1}}(m)+\Pi_{\mathcal F_{s_2}}(m), which gives the first bound in (A.4). The second follows from Lemma A.3, with the additive \log 2 accounted for by the constant term. This also covers p=0, when B(H,0)=1.

Write Z:=\sup_{h\in\mathcal H}|P_Qh-Ph|. Introduce an independent ghost sample with empirical measure P'_Q = \frac1Q \sum_{j=1}^Q \delta_{q'_j}, and write Z':=\sup_{h\in\mathcal H}|P_Qh-P'_Qh|. Conditional on the original sample, P_Qh-Ph=\mathbb E[P_Qh-P'_Qh\mid q_1,\dots,q_Q] for every h. Convexity of the supremum norm and of x\mapsto e^{\lambda x} therefore gives \mathbb E e^{\lambda Z}\le\mathbb E e^{\lambda Z'} for every \lambda>0.

By exchangeability of each original–ghost pair, Z' has the same distribution after independent random swaps. Encode the swaps by Rademacher signs \sigma_1,\dots,\sigma_Q. Conditional on the combined sample, a fixed restriction of h gives coefficients d_j=h(q_j)-h(q'_j)\in\{-1,0,+1\}, and the randomized difference is Q^{-1}\sum_{j=1}^Q\sigma_jd_j. There are at most \Pi_{\mathcal H}(2Q) such restrictions. Hoeffding’s exponential-moment bound, together with e^{\lambda|x|}\le e^{\lambda x}+e^{-\lambda x}, thus gives

Markov’s inequality with \lambda=Qu now proves (A.5).

Convexity and reflection invariance are immediate from the definition of the class. Since [-1,1] is compact, the set of all probability measures on it is weakly compact. It remains to show that the margin class is weakly closed. Suppose G_m\Rightarrow G. Fix 0<t<1 and choose \eta>0 such that t+\eta<1. By the Portmanteau theorem applied to the open interval (-t-\eta,t+\eta),

Letting \eta\downarrow0 proves the margin inequality for t<1; at t=1 it is automatic because G([-1,1])=1\le L. Thus \mathcal G_{beta,L} is weakly closed and hence compact. Taking t\downarrow0 shows that G(\{0\})=0.

For continuity of theta, let \psi_\delta be the continuous clipped-sign function that equals -1 on [-1,-\delta], equals z/\delta on [-\delta,\delta], and equals 1 on [\delta,1]. Uniformly over the margin class,

The integral of \psi_\delta is weakly continuous. First letting m\toinfinity and then \delta\downarrow0 shows that theta(G_m)\totheta(G).

Each coordinate of K_nG is the integral of a bounded continuous polynomial in z, so the channel map is weakly continuous; both the channel and target maps are affine by linearity of integration. Finally, reflection changes the sign of the target, and the Bernstein identity

gives the channel identity in (B.1).

Let

Because all categorical laws are within Hellinger distance \sqrt2, we have D=omega_{\mathcal X}(\sqrt2). Set

Our goal is to show an estimator that has expected absolute error at most 7\delta/2, uniformly over \mathcal X. If D=0, the target is constant, and the corresponding constant estimator achieves zero error. If \varepsilon_0\ge\sqrt2, then \delta=D, and the midpoint of [\tau_-,\tau_+] has error at most D/2 = \delta/2. We may therefore assume \varepsilon_0<\sqrt2. The scaling lemma gives

In particular, \delta=0 would imply D=0. If \delta\ge D/6, the midpoint again has error at most D/2\le3\delta. It remains to consider

which, together with (Proof), implies Q>3A^2.

We now construct a sample-split interval-search estimator. It proceeds through stages s=0,\dots,S-1 and maintains a nested sequence of intervals I_s intended to localize \tau. Start from I_0=[\tau_-,\tau_+]. The observations are divided into S disjoint blocks, one per stage, with block sizes specified below. At stage s, write I_s=[a_s,b_s] and \ell_s=b_s-a_s. Using only block s, test the lower third of I_s against the upper third through the composite hypotheses

If the test chooses H_{s}^-, retain the lower two thirds, I_{s+1}=[a_s,a_s+2\ell_s/3]; if it chooses H_{s}^+, retain the upper two thirds, I_{s+1}=[a_s+\ell_s/3,b_s]. The two candidate intervals overlap in the middle third of I_s. Thus, when \tau lies in that middle third, either test outcome retains it.

The interval lengths are deterministic and satisfy

Stop at the first integer S for which \ell_S\le6\delta, and return the midpoint \widehat\tau of I_S. It remains to choose the stagewise tests and block sizes. For s<S, define

Here g_s is the separation between the two outer thirds, while u_s measures the current interval length in units of the target accuracy \delta. Since \ell_s>6\delta, we have r_s\ge\varepsilon_0, while (Proof) gives r_s^2\le1/3<2. Thus \varepsilon_0\le r_s<\sqrt2, and Lemma B.3, applied at radii \varepsilon_0 and r_s, gives

For every possible interval I_s, the observation laws p generated under H_s^- and H_s^+ form two compact convex sets. Indeed, each is the image, under the coordinate projection (p,\tau)\mapsto p, of the intersection of \mathcal X with the corresponding closed halfspace. By (?), every pair across these sets is separated by more than r_s in Hellinger distance. If either set is empty, use the corresponding deterministic decision. Otherwise, apply the test supplied by Lemma B.4 using a fresh block of

observations.

The use of a fresh block makes the testing guarantee valid conditional on the preceding outcomes. Let \mathcal F_s be the sigma-field generated by blocks 0,\dots,s-1 and any auxiliary randomization used through stage s-1, so that I_s is \mathcal F_s-measurable. Call the update at stage s unsafe if I_{s+1} does not contain \tau, and let A_s be the event that all preceding updates are safe. Since I_0 contains \tau and the intervals are nested, A_s=\{\tau\in I_s\}. On A_s, either \tau lies in the middle third, in which case an unsafe update is impossible, or the true law p belongs to one of the two tested composite hypotheses. Since block s is independent of \mathcal F_s and has law p^{\otimes k_s} conditional on that history, the robust test bound gives

We now bound the risk of this estimator. Let E_s:=A_s\cap\{\tau\notin I_{s+1}\} be the event that the first unsafe update occurs at stage s. If no update is unsafe, then \tau\in I_S and |\widehat\tau-\tau|\le\ell_S/2\le3\delta. On E_s, all preceding updates were safe, so \tau\in I_s immediately before the unsafe update. Every later interval, and hence \widehat\tau, remains inside I_s; therefore the loss on E_s is at most \ell_s. Moreover, (?) gives

The events E_s are disjoint, and consequently

Finally, \ell_{S-1}>6\delta and the interval lengths form a geometric sequence, so

Thus the expected loss is at most 7\delta/2.

It remains to verify that the planned independent blocks fit within the original sample. From (?),

Minimality of S gives 6<u_{S-1}\le9. Writing j=S-1-s, we have u_s=u_{S-1}(3/2)^j, and hence

Also,

where the second inequality uses \log_{3/2}x\le3x for x\ge1. Combining this with ?, ? gives

using A=32 and Q>3A^2. Any unused observations are ignored. The estimator is therefore feasible with Q observations, which completes the proof of (B.2).

The first inequality follows because the set of pairs admissible at radius \varepsilon_1 is contained in the set admissible at radius \varepsilon_2. For the second, take any x_i=(p_i,\tau_i)\in\mathcal X, i=0,1, satisfying h(p_0,p_1)\le\varepsilon_2. Let \overline x=(x_0+x_1)/2, and set

Convexity keeps x_0^\lambda,x_1^\lambda in \mathcal X. Their target separation is \lambda|\tau_1-\tau_0|, while affinity of the observation coordinate and joint convexity of squared Hellinger distance give

Therefore \lambda|\tau_1-\tau_0|\leomega_{\mathcal X}(\varepsilon_1). Dividing by \lambda and taking the supremum over pairs admissible at radius \varepsilon_2 proves (B.3).

First suppose that every distribution in \mathcal P_0\cup\mathcal P_1 has strictly positive coordinates. Let (p_0,q_1) maximize the Hellinger affinity

over \mathcal P_0\times\mathcal P_1. For fixed q_1, the map p\mapsto\rho(p,q_1) is concave. First-order optimality at p_0 along every segment from p_0 to p\in\mathcal P_0 gives

Similarly, for every q\in\mathcal P_1,

Use the likelihood-ratio test between p_0^{\otimes k} and q_1^{\otimes k}, namely, \phi = \mathbf 1 \left\{\prod_{j=1}^k q_1(Y_j)/p_0(Y_j)\ge1 \right\}. For p\in\mathcal P_0, Markov’s inequality and (Proof) yield

The inverse likelihood ratio and (Proof) give the same bound for the other error. Since maximizing affinity is equivalent to minimizing Hellinger distance,

which proves (B.4) under strict positivity.

To remove that assumption, let u be the uniform law and, for \eta\in(0,1), set

Apply the preceding construction to obtain a randomized test \phi_\eta, viewed as a vector in [0,1]^{m^k}. Along a sequence \eta_\ell\downarrow0, compactness gives coordinatewise convergence to a randomized test \phi. Uniform continuity of Hellinger affinity shows that the maximal affinities of the smoothed sets converge to the maximal affinity of the original sets. For each fixed p\in\mathcal P_0 and q\in\mathcal P_1, the corresponding smoothed product laws converge in total variation to p^{\otimes k} and q^{\otimes k}. Together with coordinatewise convergence of the tests on the finite sample space, this implies convergence of their testing risks. Passing to the limit proves the result on the boundary of the simplex.

Consider the fixed family

For 0<t<m there is no mass in [-t,t], while for m\le t<1 the mass is m^beta\le t^beta. Hence the family lies in \mathcal G_{beta,L}, independently of Q,n, and theta(G_{\sigma,m})=\sigma m^beta.

Let P_{\sigma,m}=K_nG_{\sigma,m}. Augmenting the observation by revealing whether the latent draw came from the central component and then applying data processing gives

where the last inequality uses

for m\le1/2. Since squared Hellinger distance is bounded by Kullback–Leibler divergence, taking

for \varepsilon\le c_0(beta) ensures both m\le1/2 and h(P_{+1,m},P_{-1,m})\le\varepsilon. Meanwhile,

This proves (C.1). Its minimax consequence follows from Theorem 4.1, monotonicity of the modulus, and a fixed reduction of the constant multiplying Q^{-1/2} when necessary.

Because G\in\mathcal G_{beta,L}, G(\{0\})=0. Write

and define the one-sided small-ball functions

The construction treats the two sides of zero separately. Starting at zero, divide as much of the mass on each side as possible into blocks of mass 1/M, and replace each block by an atom at its endpoint farther from zero. Any atom needed to round the number of blocks is placed at +1 or -1. Moving mass outward in this way is what preserves the margin bound.

To make this precise, choose an integer m_+ nearest to Mp_+ and put m_-=M-m_+. These are the numbers of atoms assigned to the positive and negative sides. They satisfy

Let

Here k_\pm count the complete mass-1/M blocks and e_\pm account for the possible rounding atom. Since Mp_++Mp_-=M, we have e_+,e_-\in\{0,1\} and e_++e_-\in\{0,1\}. For j=1,\dots,k_+ and j=1,\dots,k_-, respectively, define the outward quantiles

Thus x_j^+ and -x_j^- are the endpoints farther from zero of the jth cumulative blocks on the two sides. Set

For 0<t<1, the generalized-inverse property gives x_j^+\le t if and only if j/M\le F_+(t). Consequently, \#\{j:x_j^+\le t\}=\lfloor MF_+(t)\rfloor; the same argument applies on the negative side. Because the possible extra atoms at \pm1 are not counted, we obtain

In particular, the discretization places no more mass near zero than the original distribution:

At t=1, the required inequality is 1\le L. Hence G^{(M)}\in\mathcal G_{beta,L} with exactly the same margin constant.

The positive mass of G^{(M)} is m_+/M, so (?) gives

Equation (?) shows directly that, for every 0<t<1,

At t=1, the corresponding errors are at most 1/(2M) by (?). Thus the one-sided distribution functions of G^{(M)} converge to those of G, which implies G^{(M)}\Rightarrow G. Each coordinate of K_nG^{(M)} is the integral of a bounded continuous polynomial in z, so weak convergence implies that K_nG^{(M)} converges coordinatewise to K_nG. Since the output alphabet is finite, this implies

Finally, if H is the reflection of G, define H^{(M)} to be the reflection of G^{(M)}.

If Q>M, the right side of (D.2) is at least one and the claim is trivial. Assume henceforth that Q\le M.

We compare two joint laws on the common augmented variables (q,R,J,S)=((q_j,R_j,J_j,S_j))_{j=1}^Q. Under the permutation law, first draw R_1,\dots,R_Q iid uniformly from [M] and, conditional on R, draw the q_j independently and uniformly within I_{R_j}. Independently draw a uniform permutation \Pi of [M], set J_j=\Pi(R_j), and then draw the counts conditionally independently according to

This construction generates q_1,\dots,q_Q\overset{\mathrm{iid}}{\sim} \lambda, and its (q,S)-marginal is \overline{\mathbb P}^{Q,n}_{G_M,M}.

Under the iid law, generate (R,q) in the same way, but draw J_1,\dots,J_Q iid uniformly from [M], independently of (R,q), and then use the same conditional distribution of S given J. Its (q,S)-marginal is \lambda^{\otimes Q}\otimes(K_nG_M)^{\otimes Q}. Since projection onto (q,S) cannot increase total variation, it is enough to compare these two augmented laws.

Let

The uniform law on \mathcal D_{Q,M} is exactly the law obtained by drawing Q labels uniformly without replacement from [M], with the jth coordinate recording the label drawn for observation j.

Let A:=\{R\in\mathcal D_{Q,M}\} under the permutation law. Conditional on A, the vector R is uniform on \mathcal D_{Q,M}. For every fixed r\in\mathcal D_{Q,M}, the vector \Pi(r):=(\Pi(r_1),\dots,\Pi(r_Q)) is also uniform on \mathcal D_{Q,M}, and its distribution does not depend on r. Consequently, under the permutation law conditional on A, the vectors R and J are independent and each is uniform on \mathcal D_{Q,M}.

Under the iid law, let E:=\{R\in\mathcal D_{Q,M},\ J\in\mathcal D_{Q,M}\}. Since R and J are independent and each has iid uniform coordinates before conditioning, conditional on E they remain independent and each is uniform on \mathcal D_{Q,M}. Given (R,J), both laws draw q_j uniformly within I_{R_j} and generate S_j through the same binomial channel indexed by J_j. Hence their conditional laws agree exactly:

The probability of a collision in one iid sample of Q indices is

Using the elementary inequality \operatorname{TV}(P,P(\cdot\mid B))\le P(B^c) and the triangle inequality,

Projecting both augmented laws onto (q,S) now proves (D.2).

The key idea is to smooth the discontinuous sign function with a positive, normalized, finite-frequency Jackson kernel. The resulting convolution is a bounded trigonometric polynomial that closely follows the sign function away from its jumps; evenness then converts it into an algebraic polynomial in x=\cos\vartheta.

Let f(\vartheta):=\operatorname{sgn}(\cos\vartheta) on the circle \mathbb R/(2\pi\mathbb Z), and choose an integer r such that 2r-1>s. For m\ge2, define

where the quotient is interpreted continuously at zero. The constant c_{m,r} is chosen to give normalization \frac1{2\pi} \int_{-\pi}^{\pi} J_{m,r}(u)\,\,\mathrm d u=1.

Kernel properties. Recall that the Fejér kernel satisfies

Since J_{m,r}=c_{m,r}m^rF_m^r, it is nonnegative, even, and a trigonometric polynomial of degree d:=r(m-1).

To determine its normalization scale, put K_{m,r}^0(u):=(\sin(mu/2)/\sin(u/2))^{2r}. On |u|\le1/m, one has K_{m,r}^0(u)\asymp_r m^{2r}, while for 0<|u|\le\pi, the inequalities |\sin(u/2)|\ge|u|/\pi and |\sin(mu/2)|\le1 give K_{m,r}^0(u)\lesssim_r|u|^{-2r}. Splitting the integral at 1/m therefore yields

Only the resulting upper bound on c_{m,r} is needed below. We next bound the tail. If a\le1/m, normalized total mass bounds it by one; if a>1/m, the preceding pointwise and normalization bounds give

Thus, for every 0<a\le\pi,

Finite-frequency smoothing. Define

Since J_{m,r}\ge0 and has normalized mass one, T_m(\vartheta) is a weighted average of values in [-1,1], so |T_m(\vartheta)|\le1. Write J_{m,r}(u)=\sum_{|k|\le d}j_ke^{iku}. The convolution identity gives

where \widehat f(k) is the kth Fourier coefficient of f. Thus T_m is a trigonometric polynomial of degree at most d. Because both f and J_{m,r} are even, so is T_m, and hence it has a cosine expansion T_m(\vartheta)=a_0+\sum_{k=1}^da_k\cos(k\vartheta). If \mathsf T_k denotes the kth Chebyshev polynomial, define P(x):=a_0+\sum_{k=1}^da_k\mathsf T_k(x). Then P has degree at most d and

Localized approximation error. Let a(\vartheta) denote the circular distance from \vartheta to the nearest discontinuity of f, namely \pi/2 or 3\pi/2. Whenever |u|<a(\vartheta), the values f(\vartheta-u) and f(\vartheta) agree. Therefore (?) implies

If x=\cos\vartheta, then |x|=\sin a(\vartheta)\le a(\vartheta). For n\ge2r, take m=1+\lfloor n/r\rfloor. Then d=r(m-1)\le n and m\ge n/r. Since 2r-1>s, the last display implies (E.1), after enlarging the constant. For the finitely many integers 1\le n<2r, take P_n\equiv0 and enlarge C_s.

For |z|\le n^{-1}, the integrand is bounded by two and the margin condition gives probability at most Ln^{-beta}. On the dyadic shell

the pointwise error is at most C_s2^{-js} and the shell probability is at most L(2^{j+1}/n)^beta until the upper endpoint reaches one. Consequently,

because s>beta, proving (E.2).

The upper bound follows from Lemma E.1, Lemma E.2, because \mu_beta([-t,t])=t^beta.

For the lower bound, we use only the inverse theorem of [Kopotun, Theorem 9.1]. The power weight w_beta(x):=|x|^{beta-1} is the generalized-Jacobi weight with one designated point z_1=0 and exponent beta-1>-1, and belongs to the class treated there. Fix an integer r>beta, which will be the order of the modulus, and a sufficiently large constant A. Specializing Kopotun’s theorem to p=1, and absorbing the normalization of \mu_beta into the constant, gives

Here \norm{g}_{1,w_beta}:=\int_{-1}^1|g(x)|w_beta(x)\,\,\mathrm d x, E_k(f)_{1,w_beta}:=\inf_{P\in\Pi_k}\norm{f-P}_{1,w_beta}, and omega_\varphi^r(f,A,m^{-1})_{1,w_beta} is the complete weighted modulus of smoothness defined in [Kopotun, Section 3]. The normalization of \mu_beta gives E_k(\operatorname{sgn})_{1,w_beta}=(2/beta)E_k(beta). Kopotun uses \Pi_k for polynomials of degree less than k, whereas our \Pi_k contains polynomials of degree at most k; this one-degree shift only affects the constants.

For f=\operatorname{sgn}, the definition of the complete modulus simplifies substantially. Its main-part finite-difference term is evaluated away from a neighborhood of zero and vanishes because \operatorname{sgn} is constant on each remaining component. Only the local best-approximation term remains:

for all sufficiently small t. The zero polynomial gives an upper bound of order t^beta. For the reverse bound, the interval [-b_t,b_t] contains [-2At,2At], and the substitution x=tu gives

The last infimum is strictly positive: \Pi_{r-1} is a finite-dimensional closed subspace of the weighted L_1 space, whereas \operatorname{sgn} is not equal almost everywhere to a polynomial. We have therefore shown that

Together, (?) and (?) give a lower bound of order m^{-beta} on a weighted average of E_1(beta),\ldots,E_m(beta) for all sufficiently large m. It remains to extract a pointwise lower bound for E_n(beta).

Write E_k:=E_k(beta). By the upper bound already established, E_k\lesssim k^{-beta}. Apply (?) with m=Mn, where M\ge2 is a fixed integer to be chosen. By (?),

Split the sum at n. Using E_k\lesssim k^{-beta} on the first part and monotonicity E_k\le E_n for k\ge n on the second gives

Consequently,

Since r>beta, choose the fixed M large enough that C_3M^{-r}\le(c/2)M^{-beta}. The first term on the right can then be absorbed into the left, giving E_n\gtrsim n^{-beta}. Decreasing the implicit constant handles the finitely many small values of n not covered by (?).

The polynomial-approximation error admits a dual certificate h_m that annihilates every polynomial in \Pi_m while being maximally correlated with \operatorname{sgn}. Perturbing the reference measure in the directions \pm h_m will then produce moment-matched distributions with oppositely separated targets.

Let f:=\operatorname{sgn} and use the dual pairing

Since E_m(beta) is the L_1 distance from f to the subspace \Pi_m, it has the following dual representation; see Remark E.5.

The maximum is attained by weak-^* compactness. We may choose a maximizer that is odd. Indeed, if h is feasible, then

is feasible, has L_infinity norm at most one, and has the same objective value because \mu_beta is symmetric, \Pi_m is reflection invariant, and \operatorname{sgn} is odd.

Let h_m be such an odd maximizer and define G_{m,\pm} by (E.4). These measures are nonnegative because \norm{h_m}_infinity\le1. Orthogonality to the constant polynomial makes their total masses one; orthogonality to x^j, j=0,\dots,m, makes their moments equal. Moreover,

Oddness gives G_{m,-}=G_{m,+}^\dagger. Finally, symmetry of \mu_beta gives \int\operatorname{sgn}\,\,\mathrm d\mu_beta=0, so (?) gives

and reflection gives the target of G_{m,-}.

It remains to verify the claims for the adjusted pair. For every 0<t<1, the outer atoms do not contribute and

at t=1, the required inequality is 1\le L. Thus H_{m,\pm}\in\mathcal G_{beta,L}. Adding the same symmetric measure to the two raw alternatives preserves their moment matching, and symmetry of H_0 together with reflection of the raw pair gives H_{m,-}=H_{m,+}^\dagger. Finally, theta(H_0)=0, so

Define the coefficient vector T_n^*g algebraically by

The vectors v_0,\dots,v_n in ? form an orthonormal basis of (\mathbb R^{n+1},\nu_n) and satisfy T_nv_j=\sqrt{\lambda_{j,n}}L_j. Although g need not belong to L_2, every integral below is finite because g\in L_1 and L_j is bounded. Hence

Parseval in the finite-dimensional coefficient space gives (G.1), because