In this paper, we consider a multi-armed bandit (MAB) instance and study how to identify the best arm when arm commands are conveyed from a central learner to a distributed agent over a discrete memoryless channel (DMC). Depending on the agent capabilities, we provide communication schemes along with their analysis, which interestingly relate to the zero-error capacity of the underlying DMC.
The following are equivalent (see [Shannon1956]):
Here, M(n)=1 ~\forall ~n, so no fixed-blocklength, no-feedback protocol can convey even a binary command with zero-error.
If there exist \mu\neq \mu' such that W\mu=W\mu' but \arg\max_i \mu_i\neq \arg\max_i \mu'_i, then no algorithm can be \delta-correct for all instances (for any \delta<1).
Assume W is known and invertible. Let \sigma_{\min}(W) denote the smallest singular value of W. Then, relative to the noiseless setting, the single-shot + unmixing baseline incurs an inflation governed by
where \tilde{O}(\cdot) hides universal constants and logarithmic factors and
Let {\hat{\tilde\mu}} be an estimator of {\tilde\mu} and define \hat\mu=W^{-1}{\hat{\tilde\mu}}. By submultiplicativity,
In the noiseless (direct-actuation) setting, standard fixed-confidence BAI algorithms satisfy (?) pulls. Thus, to achieve the same accuracy on \mu as in the noiseless case, it suffices to estimate {\tilde\mu} more accurately by a factor \sigma_{\min}(W). Since statistical estimation error scales as t^{-1/2}, shrinking the target error by a factor \sigma_{\min}(W) increases the required sample size by a factor 1/\sigma_{\min}(W)^2, yielding (1).(Note that \sigma_{\min}(W)\le 1 for any square stochastic matrix, so 1/\sigma_{\min}(W)^2\ge 1 and the baseline never improves over the noiseless case.)
In general, the baseline inflation is governed by the conditioning of the mixing map via 1/\sigma_{\min}(W)^2. For the typewriter mixing matrix W=(1-\varepsilon)I+\varepsilon S, when K is even we have \sigma_{\min}(W)=|1-2\varepsilon|, so the inflation reduces to (1-2\varepsilon)^{-2} and diverges as \varepsilon\to\tfrac12, where identifiability fails.
K=6 (even): \sigma_{\min}(W)=|1-2\varepsilon| so baseline inflation blows up as \varepsilon\to\tfrac12 and is impossible at \varepsilon=1/2.
K=5 (odd): \sigma_{\min}(W)>0 for all \varepsilon \in (0,1) so baseline remains identifiable (no singularity), but still suffers conditioning-based inflation.
The codebook \{\varphi(i)\}_{i=0}^4\subseteq \{0,\dots,4\}^2 is zero-error for the typewriter channel (with C_5). Equivalently, \{\varphi(i)\} is an independent set in C_5^{\boxtimes 2}, so \alpha(C_5^{\boxtimes 2})\ge 5.
Assume there exists a fixed-length zero-error update code of blocklength n_u with M(n_u)=\alpha(G^{\boxtimes n_u})\ge K. Assume the agent executes the decoded update immediately upon decoding, i.e., on the last channel use of the packet.
Let \mathsf A be any \delta-correct clean BAI algorithm and \tau_{\rm clean} its pull count. Then there exists an algorithm \widetilde{\mathsf A} over the actuation link that is \delta-correct and whose total number of physical rounds satisfies
Run \mathsf A as a virtual algorithm. Partition time into consecutive blocks of length n_u. In block i, transmit the n_u-symbol zero-error codeword for the arm requested by \mathsf A at virtual step i. By assumption, the decoded arm is executed on the last slot of the block; use the reward from that last slot as the virtual reward fed back to \mathsf A. All intermediate rewards within each block can be logged but ignored. Zero decoding error ensures the virtual interaction matches the clean bandit interaction exactly. Thus, \delta-correctness is preserved and each virtual pull costs n_u physical rounds. Full proof in App. ?.
If \mathsf A satisfies \tau_{\rm clean}\le N_{\rm clean}(\delta,\mu), then \tau\le n_u\,N_{\rm clean}(\delta,\mu).
For K=5, Lemma 2 gives an independent set of size 5 in C_5^{\boxtimes 2}, so M(2)\ge 5 while M(1)=\alpha(C_5)=2<5. Hence n^\star(5)=2. For K=6, since \alpha(C_6)=3, we have M(1)=3<6 and by Cartesian-product coding, M(2)\ge \alpha(C_6)^2 = 9 \ge 6, hence n^\star(6)=2 for Scheme 1 type of codes. Therefore, a fully flexible zero-error update packet exists with blocklength n_u=2 for both K=5 and K=6. By Lemma 2, the number of physical rounds satisfies \tau\le 2\,N_{\rm clean}(\delta,\mu). These guarantees hold for all \varepsilon\in(0,1), whereas, the single-shot baseline can be non-identifiable at \varepsilon=\nicefrac{1}{2} for even K.
Instead of zero-error codes, one could preshare a classical block code (e.g., Reed–Solomon) with small decoding error. From the learner’s viewpoint this is equivalent to replacing W by an effective DMC \widetilde W on \mathcal K, so the analysis reduces to the single-shot baseline of Section ? with \sigma_{\min}(\widetilde W) instead of \sigma_{\min}(W).
Let G=(V,E) be the confusability graph of W and \mathcal S\subseteq V an independent set. If the encoder restricts to inputs X\in\mathcal S and the decoder knows \mathcal S, then X can be decoded with zero-error from a single channel output Y.
For the typewriter channel on C_6, if X is restricted to an independent set \mathcal S\in\{\mathcal S_{\mathrm{even}},\mathcal S_{\mathrm{odd}}\} known to the decoder, then X can be decoded from a single output Y with zero-error.
Consider Scheme 2 with K=6 (public alternating schedule \mathcal S_{\rm even}, \mathcal S_{\rm odd}). In each slot, the transmitted arm is decoded with zero-error in one channel use (Lemma 4). Let \mathsf A be any \delta-correct clean BAI algorithm, let \tau_{\rm clean} be its (random) number of pulls under no channel noise, and let a_1,\ldots,a_{\tau_{\rm clean}} denote the random sequence of arm requests produced by \mathsf A on the clean instance, and define the parity class sequence
Then there exists an actuation wrapper that simulates \mathsf A over the noisy link with the same error probability and a stopping time \tau satisfying
The wrapper runs \mathsf A virtually and only counts rewards when the requested arm belongs to the active parity class; otherwise it waits one slot and then transmits the arm. Zero-error decoding ensures the counted interaction matches the clean bandit path. If the i-th request has the same parity as the (i{-}1)-th, the wrapper must wait one extra slot; otherwise the next slot already has the correct parity. Summing these waiting slots yields (3). Full proof in App. ?.
Equation (3) shows that the slowdown under Scheme 2 depends on the local pattern of parity requests, not merely on the total number of pulls from each parity class. In particular,
When the requested parity alternates often, the slowdown is close to 1; when the clean algorithm makes long runs within the same parity class, the slowdown approaches 2.
Assume 1-subGaussian rewards, then PSE is \delta-correct. Moreover, with probability at least 1-\delta, its total number of physical rounds satisfies
where R is the number of executed phases and N_{\rm SE}(\delta,\mu) is the (instance-dependent) pull complexity of the clean phased successive-elimination algorithm (standard analyses), i.e.,
Define the standard uniform concentration event \mathcal E \triangleq \bigcap_{a\in\mathcal K}\bigcap_{t\ge 1} \left\{|\widehat\mu_a(t)-\mu_a|\le \beta(t)\right\}. By a union bound over arms and times, \Pr(\mathcal E)\ge 1-\delta. On \mathcal E, the elimination rule of phased successive elimination is sound: a^\star is never removed, and any suboptimal arm a is eliminated once 4\beta(t)\le \Delta_a at the end of some phase (since \mathrm{UCB}_a\le \mu_a+2\beta(t) and \mathrm{LCB}_{b_r}\ge \mathrm{LCB}_{a^\star}\ge \mu_{a^\star}-2\beta(t)). Crucially, PSE’s counted rewards are generated by exactly the same arm sequence and sample sizes as the clean phased-SE algorithm (the additional n_r rounds during plan installation are simply ignored for the statistical test). Therefore, on \mathcal E, the number of counted pulls until termination is at most N_{\rm SE}(\delta,\mu). Finally, each executed phase incurs n_r additional physical rounds for plan installation (switching latency), yielding \tau \le N_{\rm SE}(\delta,\mu)+\sum_{r=1}^R n_r on \mathcal E. See full proof in App. ?.
For \beta(t)=\sqrt{2\log(8Kt^2/\delta)/t}, the condition 4\beta(t)\le \Delta is implied by t \ge \frac{c}{\Delta^2}\log\!\Big(\frac{c'K}{\delta\Delta^2}\Big) for universal constants c,c'>0. Consequently, R^\star = 1+\lceil \log_2 T_{\max}\rceil = O(\log(1/\Delta_{\min})).
Consider the one-sided typewriter channel with \varepsilon\in(0,1), so the confusability graph is C_K. Assume the simple plan family \mathcal P_r=\{\,\text{plan}(S,m_r):\emptyset\neq S\subseteq\mathcal K\,\}, so |\mathcal P_r|\le 2^K-1 for all r. (i) K=5. Using the base-5 digit on C_5 (two uses per digit), n_r = 2\Big\lceil \log_5 |\mathcal P_r| \Big\rceil \le 2\Big\lceil \log_5(2^5) \Big\rceil = 6. (ii) K=6. Using the calendar interface on C_6 (one ternary digit per use), n_r = \Big\lceil \log_3 |\mathcal P_r| \Big\rceil \le \Big\lceil \log_3(2^6) \Big\rceil = 4. In both cases, Theorem 1 gives \tau \le N_{\rm SE}(\delta,\mu) + O(R) with small constants: at most 6R on C_5 and 4R on C_6.
Thus when the statistical term N_{\rm SE}(\delta,\mu) dominates (small gaps / small \delta), PSE is dramatically better than per-pull zero-error updates, which incur a multiplicative constant \approx n^\star(\mathcal{K}).
9 IEEEtran references
Define the event
If rewards are 1-subGaussian, then \Pr(\mathcal E)\ge 1-\delta.
Fix an arm a and a time t\ge 1. Since rewards are 1-subGaussian, the empirical mean satisfies
Plugging in \varepsilon=\beta(t) yields
By a union bound over all a\in\mathcal K and all t\ge 1,
hence \Pr(\mathcal E)\ge 1-\delta.
Assume every phase-r plan packet in PSE is decoded with zero error, so the agent executes the intended phase plan \text{\rm plan}(S_r,m_r) exactly. Let (A^{\rm cnt}_s,R^{\rm cnt}_s)_{s\ge 1} denote the sequence of counted arm pulls and counted rewards produced by PSE, indexed by counted time s=1,2,\dots, i.e., ignoring all installation rounds. Let (A^{\rm clean}_s,R^{\rm clean}_s)_{s\ge 1} denote the pull, or equivalently reward, sequence of the clean phased-SE algorithm run with the same \beta(\cdot), the same phase budgets m_r, and the same elimination rule.
Then there exists a coupling under which
In particular, PSE and clean phased-SE compute identical empirical means from counted samples, hence produce the same active sets (S_r) and stop after the same number of counted pulls.
Construct a probability space as follows. For each arm a\in\mathcal K, generate two independent i.i.d. sequences:
each distributed as the arm-a reward law (mean \mu_a, 1-subGaussian). Interpret R^{(c)} as the rewards that will be used for counted pulls of arm a, and R^{(u)} as rewards used for uncounted pulls, i.e., installation rounds. This is valid because, in the true system, every time arm a is executed the reward is an independent draw from the same law; splitting draws into two independent pools preserves the joint law of any finite collection of executed rewards.
Now run the clean phased-SE algorithm and define that its j-th pull of arm a receives reward R^{(c)}_{a,j}. Run PSE on the same probability space, with the following reward assignment:
During execution (counted) rounds, whenever PSE executes arm a for the j-th counted time, it receives reward R^{(c)}_{a,j}.
During installation (uncounted) rounds, PSE executes the hold arm h (the last arm pulled in the previous phase) and receives rewards from the R^{(u)} pool (which are never used in estimates).
Because plan packets decode with zero error, in each phase r, PSE executes exactly the intended plan \text{plan}(S_r,m_r) in its execution segment. Therefore, the counted arm sequence of PSE in phase r is exactly the same as the arm sequence of the clean phased-SE algorithm in phase r (both pull each a\in S_r exactly m_r times). Under the construction above, they also receive exactly the same counted rewards, namely the corresponding R^{(c)} samples. Hence, after phase r, both algorithms compute identical empirical means \widehat\mu_a(t_r) for each a\in S_r, apply the same elimination rule, and therefore produce the same S_{r+1}.
By induction over phases, the entire counted pull/reward sequence agrees almost surely under this coupling.
On the event \mathcal E from Lemma 5, PSE never eliminates the best arm a^\star. Moreover, fix any suboptimal a\neq a^\star. For any phase index r such that
arm a is eliminated at the end of phase r (i.e., a\notin S_{r+1}).
Fix a phase r and let t_r=2^r-1 be the counted sample size per surviving arm at the end of phase r.
(i) The best arm is not eliminated. Let b_r\in\arg\max_{b\in S_r}\mathrm{LCB}_b where
On \mathcal E, we have \widehat\mu_{a^\star}(t_r)\ge \mu_{a^\star}-\beta(t_r), hence
Also on \mathcal E, for any b\in S_r we have \widehat\mu_b(t_r)\le \mu_b+\beta(t_r), so
Therefore \mathrm{UCB}_{a^\star}\ge \mu_{a^\star}\ge \mathrm{LCB}_{b_r}, so a^\star satisfies the retention condition \mathrm{UCB}_{a^\star}\ge \mathrm{LCB}_{b_r} and is not eliminated.
(ii) A suboptimal arm is eliminated once 4\beta(t_r)<\Delta_a. Fix a\neq a^\star. On \mathcal E we have
Also, since b_r maximizes \mathrm{LCB} over S_r and a^\star\in S_r by part (i),
Combining the two displays,
If 4\beta(t_r)<\Delta_a, then -\Delta_a+2\beta(t_r)<-2\beta(t_r), hence
Therefore a fails the retention test \mathrm{UCB}_a\ge \mathrm{LCB}_{b_r} and is eliminated, i.e., a\notin S_{r+1}.
Assume plan packets are decoded with zero error (as in the theorem statement). Let R be the number of executed phases, i.e., the number of iterations of the while-loop in Algorithm ?. Define the number of counted pulls made by PSE as
In each executed phase r, PSE spends exactly n_r physical rounds to transmit the plan packet (during which it pulls the hold arm and discards these rewards for estimation), and then spends exactly |S_r|m_r physical rounds executing the plan (counting those rewards). Therefore the total number of physical rounds is exactly
Next, by Lemma 6, the counted pull/reward sequence of PSE is (under a coupling) identical to that of the clean phased-SE algorithm with the same (m_r) and elimination rule. In particular, PSE and clean phased-SE stop after the same number of counted pulls and output the same arm as a function of the counted data. Let \tau_{\rm SE} denote the (random) stopping time (number of pulls) of this clean phased-SE algorithm. Then
hence, in distribution as well.
By Lemma 5, \Pr(\mathcal E)\ge 1-\delta. On \mathcal E, Lemma 7 implies that a^\star is never eliminated and every suboptimal arm is eventually eliminated, so both algorithms terminate and output a^\star on \mathcal E. Therefore PSE is \delta-correct.
Finally, standard analyses of phased successive elimination imply that with probability at least 1-\delta,
where N_{\rm SE}(\delta,\mu) is the usual instance-dependent clean sample complexity bound (one such bound is stated in Theorem 1). Combining with (Proof) yields, with probability at least 1-\delta,
which concludes our proof.
The text on this page is from arXiv:2604.02255v1 by Merve Karakas, Osama Hanna, Lin F. Yang and Christina Fragouli, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.
The record of each item is in analysis/objects/2604/2604.02255.jsonl.