A One-Sided McDiarmid-Type Inequality for Heavy Tails via Reverse KL Tensorization

For u\ge0 and 1<p\le2,

as shown in [behnamnia2025logsumexponential, Lemma B.8]. Indeed, the two sides are equal at u=0 and

For 0\le u\le1, we use u^{p-1}\ge u and e^{-u}\ge1-u. For u\ge1, we use u^{p-1}\ge1.

Fix \beta>0. Since f(\bm{X})=\sup_kf(\bm{X})-D_k(\bm{X})(\bm{X}) and \sup_kf(\bm{X}) does not depend on X_k,

For the last inequality (?), we apply (Proof) with u=\beta D_k(\bm{X})(\bm{X})\ge0 a.s. and take expectations

The left-hand side is positive, so the right-hand side is also positive. Since \ln y\le y-1 for y>0,

Adding \beta\mathbb{E}^{(k)}[D_k(\bm{X})(\bm{X})] to both sides proves (?). By Proposition A.1, there is a constant b with \beta (f-b)\le0 a.s. Applying the reverse KL tensorization (Proposition 3.4) to h=\beta (f-b),

The last inequality is by (?). Combined with Chernoff’s inequality,

If V=0, f is almost surely constant, so the tail probability is zero. For V>0, taking \beta=(t/V)^{1/(p-1)} proves (3.1). The equivalent form (?) follows by solving for t.

Let \bm{Y}=(Y_1,\ldots,Y_n) be an independent copy of \bm{X}. Define random vectors H^{(k)}, for k=0,\dots,~n, taking values in \mathcal X^n as

so that H^{(0)}=\bm{Y} and H^{(n)}=\bm{X}. Then,

Fix k and write the elements of H^{(k)} except for the k-th element as

Conditionally on \bm{U}_k, the variables X_k and Y_k are independent and have the same distribution. Define a random variable

With this notation, we can write D_k(\bm{U}_k,X_k) as D_k(\bm{U}_k,X_k)=s_k(\bm{U}_k)-f(\bm{U}_k,X_k).
Since both f(\bm{U}_k,X_k) and f(\bm{U}_k,Y_k) are at most s_k(\bm{U}_k) almost surely, by triangle inequality,

Minkowski’s inequality therefore gives

Since \bm{U}_k has the same distribution as \bm{X}_{-k},

By non-negativity, for every k,

Therefore, we obtain

Hence, by Minkowski’s inequality,

We now show that (?) implies f(\bm{X})\in L_p(P_{\bm{X}}). Since f(\bm{Y}) is finite almost surely, there exists M<\infty such that \rho\coloneqq\mathbb P\bigl(|f(\bm{Y})|\leq M\bigr)>0. On the event \{|f(\bm{Y})|\leq M\},

Using the independence of \bm{X} and \bm{Y}, we obtain

The left-hand side is finite by (?), so

Since |f(\bm{X})|\leq M+\bigl(|f(\bm{X})|-M\bigr)_+, it follows that

Thus f(\bm{X})\in L_p(P_{\bm{X}}), and therefore for every 0<r\leq p,

Now we prove that f(\bm{X}) can have an infinite variance. For 1<p<q, let X have the Pareto distribution with scale parameter one and shape parameter q, so

Let X_1,\ldots,X_n be i.i.d. copies of the Pareto random variable X. We consider f(\bm{X})=-\frac{1}{n}\sum_{i=1}^n X_i. Then, for any fixed x\in\mathcal X^n, we have

Since z\ge 1 almost surely and the above expression is decreasing in z,

Consequently,

Therefore,

and hence

Since p<q, the Pareto random variable X has a finite p-th moment, and therefore

It follows that

Thus Assumption 3.1 is satisfied.

On the other hand,

and therefore

Taking expectations gives

For a Pareto random variable with tail index q, \mathbb E[X^r]<\infty\Longleftrightarrow r<q, and hence \mathbb E[X^q]=\infty. Therefore,

In particular, if 1<p<q\le 2,

Hence, for 1<p<q\le 2, we obtain an example for which V<\infty i.e. Assumption 3.1 is satisfied, while simultaneously f(\mathbf X)\notin L_q(P_{\bm{X}}) and f(\mathbf X)\notin L_2(P_{\bm{X}}).

For S\subseteq[n], we write X_S\coloneqq\{X_i:i\in S\}.

Define the projection \Pi_S of integrable \sigma(X_{[n]})-measurable random variables onto \sigma(X_S)-measurable ones, and a \sigma(X_S)-measurable random variable g_S, by

Using the tower property,

Also, the random variable g_S is nonnegative. Thus, g_S is a probability density with respect to P_{\bm{X}}. Since P_{\bm{X}} is a product measure,

Moreover,

because for any measurable and integrable u, \mathbb E^{(k)}[u] does not depend on x_k. Therefore, the operators \mathbb E^{(1)},\ldots,\mathbb E^{(n)} are commuting projections. Consequently, for S\subseteq[n], the operator \Pi_S=\prod_{j\notin S}\mathbb E^{(j)} is independent of the order of the factors. This leads to

In particular,

The projections also satisfy

because applying both operators integrates all the coordinates in S^c\cup T^c=(S\cap T)^c. If k\notin S, then the commutation of the coordinate projections gives

We now record some identities for KL divergences. Write \rho\coloneqq e^{h(\bm{X})}/Z_h, so P_h=\rho\,P_{\bm{X}}. We have

By definition, D_{\mathrm{KL}}(P_h\Vert P_{\bm{X}})=\mathbb E_{P_h}[\ln\frac{dP_h}{dP_{\bm{X}}}]=\mathbb E[\rho\ln\rho]. Substituting \ln\rho=h(\bm{X})-\ln Z_h,

By definition, D_{\mathrm{KL}}(P_{\bm{X}}\Vert P_h)=\mathbb E_{P_{\bm{X}}}[\ln\frac{dP_{\bm{X}}}{dP_h}]=\mathbb{E}[\ln(1/\rho)]. Again, by \ln\rho=h(\bm{X})-\ln Z_h,

Using these identities,

Our goal is to bound the reverse KL in (?). Our strategy is to first bound \mathbb E \left[ \left(\frac{e^{h(\bm{X})}}{Z_h}-1\right)h(\bm{X}) \right]. Then, we subtract the forward KL term D_{\mathrm{KL}}(P_h\Vert P_{\bm{X}}) from both sides to obtain a bound on the reverse KL D_{\mathrm{KL}}(P_{\bm{X}}\Vert P_h).

The first step is to decompose \mathbb E \left[ \left(\frac{e^{h(\bm{X})}}{Z_h}-1\right)h(\bm{X}) \right]. Fix a permutation \pi of \{1,\dots,n\}. We define T_i\coloneqq\{\pi(1),\dots,\pi(i)\}, with T_0\coloneqq\varnothing. By conditioning successively on the independent random variables and using a telescoping sum,

Denote the set of indices preceding k in \pi by S_k(\pi), so taking k=\pi(i) gives S_{\pi(i)}(\pi)=T_{i-1}. Using this notation,

Now let \pi be uniform on the symmetric group \mathfrak{S}_n\coloneqq \{\text{all permutations of{1,…,n}}\}. Averaging over \pi gives

For fixed k\in[n] and fixed S\subseteq[n]\setminus\{k\},

where the probability is over the random permutation \pi. As a check, for fixed k,

For k\in[n] and S\subseteq[n]\setminus\{k\}, define

To express the summands in (?), we define the random variables

Using this notation, from (?), we obtain

Thus,

The next step is to apply the conditional Donsker–Varadhan inequality to bound \mathbb E[a_kh(\bm{X})]=\mathbb E[(b_k-\ell_k)h(\bm{X})]. Fix x_{-k}. We apply the Donsker–Varadhan variational formula (Lemma C.2 in Appendix) to the k-th coordinate, with reference measure P_{X_k}, test function z\mapsto h(x_{-k},z), and density \rho\coloneqq b_k/\ell_k. To confirm that \rho is admissible, we can verify

since \ell_k is \sigma(\bm{X}_{-k})-measurable and (?) implies \mathbb E^{(k)}[b_k]=\ell_k. Applying the Donsker–Varadhan variational formula (Lemma C.2 in Appendix) with these choices gives, P_{\bm{X}_{-k}}-a.s. ,

Subtracting \mathbb E^{(k)}[h(\bm{X})] from both sides gives, P_{\bm{X}_{-k}}-a.s. ,

Multiplying (?) by \ell_k gives, P_{\bm{X}_{-k}}-a.s. ,

Since \ell_k is \sigma(\bm{X}_{-k})-measurable, it can be moved inside \mathbb E^{(k)}. Hence, P_{\bm{X}_{-k}}-a.s. ,

Similarly, P_{\bm{X}_{-k}}-a.s. ,

Consequently, P_{\bm{X}_{-k}}-a.s. ,

Taking expectation with respect to the remaining variables \bm{X}_{-k}, or equivalently taking the full expectation with respect to P_{\bm{X}}, gives

By summing over k,

We then bound the first term of (?). Using the property (?) and the non-negativity of g_S, we write g_SP_{\bm{X}} for the probability measure on \mathcal X^n given by (g_SP_{\bm{X}})(A)\coloneqq\int_A g_S\,dP_{\bm{X}}, so that \frac{d(g_SP_{\bm{X}})}{dP_{\bm{X}}}=g_S. Since e^{h(\bm{X})}/Z_h>0, each g_S is strictly positive almost surely. Thus, each measure g_SP_{\bm{X}} satisfies P_{\bm{X}}\ll g_SP_{\bm{X}}. This also defines \ell_kP_{\bm{X}} and b_kP_{\bm{X}} accordingly. We write \ell_k P_{\bm{X}} for the probability measure on \mathcal X^n given by (\ell_k P_{\bm{X}})(A)\coloneqq\int_A \ell_k\,dP_{\bm{X}}, so that \frac{d(\ell_k P_{\bm{X}})}{dP_{\bm{X}}}=\ell_k, and similarly for b_k P_{\bm{X}}. Using this notation, each summand in the first term of (?) is

We bound this quantity by the joint convexity of the KL divergence,

For S\subseteq[n]\setminus\{k\}, by applying the Pythagorean identity for conditional projections (Lemma C.1 in Appendix) with \mathcal G=\sigma(X_S) and f=g_{S\cup\{k\}},

where the entropy (\rm Ent) is defined in Lemma C.1. Note that, we can apply Lemma C.1 because, by the tower property,

(This is the same identity \mathbb E^{(k)}[g_{S\cup\{k\}}]=g_S established earlier.) Since h\le0 a.s., e^{h(\bm{X})}/Z_h\le Z_h^{-1} a.s. Conditional expectation preserves the bound, so 0<g_S\le Z_h^{-1} a.s. In addition, the function x\mapsto x|\ln x| is bounded on (0,Z_h^{-1}]. Therefore, \mathbb E[|g_S\ln g_S|]<\infty. If moreover S\subseteq T, then \ln g_S is \sigma(X_S)-measurable and g_S=\mathbb E[g_T\mid X_S], so \mathbb E[g_T|\ln g_S|]=\mathbb E[g_S|\ln g_S|]<\infty. Consequently, for S\subseteq T,

Summing (?) over k and substituting (?),

By (?), we have

Therefore,

We now bound the second term of \eqref{arc_dv}. For T\subseteq[n], define

For S\subseteq[n]\setminus\{k\},

Hence, using (?) and (?),

Using this relationship,

The second equality holds because the quantity \ln\mathbb E^{(k)}[e^{h(\bm{X})}]-\mathbb E^{(k)}[h(\bm{X})] depends on \bm{X} only through \bm{X}_{-k}. Since the sum has finitely many terms,

Since \mathbb{E}[\ell]=\mathbb{E}[\mathbb{E}^{(k)}[\ell]]=\mathbb{E}[\ell_k]=1,

We have thus proved, by (?), (?), and (?),

Using (?),

With

this proves the claim. The equivalent form follows since

For f(\bm{X})=\overline X_n, we have D_k(\bm{X})(\bm{X})\le(b_k-X_k)/n, so independence gives

Applying Theorem 3.1 yields (?), and inverting the tail bound gives (?).

Define the block-level upper bounds and the function f as follows: b_j\coloneqq\frac{1}{m_j}\sum_{i=1}^{m_j}b_{ji}, B\coloneqq\operatorname{med}(b_1,\ldots,b_K), f\coloneqq\widehat\mu_{\mathrm{MoM}}-B. By assumption, Y_j\le b_j for every j. This implies f\le0 almost surely. The finite moment assumptions on the samples also imply that \widehat\mu_{\mathrm{MoM}} is integrable. We denote the essential supremum with respect to X_{ji}, with all other observations held fixed, by \sup_{ji}f. Increasing X_{ji} by u\ge0 increases Y_j by u/m_j. Since the median is 1-Lipschitz in each of its arguments,

We write \mathbb{E}^{(ji)} for the expectation over X_{ji} with the other observations fixed. The independence of the samples gives the bound

Applying Theorem 3.1 to f=\widehat\mu_{\mathrm{MoM}}-B yields (3.9) because f-\mathbb{E} f=\widehat\mu_{\mathrm{MoM}}-\mathbb{E} \widehat\mu_{\mathrm{MoM}}.

If all blocks have the same size m_j=m, then V_{\mathrm{mom}}=m^{-p}\sum_{j=1}^K\sum_{i=1}^m M_{ji,p}=\overline M_n K^p n^{1-p}, where \overline M_n\coloneqq\frac 1n \sum_{j=1}^K\sum_{i=1}^mM_{ji,p}. Substituting this into (3.9) gives (3.10).

The proof of Corollary 3.11 relies on the next lemma.

Using the first part of Proof Proof and the equivalence

proves the first part of the claim in Corollary 3.11. Using the second part of Proof Proof and the equivalence of (\sqrt{3}/2)^K\le\delta \Leftrightarrow K\ge\frac{\ln(1/\delta)}{\ln(2/\sqrt3)} proves the second part of the claim in Corollary 3.11.

As in the proof of Corollary 3.11, we use \mathbb P(Y_j-\mu\ge t)\le\overline q_t (Corollary 3.6) and the identity (?). Instead of the Chernoff bound, we can use the union bound together with independence. For every t>0,

Define

Countability ensures measurability, and for any fixed h_0\in\mathcal{H},

Hence T_n is integrable. We apply Theorem 3.1 to f(X)=T_n. For k\in[n], let

Nonnegativity of the losses implies \sup_k f(X)\le T_{n,k}. For \eta>0, choose an \bm{X}_{-k}-measurable index h_{k,\eta} such that

Such a choice is obtained by taking the first qualifying index in a fixed enumeration of \mathcal{H}. Evaluating T_n at this index yields

Since h_{k,\eta} depends only on \bm{X}_{-k}, independence and (?) give \mathbb{E}^{(k)}[\ell_{h_{k,\eta}}(X_k)^p]\le M_p almost surely. Conditional Minkowski’s inequality therefore gives

Letting \eta\downarrow0 through a deterministic sequence and summing over k, we obtain

Theorem 3.1 now implies that, with probability at least 1-\delta,

Finally, let X' be an independent copy of X, independent of \varepsilon. Symmetrization yields

The equality follows by exchanging each pair (X_k,X'_k) according to the sign of \varepsilon_k, and the last inequality follows by splitting the supremum of the difference into two suprema. Combining the last two displays proves the claim.

Let \bm{Y}=(Y_1,\ldots,Y_n) be an independent copy of \bm{X}. For k=0,\ldots,n, define H^{(k)}\colon \mathcal X^n\times\mathcal X^n\to\mathcal X^n by

We note that H^{(0)}(x,y)=x and H^{(n)}(x,y)=y. For simplicity, we write

s_k(x_{-k}) is \sup_k f(x) in the main body. We make the independence of \sup_i f(x) and x_k explicit. With this notation, we can rewrite D_k(x) as D_k(x)=s_k(x_{-k})-f(x).

For each k, the vectors H^{(k-1)}(x,y) and H^{(k)}(x,y) agree in every coordinate except the k-th coordinate. By writing H^{(k)}_{-k}(x,y) to denote the elements of H^{(k)}(x,y) except for the k-th element, H^{(k)}_{-k}(x,y)=H^{(k-1)}_{-k}(x,y). Hence, for almost every (x,y), by the definition of s_k,

By adding both sides,

Summing over k\in[n] gives

Since D_k\geq 0 almost surely, taking the positive part yields, for almost every (x,y),

We then define

Each q_j(x_{-j}) is nonnegative. Also, by Assumption 3.1, \sum_{j=1}^n q_j(\bm{X}_{-j})\leq V P_{\bm{X}}\text{-a.s.}. It follows that, for every k,

Thus, by Fubini’s theorem, for P_{\bm{X}}-almost every x,

Minkowski’s inequality and monotonicity of L_p norm now yields

Equivalently,

Since f(\bm{Y}) is finite almost surely, there exists M\in\mathbb R such that \rho\coloneqq\mathbb P(f(\bm{Y})\leq M)>0. On the event \{f(\bm{Y})\leq M\},

It follows that, for P_{\bm{X}}-almost every x,

Therefore,

for P_{\bm{X}}-almost every x. This proves that f(\bm{X}) is essentially bounded above.

By the Chernoff bound and the Herbst argument (?),

where P_{tf} is the tilted measure defined in (?). For i\in[n], we define the conditional counterpart of a tilted measure P_h by tilting the i-th coordinate only. For fixed x_{-i}\in\mathcal{X}^n, let P^{(i)}_h(\cdot\mid x_{-i}) be the probability measure on the i-th coordinate space \mathcal{X} with

We write the KL divergence between conditional tilted measure and the marginal distribution as a function on \mathcal{X}^{n-1} as

For the conditional KL divergence averaged against the marginal of P_h,

By subadditivity of entropy [boucheron2013concentration, Theorem 4.22], [posta1997spectral, Proposition 4.1], also known as KL tensorization,

Denote a function D_i on \mathcal X^n defined as D_i(x_1^n)=\sup_i f(x_1^n)-f(x_1^n),

The last inequality holds because x-\ln(1+x)\le\frac{x^p}{p} for p\in(1,2] and x\ge0.
Thus,

Combined with (Proof),

The bound is worse than (?). To achieve the same bound as (?) by bounding the conditional KL divergence of the form D_{\mathrm{KL}}^{(i)}\!\left(-t D_i\right)(x_{-i}) \le C_p t^p\,\mathbb{E}^{(i)}[D_i^p](x_{-i}), we need C_p=\frac{p-1}{p}. However, C_p=\frac{p-1}{p} is not valid uniformly over all p\in(1,2] and all nonnegative random variables. For example, let

Then, this random variable satisfies Assumption 3.1 with n=1 and f(X)=-X. A direct calculation gives

On the other hand,

Thus C_p = \frac{p-1}{p} fails already at p = \frac65, so this method cannot recover (?) uniformly over p\in(1,2].

Both f and f_{\mathcal G} are probability densities with respect to P_{\bm{X}}, so {\rm Ent}_{P_{\bm{X}}}(f)=\mathbb E[f\ln f] and likewise for f_{\mathcal G}. Since f_{\mathcal G} is \mathcal G-measurable, the set N\coloneqq\{f_{\mathcal G}=0\} lies in \mathcal G, so it is an admissible test set in the defining property of conditional expectation, \int_A f\,d\mu=\int_A\mathbb E[f\mid\mathcal G]\,d\mu for all A\in\mathcal G. Taking A=N gives

because f_{\mathcal G} vanishes on N. As f\ge0, this forces f=0 \mu-a.e. on N. Consequently, if A\in\mathcal F satisfies \int_A f_{\mathcal G}\,d\mu=0, then f_{\mathcal G}=0 \mu-a.e. on A, i.e. \mu(A\setminus N)=0, and hence \int_A f\,d\mu=0. Therefore f\mu\ll f_{\mathcal G}\mu, and for every A\in\mathcal F,

so that \frac{d(f\mu)}{d(f_{\mathcal G}\mu)}=f/f_{\mathcal G} holds f_{\mathcal G}\mu-a.e., the values on N being irrelevant. Thus

and

For the second term, note that (\ln f_{\mathcal G})^{\pm} is \mathcal G-measurable and nonnegative, so pulling it out of the conditional expectation (valid for nonnegative factors, without integrability assumptions) and using the tower property gives

\mathbb E[f_{\mathcal G}\ln f_{\mathcal G}] \le\mathbb E[f\ln f]<\infty by Jensen applied to the convex function x\mapsto x\ln x. Also, \mathbb E[-f_{\mathcal G}\ln f_{\mathcal G}]\le\frac{1}{e} since \forall x\ge0, x\ln x\ge-\frac1e. Subtracting, \mathbb E[f\ln f_{\mathcal G}]=\mathbb E[f_{\mathcal G}\ln f_{\mathcal G}] ={\rm Ent}_{P_{\bm{X}}}(f_{\mathcal G}), which proves the claim.