Efficient Agnostic Learning of Univariate Densities Under Local Differential Privacy

We prove that the algorithm ? satisfies the requirements of the lemma.

Let j=\sum_{\ell\in [L]} b_\ell 2^{L-\ell}, be the L-bit binary representation of j, where b_\ell\in\{0,1\}. At depth \ell, Algorithm ? adds a node to \mathcal C(j) when b_\ell=1. Let s_0=0, and for each \ell\in [L] define s_\ell := \sum_{r\in [\ell]} b_r2^{L-r}. We show that, after processing depth \ell, the nodes selected so far are pairwise disjoint and their union is exactly [1,s_\ell]. If b_\ell=0, the algorithm does not add a node and s_\ell=s_{\ell-1}, so the claim holds. Otherwise, b_\ell=1. Let q=2^{L-\ell}. Note that at depth \ell, the variable s is initially equal to s_{\ell-1}, and every term in s_{\ell-1} is divisible by q, therefore v=\frac{s_{\ell-1}}{q} is an integer. So, the node selected by the algorithm is I_{\ell,v} = \{vq+1,\ldots,(v+1)q\} = \{s_{\ell-1}+1,\ldots,s_{\ell-1}+q\}. This node is disjoint from all previously selected nodes and is immediately adjacent to their union. Since s_\ell=s_{\ell-1}+q, the union of the selected nodes after depth \ell is exactly [1,s_\ell]. In particular at depth L, we have s_L = \sum_{\ell\in [L]}b_\ell2^{L-\ell} = j. Consequently, the nodes in \mathcal C(j) are pairwise disjoint and exactly cover [1,j], proving (1).

The algorithm considers each depth \ell\in[L] exactly once and adds at most one node during that iteration. Thus, \mathcal C(j) contains at most one node at every depth, proving (2). It also follows that |\mathcal C(j)| = \sum_{\ell\in [L]} b_\ell \leq L.

Finally, the algorithm performs L iterations, with a constant amount of operations at every iteration. Hence, it constructs \mathcal C(j) in O(L) time, which completes the proof.

Fix an arbitrary probability distribution P^\star satisfying the assumptions of the Proposition 5.4, and set k=(C+1)(d+1)s. By Lemma B.2,

We use the deterministic post-sampling adaptations of Theorem 17 of [acharya2017sample] for d>0 and Theorem 10 for d=0, with the supplied multiset Z replacing the sampling step. Their error analyses apply once the required \mathcal{A}_k-distance bound holds, which we have established above. With suitable universal constants and approximation tolerances, they therefore give

The output has at most Cs pieces of degree at most d, and the running time follows from the same deterministic algorithms on M input points. The conversion from Kolmogorov distance is used only in the analysis and requires no additional computation.

Now, we prove the utility and privacy gaurantee of our lazy PrefixQuery Algorithm ?.

Set L:=\lceil\log_2D\rceil. We first prove (1). Fix a user u\in[n], and let T_u:=\mathcal{M}(X_u,\varepsilon,D). Note that the depth \ell_u and Hadamard coordinate r_u are sampled independently of the input. Therefore, it is enough to compare the conditional probabilities of the reported bit Y_u. For any x,x'\in[D] and any possible transcript (\ell,r,y):

For any (\ell,r), the unperturbed signs corresponding to x and x' are b=\operatorname{Had}_\ell(v_\ell(x),r) and b'=\operatorname{Had}_\ell(v_\ell(x'),r), respectively. If b=b', then the two conditional distributions are identical. If b=-b', then the largest possible likelihood ratio is \frac{p_\varepsilon}{1-p_\varepsilon}=e^\varepsilon, which proves that \mathcal{M} is \varepsilon-LDP.

Also, note that each user produces its transcript T_u using only its own input and independent local randomness. All calls to \mathsf{PrefixQuery} are performed by the server after the transcript \operatorname{Tr}_{\mathcal{M}}(X)=(T_1,\ldots,T_n) has been collected. These calls only post-process \operatorname{Tr}_{\mathcal{M}}(X), so the protocol is non-interactive \varepsilon-LDP, regardless of the number of queries.

Now, we prove (2). The prefixes j=0 and j=D are exact by definition. Let j\in\{1,\cdots,D-1\} be any remaining query index, and x\in[D] be any domain point. Also, let \mathcal{C}(j)=\{(\ell_i,v_i): i\in[k]\} and

be defined as in Algorithm ?.

We show that:

Here, all expectations are over the depth sampling (\ell_u), the Hadamard-coordinate sampling (r_u), and randomized response (Y_u).

Conditioned on X_u=x, \ell_u=\ell, and r_u=r, randomized response satisfies:

Since \ell_u is uniform over \{1,\ldots,L\}, it follows from Hadamard orthogonality (Claim C.1) that

Recall that v_{\ell_i}(x)=\left\lfloor\frac{x-1}{2^{L-\ell_i}}\right\rfloor denotes the index of the unique depth-\ell_i node containing x. Since the intervals in \mathcal C(j) are disjoint and their union is [1,j], the last expression equals \boldsymbol 1\{x\leq j\}, proving (I).

By Lemma 5.2, the cover contains at most one node at every depth. Since the user samples only one depth, at most one term in the definition of Z_u can be nonzero. Furthermore, Y_u and every Hadamard entry belong to \{-1,+1\}. Hence |Z_u|\leq L/\theta_\varepsilon, proving (II).

For the fixed dataset X, the variables Z_1,\cdots,Z_n are independent, although they need not be identically distributed. Let \widetilde Z_j = \dfrac{1}{n} \sum\limits_{u\in [n]}Z_u. By (I),

Using (II) and Hoeffding’s inequality for independent bounded random variables implies:

Taking a union bound over the at most D prefix queries results in:

Since \theta_\varepsilon\geq\varepsilon/4 for 0<\varepsilon\leq1 and L\leq3\log D for D\geq2, the right-hand side is at most \beta under the stated sample-size condition. Finally, clipping \widetilde Z_j to [0,1] cannot increase its distance from Z_j(X)\in[0,1], which concludes the proof for (2).

Finally, we prove (3). Each user constructs and stores its report in O(1) time and O(1) words, so user preprocessing takes O(n) total time. For each prefix query, the server constructs the dyadic cover in O(L) time and scans the n reports in O(n) time, using a cover representation indexed by depth. Thus, one query takes O(n+\log D) time, and q\geq1 queries take O(q(n+\log D)) total time including user preprocessing, without materializing the full tree or a length-D vector.

Putting together, we are ready to prove the utility and privacy guarantee of our LDP synthetic data generator in Algorithm ?.

Set D:=\lceil2R/\delta\rceil+2 and L:=\lceil\log_2D\rceil. Since \delta\leq R/2, we have R/\delta\geq2, and \frac{2R}{\delta} \leq D \leq\frac{4R}{\delta}. In particular, \log D=O(\log(R/\delta)) and L=O(\log(R/\delta)).

Let the real line partitioning B_1=(-\infty,-R), B_j=[c_{j-1},c_j), B_D=[R,+\infty), be as in Definition 6.2, where c_0= -R-\delta, c_j= -R+\min\{(j-1)\delta,2R\}, c_D=R+\delta.

We first prove (1). Each user computes the interval index \widetilde X_u=\operatorname{Disc}_{R,\delta}(X_u) locally and applies the randomizer in Algorithm ?. For any two real inputs, their indices belong to [D], so Theorem 6.1 proves \varepsilon-local differential privacy. The protocol is non-interactive, and all remaining steps are post-processing of the transcripts \operatorname{Tr}.

Now, we prove (2). Let \widetilde P=\operatorname{Disc}_{R,\delta}(P), and write F_{\widetilde P}(x)= \mathbb{P}_{X\sim \widetilde P}(X\leq x) for its CDF. By Definition 6.2, F_{\widetilde P} is continuous and linear on each interval [c_{j-1},c_j], and

Let \widetilde X=\left(\widetilde X_1,\cdots,\widetilde X_n\right) be the discretized dataset, and define its prefix frequencies by

Let \alpha':=\alpha/2. Conditioned on any realization of \widetilde X, Theorem 6.1, with accuracy parameter \alpha'/2 and failure probability \beta/2, gives

with probability at least 1-\beta/2 over the local randomizers, provided that n \geq \frac{288(\log D)^2}{\varepsilon^2(\alpha'/2)^2} \log\left(\frac{4D}{\beta}\right).

Since the prefix-oracle guarantee holds for every fixed dataset, the same probability bound holds over the joint sampling and local randomness.

Furthermore, since X_1,\cdots,X_n are drawn i.i.d. from P, we have \mathbb{E}_{X\sim P}\left[Z_j\left(\operatorname{Disc}_{R,\delta}(X)\right)\right]=\mathbb{E}_{X\sim P}\left[Z_j\left(\widetilde X\right)\right]=F_{\widetilde P}(c_j). Using Hoeffding’s inequality, followed by a union bound over the prefixes, gives

Combining the two bounds, with probability at least 1-\beta over the sampling and local randomizers, the triangle inequality gives

for every j\in\{0,\cdots,D\} simultaneously. We condition on this event throughout the rest of the proof. Furthermore, \widehat Z_0=0 and \widehat Z_D=1, but the \widehat Z_j’s are not necessarily monotone.

For each \ell\in[M], the binary search maintains the condition A=\widehat Z_a<u_\ell\leq\widehat Z_b=B. At termination, a=b-1, so that B-A>0 and \lambda_\ell=(u_\ell-A)/(B-A)\in(0,1]. Thus, the interpolation step is well-defined, and it holds that u_\ell =(1-\lambda_\ell)\widehat Z_{b-1}+\lambda_\ell\widehat Z_b, and F_{\widetilde P}(z_\ell) =(1-\lambda_\ell)F_{\widetilde P}(c_{b-1})+\lambda_\ell F_{\widetilde P}(c_b). Therefore, the simultaneous accuracy bound implies:

Write F_H for the CDF of H. Fix any x\in\mathbb{R}. By the preceding accuracy bound and monotonicity of F_{\widetilde P}, if z_\ell\leq x, then u_\ell \leq F_{\widetilde P}(z_\ell)+\alpha' \leq F_{\widetilde P}(x)+\alpha'. Similarly, if z_\ell>x, then u_\ell \geq F_{\widetilde P}(z_\ell)-\alpha' \geq F_{\widetilde P}(x)-\alpha'.

Since u_\ell=(\ell-1/2)/M, these inequalities give

Putting together, gives

This argument does not require the estimated prefix values to be monotone.

Finally, to see (3), computing the interval index and generating the local randomized response each take O(1) time per user. Thus, user preprocessing takes O(n) total time. Generating the synthetic dataset requires M binary searches, each with at most L prefix queries and constant additional work per step. By Theorem 6.1, each query takes O(n+L) time. Thus, the total run-time of the algorithm is

where we used M=\lceil1/\alpha\rceil=O(1/\alpha) and L=O(\log(R/\delta)). The assumption \delta\leq R/2 is used only to simplify the bounds.

We now prove the guarantees of our LDP agnostic learning algorithm.

Let C\geq3 be a universal constant for which Proposition 5.4 holds, and set t=(C+1)(d+1)s, and M=\left\lceil\frac{2t}{\alpha}\right\rceil.

Let \widetilde P:=\operatorname{Disc}_{R,\delta}(P) with support I_{R,\delta}:=[-R-\delta,R+\delta]. Using Theorem 6.3, with accuracy parameter \alpha/(2t), the stated sample size suffices to obtain a synthetic multiset \mathcal{Z}\subseteq I_{R,\delta} of size M, with empirical probability measure H, such that

with probability at least 1-\beta. We condition on this event for the rest of the proof.

Run the deterministic algorithm from Proposition 5.4 on I_{R,\delta},\mathcal{Z},s,d, with parameter \eta=\alpha. Since \widetilde P has a density supported on I_{R,\delta}, the Kolmogorov bound (?) satisfies the requirement of the proposition for P^\star=\widetilde P. Thus, the output of the deterministic piece-wise polynomial learner Q\in\mathcal{P}_{Cs,d} is nonnegative and satisfies:

Fix any F\in\mathcal{F}. We can write:

where the first inequality follows from the triangle inequality, and the second follows from Definition 2.3 and the fact that \operatorname{Disc}_{R,\delta} is a Markov operator and does not increase L_1 distance (see Lemma C.2).

Consequently, we can write:

Putting together with (?), we get:

Since Q is a piece-wise polynomial supported on the bounded interval I_{R,\delta}, it is integrable. Let m_0:=\int_{\mathbb{R}}Q(x)\,dx < \infty, and define the normalized output distribution by:

If m_0>0, then

If m_0=0, then \|Q-\widetilde P\|_1=1 and \|\widehat P-\widetilde P\|_1\leq2. Therefore, in both cases, we have \|\widehat P-\widetilde P\|_1\leq2\|Q-\widetilde P\|_1. Note that \widehat P remains a piece-wise polynomial, since normalization preserves the number and degrees of the polynomial pieces, and the fallback density \operatorname{Unif}(I_{R,\delta}) has at most three pieces on \mathbb{R}.

Moreover, we can write:

where the first two inequalities follow from the triangle inequality, the third follows because \operatorname{Disc}_{R,\delta} is a Markov operator and does not increase L_1 distance (see Lemma C.2), and the last follows from Definition 2.3.

Finally, we can write:

where the first inequality follows from the triangle inequality, the second follows from (?), the third follows from (?), and the last follows from (?). Taking the infimum over F\in\mathcal{F} gives

Privacy and non-interactivity follow from Theorem 6.3, since fitting and normalization are post-processing operations. By Theorem 6.3 and Proposition 5.4, the total running time is

where \log_+(x):=\max\{1,\log x\}, and the \widetilde{O} term hides only logarithmic factors in M and d+1.

For the chosen sample size, n=\Omega(\log(R/\delta)), so n+\log(R/\delta)=O(n), t=O(s(d+1)), M=O(s(d+1)/\alpha), and M/s=O((d+1)/\alpha). Absorbing only logarithmic factors in s, d+1, and 1/\alpha therefore gives the stated runtime bound in the theorem. Integrating and normalizing the polynomial pieces takes O(s(d+1)) additional operations, which is absorbed in the bound.

We prove that bounding the polynomial approximation and discretization errors of mixture classes reduces to bounding the corresponding errors for the single-component classes.

Fix an arbitrary mixture F=\sum_{i\in[m]}w_iF_i\in{m}\text{-}\text{mix}(\mathcal{F}).

First we prove (1). By Definition 6.2, \operatorname{Disc}_{R,\delta}(F) = \sum_{i\in[m]}w_i\operatorname{Disc}_{R,\delta}(F_i). Therefore, the triangle inequality gives

Taking the supremum over F\in{m}\text{-}\text{mix}(\mathcal{F}) proves one direction. The reverse inequality follows from \mathcal{F}\subseteq{m}\text{-}\text{mix}(\mathcal{F}).

Next, we show (2). Fix \eta>0. For each i\in[m], choose P_i\in\mathcal{P}_{s,d} satisfying

and set P:=\sum_{i\in[m]}w_iP_i. Each P_i has at most s-1 breakpoints. Their common refinement therefore has at most m(s-1)+1 interval pieces, and taking weighted sums does not increase the degree of the underlying polynomials. Since P is nonnegative and integrable, we have P\in\mathcal{P}_{m(s-1)+1,d}. Moreover,

Consequently,

Taking the supremum over F\in{m}\text{-}\text{mix}(\mathcal{F}) and letting \eta tend to zero proves the result.

Putting together, we obtain our upper bound for LDP agnostic learning log-concave distributions.

Let \mathcal{F}={m}\text{-}\text{mix}\left(\mathcal{L}(R,\sigma_{\min},\sigma_{\max})\right) be the class of log-concave mixtures. Set \rho=\frac{\alpha}{25}, s_0=m(s_\rho-1)+1, d=1, where s_\rho is the integer from Lemma 8.6 at accuracy \rho. Thus, s_0=\widetilde{O}(m/\sqrt{\rho}).

Also let the discretization parameters T=R+\sigma_{\max}\left(1+\log\left(\frac{4}{\rho}\right)\right), J=\left\lceil\frac{8T}{\rho\sigma_{\min}}\right\rceil, and \delta=\frac{2T}{J} be defined as per Lemma 8.7. Since T>0 and J\geq4, we have 0<\delta\leq T/2. We can apply Theorem 7.1 with class \mathcal{F}, discretization parameters (T,\delta), privacy parameter \varepsilon, failure probability \beta, accuracy parameter \rho, and polynomial parameters (s_0,d), using sample size n=O\left( \frac{s_0^2(d+1)^2}{\varepsilon^2\rho^2} \log^2\left(\frac{T}{\delta}\right) \log\left(\frac{T}{\beta\delta}\right) \right). The output then satisfies

with probability at least 1-\beta, where \mathrm{OPT}^*=\inf_{F\in\mathcal{F}}\|P-F\|_1.

By Lemmas 8.6 and 8.2,

Moreover, Lemmas 8.7 and 8.2 give

Substituting these two bounds gives

Finally, s_0=\widetilde{O}(m/\sqrt{\rho}), d=1, and our parameter choices give J=O\left( \frac{R+\sigma_{\max}}{\rho\sigma_{\min}} \log\left(\frac{4}{\rho}\right) \right), and \log\left(\frac{T}{\delta}\right) =O\left( \log\left(\frac{2(R+\sigma_{\max})}{\alpha\sigma_{\min}}\right) \right). Substituting the chosen parameters into the sample and running-time bounds of Theorem 7.1 gives the stated complexities, suppressing only logarithmic factors in 1/\alpha in the sample bound, and in m and 1/\alpha in the running time. The output has \widetilde{O}(m/\sqrt{\alpha}) polynomial pieces and degree at most 1. Privacy and non-interactivity also follow from Theorem 7.1.

We prove a bound on the piece-wise polynomial approximation of log-concave distributions.

Let s_\alpha be the integer from Lemma E.1. For every f\in\mathcal{L}(R,\sigma_{\min},\sigma_{\max}), that lemma gives a normalized P\in\mathcal{P}_{s_\alpha,1} satisfying \|P-f\|_1\leq\alpha. Therefore, \inf_{P\in\mathcal{P}_{s_\alpha,1}}\|P-f\|_1 \leq\alpha. Taking the supremum over f\in\mathcal{L}(R,\sigma_{\min},\sigma_{\max}) proves the result.

Next, we prove a bound on the discretization error of log-concave distributions.

Fix an arbitrary distribution F\in\mathcal{L}(R,\sigma_{\min},\sigma_{\max}), and let f be its density, with mean \mu and standard deviation \sigma.

Define g(x):=\sigma f(\mu+\sigma x) for x\in\mathbb{R}. The function g is log-concave, and the change of variables y=\mu+\sigma x gives

Thus, g is a log-concave probability density with mean zero and variance one. By Lemma 5.5(a) of [lovasz2007geometry], \sup_{x\in\mathbb{R}}g(x)\leq1. Since f(x)=\frac{1}{\sigma}g\left(\frac{x-\mu}{\sigma}\right), we obtain \sup_{x\in\mathbb{R}}f(x) =\frac{1}{\sigma}\sup_{z\in\mathbb{R}}g(z) \leq\frac{1}{\sigma}.

We next bound the total tail mass. For X\sim f, the random variable Y=(X-\mu)/\sigma has density g. Since \mathbb{E}[Y^2]=1, Lemma 5.17 of [lovasz2007geometry] gives, for every t>1,

Since |\mu|\leq R and \sigma\leq\sigma_{\max}, it follows that

Therefore, F(\mathbb{R}\setminus[-T,T])\leq\eta/4.

Let D=J+2, and let B_j’s and c_j’s be the intervals and endpoints from Definition 6.2, with parameters (T,\delta). For each j\in\{2,\cdots,D-1\}, let h_j=c_j-c_{j-1}\leq\delta and

For every x\in B_j, we have

Since f is nonnegative and unimodal, the sum of its binwise differences between the supremum and infimum is bounded by its total increase plus its total decrease, each of which is at most \sup_{x\in\mathbb{R}}f(x). Therefore,

Integrating over x\in B_j and summing over the internal intervals yields

The original and discretized measures restricted to the two tails each have total mass F(\mathbb{R}\setminus[-T,T]). Their L_1 difference is therefore at most twice this mass. Combining the two bounds gives

Taking the supremum over F proves the result.

For any interval I=(a,b], we have

The same bound holds for other endpoint conventions and unbounded intervals by taking the corresponding one-sided limits of the CDFs. Taking the supremum over intervals gives

Moreover, for any collection of r\leq k disjoint intervals I_1,\cdots,I_r,

Taking the supremum over all such collections proves the result.

Let B_j’s be the intervals from Definition 6.2. Since the intervals B_j’s are pairwise disjoint, we have:

where the inequality follows because (B_j)_{j=1}^D is a measurable partition of \mathbb{R}.

Let \mathcal{F}={m}\text{-}\text{mix}\left(\mathcal{G}(R,\sigma_{\min},\sigma_{\max})\right) be the class of GMMs. Set \rho=\frac{\alpha}{25}, s_0=2m+1, d=\left\lceil C\log\left(\frac{2}{\rho}\right)\right\rceil, where C>0 is the universal constant from Lemma F.5.

Also let the discretization parameters T=R+\sigma_{\max}\left(1+\log\left(\frac{4}{\rho}\right)\right), J=\left\lceil\frac{8T}{\rho\sigma_{\min}}\right\rceil, and \delta=\frac{2T}{J} be defined as per Lemma F.3. Since T>0 and J\geq4, we have 0<\delta\leq T/2, we can apply Theorem 7.1 with class \mathcal{F}, discretization parameters (T,\delta), privacy parameter \varepsilon, failure probability \beta, accuracy parameter \rho, and polynomial parameters (s_0,d), using sample size n=O\left( \frac{s_0^2(d+1)^2}{\varepsilon^2\rho^2} \log^2\left(\frac{T}{\delta}\right) \log\left(\frac{T}{\beta\delta}\right) \right). The output then satisfies

with probability at least 1-\beta, where \mathrm{OPT}^*:=\inf_{G\in\mathcal{F}}\|P-G\|_1.

By Lemmas F.5 and 8.2,

Moreover, Lemmas F.3 and 8.2 give

Substituting these two bounds gives

Finally, our choices of T and \delta give \log\left(\frac{T}{\delta}\right) =O\left( \log\left(\frac{2(R+\sigma_{\max})}{\alpha\sigma_{\min}}\right) \right). Substituting the chosen parameters into the sample and running-time bounds of Theorem 7.1 gives the stated complexities, suppressing only logarithmic factors in m and 1/\alpha in the running time. The output has O(m) polynomial pieces and degree O(\log(2/\alpha)). Privacy and non-interactivity also follow from Theorem 7.1.

Every Gaussian density is log-concave. Therefore, \mathcal{G}(R,\sigma_{\min},\sigma_{\max}) \subseteq \mathcal{L}(R,\sigma_{\min},\sigma_{\max}).

By Lemma 8.7, we get \operatorname{Disc\text{-}Err}_{T,\delta} \left(\mathcal{G}(R,\sigma_{\min},\sigma_{\max})\right) \leq \operatorname{Disc\text{-}Err}_{T,\delta} \left(\mathcal{L}(R,\sigma_{\min},\sigma_{\max})\right) \leq\eta.

Let C>0 be the universal constant from Lemma F.4. For every g\in\mathcal{G}(R,\sigma_{\min},\sigma_{\max}), that lemma gives a normalized P\in\mathcal{P}_{s,d} satisfying \|P-g\|_1\leq\alpha. Therefore, \inf_{P\in\mathcal{P}_{s,d}}\|P-g\|_1 \leq\alpha. Taking the supremum over g\in\mathcal{G}(R,\sigma_{\min},\sigma_{\max}) proves the result.

Now, we give our lower bound proof for LDP learning of GMMs.

Define parameters K_\mu=1+\left\lfloor\frac{R}{\sigma_{\min}}\right\rfloor, K_\sigma=1+\left\lfloor\frac{\log(\sigma_{\max}/\sigma_{\min})}{\log16} \right\rfloor, K=\max\{K_\mu,K_\sigma\}, and M=\min\{m,K\}.

Taking C_0\geq100(1+\log16), the assumption gives K \geq \max\left\{ \frac{R}{\sigma_{\min}}, \frac{\log(\sigma_{\max}/\sigma_{\min})}{\log16} \right\} \geq \frac{\Lambda}{1+\log16} \geq100m. Thus, M=m and 2\leq M\leq K/100, as required by the sparse estimation lower bound in Theorem G.2.

We construct K Gaussian components g_1,\ldots,g_K\in\mathcal{G}(R,\sigma_{\min},\sigma_{\max}) and disjoint measurable sets B_1,\ldots,B_K\subset\mathbb{R} such that g_j(B_j)\geq2/3 for all j\in[K]. Let \Phi denote the standard normal CDF. We use the larger of the following two constructions.

(1) Location Gaussians. For j\in[K_\mu], set \mu_j=(2j-K_\mu-1)\sigma_{\min}, g_j=\mathcal{N}(\mu_j,\sigma_{\min}^2), and B_j=[\mu_j-\sigma_{\min},\mu_j+\sigma_{\min}). Then, the means satisfy |\mu_j|\leq(K_\mu-1)\sigma_{\min}\leq R. The set B_j’s are disjoint, and

(2) Scale Gaussians. For j\in[K_\sigma], set \sigma_j=16^{j-1}\sigma_{\min}, g_j=\mathcal{N}(0,\sigma_j^2), and B_j=\{x:\sigma_j/4\leq|x|<4\sigma_j\}. We have \sigma_j\in[\sigma_{\min},\sigma_{\max}]. Then, the sets B_j’s are disjoint, and

For p\in\Delta_K, write f_p:=\sum_{j\in[K]}p_jg_j, and let \gamma=1/3. Since the sets are disjoint, \sum_{i\neq j}g_j(B_i)\leq1-g_j(B_j). Thus, the reverse triangle inequality gives, for any p,q\in\Delta_K,

We reduce learning \mathcal{S}_{K,M} to learning the Gaussian mixture class. Given a discrete sample Y=j, the corresponding user independently draws X\sim g_j and applies the local randomizer of the Gaussian mixture learner to X. For any such randomizer Q and measurable message event A, integrating its privacy inequality gives

for all i,j\in[K]. Hence, the resulting protocol remains non-interactive \varepsilon-LDP. If Y\sim p\in\mathcal{S}_{K,M}, then X\sim f_p, and f_p has at most M\leq m Gaussian components.

Let \widehat f be the output of the learner, and define \widehat p\in\operatorname*{arg\,min}_{q\in\mathcal{S}_{K,M}} \operatorname{d_{\textsc{TV}}}(\widehat f,f_q). A minimizer exists because \mathcal{S}_{K,M} is compact and the objective is continuous in q.

On the event \operatorname{d_{\textsc{TV}}}(\widehat f,f_p)\leq\alpha, we have

This decoding is post-processing and returns an M-sparse probability vector. The resulting discrete learner has accuracy 2\alpha/\gamma and succeeds with probability at least 0.9. Choosing \alpha_0 so that 2\alpha_0/\gamma\leq c_0, Theorem G.2 therefore gives n=\Omega\left( \frac{m^2\log(K/m)}{\varepsilon^2\alpha^2} \right). Finally, since K=\Theta(\Lambda) and K/m\geq100, \log(K/m)=\Theta\left(\log\left(e+\frac{\Lambda}{m}\right)\right), which gives the stated bound.

Now, we prove how to obtain a proper learner from a non-proper learner.

Run \mathcal{A} to obtain a distribution \widehat P. Approximate each distance \operatorname{d_{\textsc{TV}}}(\widehat P,G), for G\in\mathcal{C}, to additive error at most \alpha/2, and return a member \widetilde P\in\mathcal{C} with the smallest estimated distance, breaking ties according to a fixed ordering of \mathcal{C}. This ensures that

Let P be the sampling distribution. With probability at least 1-\beta, the guarantee of \mathcal{A} gives

Condition on this event. Fix any F^\star\in\mathcal{F}. Since \mathcal{C} is an \alpha-cover, there exists G^\star\in\mathcal{C} such that \operatorname{d_{\textsc{TV}}}(F^\star,G^\star)\leq\alpha. Therefore,

Taking the infimum over F^\star\in\mathcal{F} proves the accuracy guarantee. The additional errors account for the cover and the approximate minimization.

The conversion computes |\mathcal{C}| distance approximations and makes O(|\mathcal{C}|) additional comparisons, giving O(|\mathcal{C}|(T_{\mathrm{dist}}+1)) additional running time. It uses only the output of \mathcal{A}, so it requires no additional samples and preserves privacy and non-interactivity by post-processing.

For a target additive error \tau\in(0,1), applying this construction with \alpha=\tau/4 and a cover of radius \tau/4 gives a proper (2C+1,\tau,\beta)-agnostic learner using n_{\mathcal{A}}(\tau/4,\beta) samples.

We prove the cover construction for GMMs.

Let f=\sum_{i\in[m]}w_i\mathcal{N}(\mu_i,\sigma_i^2) be an arbitrary member of the class. We construct a cover by discretizing the mixture weights, means, and standard deviations.

For the weights, let C_w be an \alpha/m-cover of \Delta_m in \ell_\infty, of size at most \lceil m/\alpha\rceil^m, as given by Claim H.4.

For the means, let \mathcal{C}_\mu be a grid on [-R,R] containing both endpoints, with consecutive points at distance at most \alpha\sigma_{\min}/2. When R=0, take \mathcal{C}_\mu=\{0\}. The grid can be chosen to have at most 1+\lceil4R/(\alpha\sigma_{\min})\rceil points, and every \mu\in[-R,R] has a grid point \tilde\mu satisfying |\mu-\tilde\mu|\leq\alpha\sigma_{\min}/4.

For the standard deviations, take

For every \sigma\in[\sigma_{\min},\sigma_{\max}], there is a grid point \tilde\sigma such that \tilde\sigma\leq\sigma<(1+\alpha/4)\tilde\sigma, and hence |\log(\sigma/\tilde\sigma)|\leq\alpha/4.

Let \mathcal{C} contain all mixtures formed from weights in C_w, means in \mathcal{C}_\mu, and standard deviations in \mathcal{C}_\sigma. Every member of \mathcal{C} belongs to the target class.

For univariate Gaussians, varying the mean and then the standard deviation gives

Choose the corresponding grid parameters for f, and write \tilde f=\sum_{i\in[m]}\tilde w_i \mathcal{N}(\tilde\mu_i,\tilde\sigma_i^2)\in\mathcal{C}. Then

Finally,

for a universal constant c>0.

Partition [0,1] into \lceil1/\alpha\rceil intervals of length at most \alpha, and use their Cartesian products to partition [0,1]^m. For every cell that intersects \Delta_m, select one point from the intersection. Any two points in the same cell have \ell_\infty distance at most \alpha, so the selected points form an \alpha-cover of \Delta_m. There are at most \lceil1/\alpha\rceil^m cells.