Generalization Bounds for Physics-Informed Neural Networks for the Incompressible Navier-Stokes Equations

Sebastien Andre-Sloan, Dibyakanti Kumar, Alejandro F Frangi and Anirbit Mukherjee

arXiv: 2603.23072v1 · PDF · 24 March 2026
Categories: cs.LG, cs.NA, math.AP, math.NA
Licence: CC BY 4.0 The site shows the text.

Abstract

This work establishes rigorous first-of-its-kind upper bounds on the generalization error for the method of approximating solutions to the (d+1)-dimensional incompressible Navier-Stokes equations by training depth-2 neural networks trained via the unsupervised Physics-Informed Neural Network (PINN) framework. This is achieved by bounding the Rademacher complexity of the PINN risk. For appropriately weight bounded net classes our derived generalization bounds do not explicitly depend on the network width and our framework characterizes the generalization gap in terms of the fluid’s kinematic viscosity and loss regularization parameters. In particular, the resulting sample complexity bounds are dimension-independent. Our generalization bounds suggest using novel activation functions for solving fluid dynamics. We provide empirical validation of the suggested activation functions and the corresponding bounds on a PINN setup solving the Taylor-Green vortex benchmark.

Items 22

Theorem

Theorem (Informal Statement of Theorem 1)

EM52

Consider neural networks {\mathcal{N}}_{\bm{w}} belonging to a class of depth-2 neural networks. For the PINN empirical risk corresponding to the (d+1)-dimensional Navier–Stokes equations, denoted by \hat{R}({\mathcal{N}}_{\bm{w}}, {\mathcal{S}}_n), where {\mathcal{S}}_n represents the training set, and the corresponding population risk R({\mathcal{N}}_{\bm{w}}), the data-averaged worst case generalization error — defined as the difference between the empirical and population risks — can be bounded as,

\begin{aligned} &\mathbb{E}_{{\mathcal{S}}_n} \left[\sup_{{\mathcal{N}}_{\bm{w}}} \left( \hat{R}({\mathcal{N}}_{\bm{w}}, {\mathcal{S}}_n) - R({\mathcal{N}}_{\bm{w}}) \right)\right] \leq \frac{C_r}{\sqrt{N_r}} + \frac{C_0}{\sqrt{N_0}}, \end{aligned}

where N_r denotes the number of training data points in the domain and N_0 the number of boundary points corresponding to the initial condition. The constants C_r and C_0 can be expressed explicitly in terms of the Lipschitz constant of the loss function, the bounds on certain norms of the weights, the viscosity coefficient, the regularization on the loss terms enforcing the initial-condition and the incompressibility condition, and the bounds on the moments of the data distribution. The constant C_r grows linearly with the viscosity coefficient.

Definition

Definition 1 (Huber Loss)

0B26

For some \delta \geq 0 the Huber loss is defined as

\begin{aligned} \ell_{H,\delta} (x) \coloneqq \begin{cases} \frac{1}{2} x^2 &\quad \mbox{for } \abs{x}\leq \delta \\ \delta \cdot (\abs{x} - \frac{1}{2} \delta) &\quad \mbox{for } \abs{x} > \delta \end{cases} \end{aligned}

Definition

Definition 2 (Neural Network for (d+1)-Navier-Stokes)

Z1XB

Let \sigma : \mathbb R \to \mathbb R, the input {\bm{z}}=({\bm{x}},t) \in \mathbb R^{d+1}, the weights {\bm{W}} \in \mathbb R^{p \times (d+1)}, where {\bm{W}} = [ {\bm{w}}_1, {\bm{w}}_2, \dots, {\bm{w}}_p ]^\top, {\bm{w}}_q \in \mathbb R^{d+1} for q\in\{1,\dots p\}, and the frozen second layer weights {\bm{A}}_1\in\mathbb R^{d\times p} and {\bm{a}}_2\in\mathbb R^p. Then we can define a neural network {\mathcal{N}}_{\bm{w}} : [0,1]^{d+1} \rightarrow \mathbb R^{d+1} whose output coordinates are denoted as mapping,

({\bm{x}},t) = {\bm{z}} \mapsto ({\mathcal{N}}_{{\bm{w}},u}({\bm{z}}), {\mathcal{N}}_{{\bm{w}},p}({\bm{z}}) )

\text{where,}

\begin{aligned} {\mathcal{N}}_{{\bm{w}},u}({\bm{z}}) \coloneqq {\bm{A}}_1\sigma({\bm{W}} {\bm{z}}), ~{\mathcal{N}}_{{\bm{w}},p}({\bm{z}}) \coloneqq \left\langle{\bm{a}}_2,\sigma({\bm{W}} {\bm{z}})\right\rangle \end{aligned}

\text{and} ~\sigma({\bm{W}} {\bm{z}}) = [\sigma({\bm{w}}^\top_1 {\bm{z}}), \sigma({\bm{w}}^\top_2 {\bm{z}}),\dots,\sigma({\bm{w}}^\top_p {\bm{z}})]^\top

Used by Proof

Definition

Definition 3 (Collocation Points for (d+1)-Navier-Stokes)

2UGN

Let the sampled collocation points available to solve the PDE be denoted as the following two sets of points,

\begin{aligned} \{ ({\bm{x}}_{ri},t_{ti}) \in [0,1]^d \times [0,1] \mid i=1,\ldots,N_r \}, ~\{ {\bm{x}}_{0j} \in [0,1]^d \mid j=1,\ldots,N_0 \} \end{aligned}

We also assume that there exists constant such that \mathbb{E}_{{\bm{z}}_{0j}}[\norm{{\bm{z}}_{0j}}_2^2] \leq C_{{\bm{z}}_0}^2, \forall j \in [N_0], where {\bm{z}}_{0j} \coloneqq ({\bm{x}}_{0j},0), and \mathbb{E}_{{\bm{z}}_i}[\norm{{\bm{z}}_i}_2^2] \leq C_{{\bm{z}}}^2, \forall i \in [N_r], where {\bm{z}}_i = ({\bm{x}}_{ri}, t_{ri}).

Used by Definition 4, Theorem 1, Proof

Definition

Definition 4 (Empirical Risk for (d+1)-Navier-Stokes)

YW93

For a Lipschitz loss function \ell : \mathbb R \rightarrow [0,\infty), a neural net {\mathcal{N}}_{\bm{w}} : [0,1]^{d+1} \rightarrow \mathbb R^{d+1} and regularization parameters \lambda_0, \lambda_1 >0, the corresponding empirical risk with collocation points given by Definition 3 can be written as,

\begin{aligned} \hat{R}({\mathcal{N}}_{\bm{w}}) \coloneqq &\frac{1}{N_r}\sum_{i=1}^{N_r} \sum_{k=1}^d\ell \left( (\partial_t {\mathcal{N}}_{{\bm{w}},u})_k + (({\mathcal{N}}_{{\bm{w}},u} \cdot \nabla) {\mathcal{N}}_{{\bm{w}},u})_k + (\nabla {\mathcal{N}}_{{\bm{w}},p})_k - \nu (\nabla^2 {\mathcal{N}}_{{\bm{w}},u})_k \right )\mid_{({\bm{x}}_{ri},t_{ri})} \\ &+ \frac{\lambda_0}{N_r}\sum_{i=1}^{N_r}\ell \left ( \nabla\cdot {\mathcal{N}}_{{\bm{w}},u} \right )\mid_{({\bm{x}}_{ri},t_{ri})} + \frac{\lambda_1}{N_0}\sum_{j=1}^{N_0} \sum_{k=1}^d\ell \left ( ({\mathcal{N}}_{{\bm{w}},u}({\bm{x}},0))_k - ({\bm{f}}_0 ({\bm{x}}))_k \right )\mid_{{\bm{x}}_{0j}} \end{aligned}

Uses Definition 3

Definition

Definition 5 (Loss Function for (d+1)-Navier-Stokes)

3O0K

We can define two separate loss functions, \ell_{res} and \ell_{0} s.t they measure the error at any space-time interior point and a spatial point, respectively, as follows,

\begin{aligned} \ell_{res}({\mathcal{N}}_{\bm{w}}, ({\bm{x}}_{ri},t_{ri})) &\coloneqq \sum_{k=1}^d\ell \left( (\partial_t {\mathcal{N}}_{{\bm{w}},u})_k + (({\mathcal{N}}_{{\bm{w}},u} \cdot\nabla){\mathcal{N}}_{{\bm{w}},u})_k + (\nabla {\mathcal{N}}_{{\bm{w}},p})_k - \nu (\nabla^2 {\mathcal{N}}_{{\bm{w}},u})_k \right )\mid_{({\bm{x}}_{ri},t_{ri})} \\ &\hspace{2.00em}+ \lambda_0 \cdot \ell \left (\grad \cdot {\mathcal{N}}_{{\bm{w}},u} \right )\mid_{({\bm{x}}_{ri},t_{ri})}\\ \ell_{0}({\mathcal{N}}_{\bm{w}}, {\bm{x}}_{0j}) &\coloneqq \lambda_1 \sum_{k=1}^d \ell \left ( ({\mathcal{N}}_{{\bm{w}},u}({\bm{x}},0))_k - ({\bm{f}}_0 ({\bm{x}}))_k \right )\mid_{{\bm{x}}_{0j}} \end{aligned}

and thus the empirical risk can we written as,

\begin{aligned} \hat{R}({\mathcal{N}}_{\bm{w}}) \coloneqq &\frac{1}{N_r}\sum_{i=1}^{N_r} \ell_{res}({\mathcal{N}}_{\bm{w}}, ({\bm{x}}_{ri},t_{ri})) + \frac{1}{N_0}\sum_{j=1}^{N_0} \ell_{0}({\mathcal{N}}_{\bm{w}}, {\bm{x}}_{0j}). \end{aligned}

Used by Proof

Definition

Definition 6 (Class of Weights for (d+1)-Navier-Stokes)

TY0V

Given weights {\bm{W}} \in \mathbb R^{p \times (d+1)}, let {\bm{w}}_t = {\bm{W}}_{:,\,d+1}, {\bm{w}}_{{\bm{x}}_m} = {\bm{W}}_{:,\,m}, and {\bm{a}}_{1k} = {\bm{A}}_{1k,:} for m, k \in \{1, \dots, d\}. Define {\bm{w}}_{\bm{x}} = \sum_{m=1}^d {\bm{w}}_{{\bm{x}}_m} and {\bm{a}}_1 = \sum_{k=1}^d {\bm{a}}_{1k}.

We can then define the following functions:

Inparaenum ((i)).

f_1({\bm{W}})={\bm{a}}_1\odot{\bm{w}}_t,

f_{2m}({\bm{W}})={\bm{a}}_{1}\odot{\bm{w}}_{{\bm{x}}_m},

f_3({\bm{W}})={\bm{a}}_2\odot{\bm{w}}_{\bm{x}},

f_4({\bm{W}})=\sum_{m=1}^d{\bm{a}}_1\odot{\bm{w}}_{{\bm{x}}_m}\odot{\bm{w}}_{{\bm{x}}_m}, and

f_5({\bm{W}})=\sum_{m=1}^d{\bm{a}}_{1m}\odot{\bm{w}}_{{\bm{x}}_m},

where {\bm{a}}_{1m}, {\bm{a}}_1, {\bm{a}}_2, {\bm{w}}_{{\bm{x}}_m}, {\bm{w}}_t, and {\bm{w}}_{{\bm{x}}} are as given above.

Now, given constants B_{f_i} for i \in [5] by the following constraints are defined,

\begin{gathered} \sum_{q=1}^p \abs{f_s({\bm{W}})_q} \leq B_{f_s} \textrm{ for $s \in \{1,3,4,5\}$}\ ; \qquad \sum_{m=1}^d\sum_{q_1,q_2 = 1}^p \abs{f_{2m}({\bm{W}})_{q_1}} \abs{a_{1m,q_2}} \leq B_{f_2}, \end{gathered}

where f_1, f_{2m}, f_3, f_4 and f_5 are the functions defined above.

Corresponding to the above, we define the class of weights {\mathcal{C}} as,

\begin{aligned} {\mathcal{C}} \coloneq \big\{{\bm{W}} \in \mathbb R^{p \times (d+1)}\ \big| {\bm{W}} \text{ satisfies the bounds given in Equation \ref{eq:fbounds}} \big\}. \end{aligned}

We further define \textrm{RS}({\mathcal{C}}) as the set of possible row vectors from the elements of {\mathcal{C}}.

Used by Proof

Theorem

Theorem 1 (Rademacher Based Upper-bound for (d+1)-Navier-Stokes)

CI3M

Let {\mathcal{N}}_{{\bm{w}}} be a neural network as in Definition 2, with weights {\bm{W}} \in {\mathcal{C}}, where the class of weights {\mathcal{C}} is defined in Definition 6. We make the following assumptions about the activation function \sigma(x):

Inparaenum ((i)).

\sigma(x) is twice differentiable,

\sigma(x), \sigma'(x), and \sigma''(x) are Lipschitz continuous with constants L_{\sigma}, L_{\sigma'}, and L_{\sigma''}, respectively, and

\sigma(x) and \sigma'(x) are bounded by constants B_{\sigma} and B_{\sigma'}, respectively.

Let the two components of the loss function, \ell_{res} and \ell_0, be defined as in Definition 5. From Definition 3, we recall the constants C_{\bm{z}} and C_{{\bm{z}}_0}, which characterize the distributions of the collocation points in the interior and on the initial-condition boundaries, respectively. We can then bound the generalization error of the empirical loss by

\begin{aligned} &\mathbb{E}_{\substack{({\bm{x}}_{ri},t_{ri}) \sim {\mathcal{D}}_1 \forall i \in [N_r] \\ {\bm{x}}_{0j} \sim {\mathcal{D}}_2 \forall j\in[N_0]}} \Bigg[\sup_{{\bm{W}} \in {\mathcal{C}}} \biggl( \frac{1}{N_r} \sum_{i=i}^{N_r} \ell_{res}({\mathcal{N}}_{\bm{w}},({\bm{x}}_{ri},t_{ri})) + \frac{1}{N_0} \sum_{j=i}^{N_0} \ell_{0}({\mathcal{N}}_{\bm{w}},{\bm{x}}_{0j})\\ &\hspace{15.00em}- \left( \mathbb{E}_{({\bm{x}}_r,t_r) \sim {\mathcal{D}}_1}[\ell_{res}({\mathcal{N}}_{{\bm{w}}},({\bm{x}}_{r},t_{r}))] + \mathbb{E}_{{\bm{x}}_0 \sim {\mathcal{D}}_2}[\ell_{0}({\mathcal{N}}_{{\bm{w}}},{\bm{x}}_{0})] \right) \biggr)\Bigg] \\ &\hspace{1.00em}\leq \frac{2 \delta (B_{{\bm{w}}} C_{\bm{z}} C_1 + C_2)}{\sqrt{N_r}} + \frac{4 \lambda_1 \delta B_{\bm{a}} (B_{{\bm{w}}} C_{{\bm{z}}_0} L_\sigma + \abs{c_0})}{\sqrt{N_0}}, \end{aligned}

here, C_1 \coloneqq 2B_{f_1}L_{\sigma'}+4B_{f_2}(B_\sigma L_{\sigma'}+B_{\sigma'}L_\sigma)+2B_{f_3}L_{\sigma'}+2\nu B_{f_4} L_{\sigma''}+2 \lambda_0 B_{f_5}L_{\sigma'} and C_2 \coloneqq B_{f_1}\abs{c_1} + B_{f_2}(2B_{\sigma'}\abs{c_0} + \abs{c_0 c_1}) + B_{f_3}\abs{c_1} + \nu B_{f_4}\abs{c_2} + \lambda_0 B_{f_5}\abs{c_1}, where B_{f_s} for s \in [5] are as defined in Definition 6, c_0 = \sigma(0), c_1 = \sigma'(0), c_2 = \sigma''(0), and \delta is the Lipschitz constant of the loss function \ell_{H,\delta} as in Definition 1. We also assume that there exist constants B_{{\bm{w}}} and B_{\bm{a}} such that \norm{{\bm{w}}_q}_2 \leq B_{{\bm{w}}}, \forall q \in [p], where {\bm{w}}_q = {\bm{W}}_{q,:}, and \sum_{q=1}^p \abs{({\bm{a}}_1)_q} \leq B_{\bm{a}}, for {\bm{a}}_1 is as in Definition 2.

Uses Definition 3

Remark

Remark 1

3V1D

The constants in Theorem 1 for specific choices of activation functions are as follows:

\sigma = \tanh:
L_\sigma = L_{\sigma'} = 1,\;\; L_\sigma'' = 2,\;\; B_\sigma = B_{\sigma'} = 1,\;\; c_0 = c_2 = 0,\;\; c_1 = 1.
\sigma = \tanh^3:
L_\sigma = 0.75,\;\; L_{\sigma'} = 1.4,\;\; L_\sigma'' = 6,\;\; B_\sigma = 1,\;\; B_{\sigma'} = 0.75,\;\; c_0 = c_1 = c_2 = 0.

Lemma

Lemma 2

GUDB

Continuing within the framework of Theorem 1, it follows that a way to ensure that the RHS is {\mathcal{O}}(\epsilon) the number of collocation points N_r and initial condition points N_0 can be chosen as,

\begin{aligned} N_r &= {\mathcal{O}}\left(\frac{\delta^2 (B_{\bm{w}} C_{\bm{z}} C_1 + C_2)^2}{\epsilon^2}\right)\\ N_0 &= {\mathcal{O}}\left(\frac{\lambda_1^2 \delta^2 B_{\bm{a}}^2 (B_{\bm{w}} C_{{\bm{z}}_0} L_\sigma + \abs{c_0})^2}{\epsilon^2}\right), \end{aligned}

where C_1 \coloneqq 2B_{f_1}L_{\sigma'}+4B_{f_2}(B_\sigma L_{\sigma'}+B_{\sigma'}L_\sigma)+2B_{f_3}L_{\sigma'}+2\nu B_{f_4} L_{\sigma''}+2 \lambda_0 B_{f_5}L_{\sigma'} and C_2 \coloneqq B_{f_1}\abs{c_1} + B_{f_2}(2B_{\sigma'}\abs{c_0} + \abs{c_0 c_1}) + B_{f_3}\abs{c_1} + \nu B_{f_4}\abs{c_2} + \lambda_0 B_{f_5}\abs{c_1}.

Used by Remark 2

Remark

Remark 2

SJIL

From the above lemma, we can further infer that a suggested ratio between the collocation points and initial points is,

\frac{N_r}{N_0} = {\mathcal{O}}\left(\frac{(B_{\bm{w}} C_{\bm{z}} C_1 + C_2)^2}{\lambda_1^2 B_{\bm{a}}^2 (B_{\bm{w}} C_{{\bm{z}}_0} L_\sigma + \abs{c_0})^2}\right)

Uses Lemma 2

Lemma

Lemma 3

VP69

Continuing in the setup of Definition 5, it follows that,

\begin{aligned} &\mathbb{E}_{\substack{({\bm{x}}_{ri},t_{ri}) \sim {\mathcal{D}}_1 \forall i \in [N_r] \\ {\bm{x}}_{0j} \sim {\mathcal{D}}_2 \forall j\in[N_0]}} \Bigg[\sup_{h\in{\mathcal{H}}} \Bigg( \frac{1}{N_r} \sum_{i=i}^{N_r} \ell_{res}(h,({\bm{x}}_{ri},t_{ri})) + \frac{1}{N_0} \sum_{j=i}^{N_0} \ell_{0}(h,{\bm{x}}_{0j}) \\ &\hspace{15.00em}- \left( \mathbb{E}_{({\bm{x}}_r,t_r) \sim {\mathcal{D}}_1}[\ell_{res}(h,({\bm{x}}_{r},t_{r}))] + \mathbb{E}_{{\bm{x}}_0 \sim {\mathcal{D}}_2}[\ell_{0}(h,{\bm{x}}_{0})] \right) \Bigg)\Bigg]\\ &\leq 2{\mathcal{R}}_{res} ({\mathcal{H}}) + 2{\mathcal{R}}_{0} ({\mathcal{H}}) \end{aligned}

where,

\begin{aligned} {\mathcal{R}}_{res} ({\mathcal{H}}) &\coloneqq \mathbb{E}_{\substack{({\bm{x}}_{ri},t_{ri}) \sim D_1 \forall i \in [N_r]}}\left[ \mathbb{E}_{\epsilon_i\sim \{\pm 1\}} \left[ \sup_{h\in{\mathcal{H}}} \left(\frac{1}{N_r} \sum_{i=i}^{N_r} \epsilon_i\ell_{res}(h,({\bm{x}}_{ri},t_{ri})) \right) \right] \right] \\ {\mathcal{R}}_0 ({\mathcal{H}}) &\coloneqq \mathbb{E}_{\substack{{\bm{x}}_{0j}' \sim {\mathcal{D}}_2 \forall j\in[N_0]}} \left[\mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_0}} \left[ \sup_{h\in{\mathcal{H}}} \left( \frac{1}{N_0} \sum_{j=i}^{N_0} \epsilon_j \ell_{0}(h,{\bm{x}}_{0j}) \right)\right]\right] \end{aligned}

Used by Proof

Lemma

Lemma 4

T3PF

The specific partial derivatives needed to expand the loss are given by:

\begin{aligned} \partial_{x_k} {\mathcal{N}}_{{\bm{w}},p}({\bm{z}}) =\left\langle{\bm{a}}_2\odot{\bm{w}}_{x_k},\sigma'({\bm{W}}{\bm{z}})\right\rangle, ~&(\partial_t{\mathcal{N}}_{{\bm{w}},{\bm{u}}}({\bm{z}}))_k =\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_t,\sigma'({\bm{W}}{\bm{z}})\right\rangle\\ \nabla\cdot{\mathcal{N}}_{{\bm{w}},u}({\bm{z}})=\sum_{m=1}^d\left\langle{\bm{a}}_{1m}\odot{\bm{w}}_{x_m},\sigma'({\bm{W}}{\bm{z}})\right\rangle,&(({\mathcal{N}}_{{\bm{w}},{\bm{u}}}\cdot\nabla){\mathcal{N}}_{{\bm{w}},{\bm{u}}})_k =\sum_{m=1}^d(\left\langle{\bm{a}}_{1m},\sigma({\bm{W}}{\bm{z}})\right\rangle)(\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{{\bm{x}}_m},\sigma'({\bm{W}}{\bm{z}})\right\rangle)\\ (\nabla^2{\mathcal{N}}_{{\bm{w}},{\bm{u}}}({\bm{z}}))_k&=\sum_{m=1}^d\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{x_m}\odot{\bm{w}}_{x_m},\sigma''({\bm{W}}{\bm{z}})\right\rangle \end{aligned}

where {\bm{z}} = ({\bm{x}}, t).

Used by Proof

Lemma

Lemma 5

81H1

(A Variation of Lemma 5.12 of [tengyucs229m]) Given a real valued function f_{\bm{w}} parameterized by {\bm{w}} and a data-set {\bm{Z}} = ({\bm{z}}_1,\ldots,{\bm{z}}_n) define, f_{\bm{w}}({\bm{Z}}) = (f_{\bm{w}}({\bm{z}}_1),\dots ,f_{\bm{w}}({\bm{z}}_n)) and g_{\bm{w}}({\bm{Z}}) = (f_{\bm{w}}({\bm{z}}_1)-c,\dots ,f_{\bm{w}}({\bm{z}}_n)-c), where c \in \mathbb R is some constant. Suppose that for any {\bm{\epsilon}} \in \{\pm 1\}^n, \sup_{\bm{w}} \left\langle{\bm{\epsilon}},g_{\bm{w}}({\bm{Z}})\right\rangle \geq 0. Then,

\begin{aligned} \mathbb{E}_{{\bm{\epsilon}}\sim\{\pm 1\}^n}\left[ \sup_{\bm{w}} \abs{\left\langle{\bm{\epsilon}},f_{\bm{w}}({\bm{Z}})\right\rangle} \right] \leq 2 \mathbb{E}_{{\bm{\epsilon}}\sim\{\pm 1\}^n}\left[ \sup_{\bm{w}} \left\langle{\bm{\epsilon}},g_{\bm{w}}({\bm{Z}})\right\rangle \right] + \abs{c}\sqrt{n} \end{aligned}

Used by Proof, Proof, Proof

Lemma

Lemma 6

ICNR

Assume that \phi : \mathbb R \rightarrow \mathbb R is an L_\phi-Lipschitz continuous, \phi(0) = c, for some constant c \in \mathbb R, and \sum_{m=1}^{p} \abs{f({\bm{W}})_m} \leq B, where f({\bm{W}}) : \mathbb R^{p \times d} \to \mathbb R^p. Then,

\begin{aligned} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \left( \frac{1}{N_r} \sum_{i=1}^{N_r} \epsilon_i \left\langlef({\bm{W}}),\phi({\bm{W}} {\bm{z}}_i)\right\rangle \right) \right] \leq \frac{2B L_{\phi}}{N_r} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}} \left[ \sup_{\bar{{\bm{w}}}\in\textrm{RS}({\mathcal{C}})} \sum_{i=1}^{N_r} \epsilon_i \left\langle\bar{{\bm{w}}},{\bm{z}}_i\right\rangle \right] + \frac{B\abs{c}}{\sqrt{N_r}} \end{aligned}

Used by Proof

Lemma

Lemma 7

53SB

(The LLM Claude was used in some parts of the proof of Lemma 7) Assume that \phi_1 : \mathbb R \rightarrow \mathbb R and \phi_2 : \mathbb R \to \mathbb R are Lipschitz continuous function with Lipschitz constants L_{\phi_1} and L_{\phi_2} respectively, and \phi_1(0) \cdot \phi_2(0) = k, for some constant k \in \mathbb R. Let \phi_1 \leq B_{\phi_1}, \phi_2 \leq B_{\phi_2} and let there be d functions f_{m},m\in[d],f_m({\bm{W}}) : \mathbb R^{p \times d} \to \mathbb R^p such that \sum_{m=1}^d \sum_{q_1,q_2 = 1}^p \abs{f_m({\bm{W}})_{q_1}} \abs{{\bm{a}}_{m,q_2}} \leq B. Then,

Used by Proof

Proof

6UTR

First we can bound the generalization error by the Rademacher expression of the empirical loss as given by Lemma 3, giving us

\begin{aligned} &\mathbb{E}_{\substack{({\bm{x}}_{ri},t_{ri}) \sim {\mathcal{D}}_1 \forall i \in [N_r] \\ {\bm{x}}_{0j} \sim {\mathcal{D}}_2 \forall j\in[N_0]}} \Bigg[\sup_{h\in{\mathcal{H}}} \Bigg( \frac{1}{N_r} \sum_{i=i}^{N_r} \ell_{res}(h,({\bm{x}}_{ri},t_{ri})) + \frac{1}{N_0} \sum_{j=i}^{N_0} \ell_{0}(h,{\bm{x}}_{0j}) \\ &\hspace{14.00em}- \left( \mathbb{E}_{({\bm{x}}_r,t_r) \sim {\mathcal{D}}_1}[\ell_{res}(h,({\bm{x}}_{r},t_{r}))] + \mathbb{E}_{{\bm{x}}_0 \sim {\mathcal{D}}_2}[\ell_{0}(h,{\bm{x}}_{0})] \right) \Bigg)\Bigg]\\ &\leq 2{\mathcal{R}}_{res} ({\mathcal{H}}) + 2{\mathcal{R}}_{0} ({\mathcal{H}}) \end{aligned}

where,

By Definition 5, we substitute the definition of \ell_{res} and \ell_0 and the partial derivatives given by Lemma 4 to get,

\begin{aligned} {\mathcal{R}}_{res} &= \mathbb{E}_{\substack{({\bm{z}}_i\sim D_1)\\\forall i\in[N_r]}}\Biggl[\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{1}{N_r}\sum_{i=1}^{N_r}\epsilon_i\Bigl\{\sum_{k=1}^d\ell\Bigl(\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{t},\sigma'({\bm{W}}{\bm{z}})\right\rangle +\sum_{m=1}^d(\left\langle{\bm{a}}_{1m},\sigma({\bm{W}}{\bm{z}}_i)\right\rangle)(\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{{\bm{x}}_m},\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle) \\ &\hspace{7.00em}+ \left\langle{\bm{a}}_{2}\odot{\bm{w}}_{x_k},\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle-\nu\sum_{m=1}^d\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{x_m}\odot{\bm{w}}_{x_m},\sigma''({\bm{W}}{\bm{z}}_i)\right\rangle\Bigr) + \lambda_0\ell\Bigl(\sum_{m=1}^d\left\langle{\bm{a}}_{1m}\odot{\bm{w}}_{x_m},\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle\Bigr)\Bigr\}\biggr)\Biggr]\Biggr], \end{aligned}

where {\bm{z}}_i = ({\bm{x}}_{ri}, t_{ri}).

\begin{aligned} {\mathcal{R}}_0 = \lambda_1 \mathbb{E}_{\substack{{\bm{x}}_{0j}' \sim {\mathcal{D}}_2 \forall j\in[N_0]}} \left[\mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_0}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \left( \frac{1}{N_0} \sum_{j=i}^{N_0} \sum_{k=1}^d\epsilon_j \ell \left ( ({\mathcal{N}}_{{\bm{w}},u}({\bm{x}}_{0j},0))_k - ({\bm{f}}_0 ({\bm{x}}_{0j}))_k \right ) \right)\right] \right], \end{aligned}

Firstly we focus on {\mathcal{R}}_{res} and we rewrite as the following sum splitting across the regularization for the divergenceless condition.

\begin{aligned} {\mathcal{R}}_{res} &= \mathbb{E}_{\substack{({\bm{z}}_i\sim D_1)\\\forall i\in[N_r]}}\Biggl[\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{1}{N_r}\sum_{i=1}^{N_r}\epsilon_i\Bigl\{\sum_{k=1}^d\ell\Bigl(\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{t},\sigma'({\bm{W}}{\bm{z}})\right\rangle +\sum_{m=1}^d(\left\langle{\bm{a}}_{1m},\sigma({\bm{W}}{\bm{z}}_i)\right\rangle)(\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{{\bm{x}}_m},\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle) \\ &\hspace{7.00em}+ \left\langle{\bm{a}}_{2}\odot{\bm{w}}_{x_k},\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle-\nu\sum_{m=1}^d\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{x_m}\odot{\bm{w}}_{x_m},\sigma''({\bm{W}}{\bm{z}}_i)\right\rangle\Bigr)\Bigr\}\biggr)\Biggr]\Biggr] \\ &\hspace{2.00em} + \mathbb{E}_{\substack{({\bm{z}}_i\sim D_1)\\\forall i\in[N_r]}}\Biggl[\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{1}{N_r}\sum_{i=1}^{N_r}\epsilon_i\Bigl\{\lambda_0\ell\Bigl(\sum_{m=1}^d\left\langle{\bm{a}}_{1m}\odot{\bm{w}}_{x_m},\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle\Bigr)\Bigr\}\biggr)\Biggr]\Biggr] \end{aligned}

Using the assumption from Definition 2 that the loss is L_\ell-Lipschitz, we can use Talagrand’s contraction lemma on each of the terms separately to get,

\begin{aligned} {\mathcal{R}}_{res}&\leq L_\ell \mathbb{E}_{\substack{({\bm{z}}_i\sim D_1)\\\forall i\in[N_r]}}\Biggl[\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{1}{N_r}\sum_{i=1}^{N_r}\epsilon_i\Bigl\{\sum_{k=1}^d\Bigl(\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{t},\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle \\ &\hspace{14.00em}+\sum_{m=1}^d(\left\langle{\bm{a}}_{1m},\sigma({\bm{W}}{\bm{z}}_i)\right\rangle)(\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{{\bm{x}}_m},\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle)\\ &\hspace{14.00em}+\left\langle{\bm{a}}_{2}\odot{\bm{w}}_{x_k},\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle-\nu\sum_{m=1}^d\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{x_m}\odot{\bm{w}}_{x_m},\sigma''({\bm{W}}{\bm{z}}_i)\right\rangle\Bigr)\Bigr\}\biggr)\Biggr]\Biggr]\\ &+ L_\ell \mathbb{E}_{\substack{({\bm{z}}_i\sim D_1)\\\forall i\in[N_r]}}\Biggl[\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{1}{N_r}\sum_{i=1}^{N_r}\epsilon_i\Bigl\{\lambda_0\sum_{m=1}^d\left\langle{\bm{a}}_{1m}\odot{\bm{w}}_{x_m},\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle\Bigr\}\biggr)\Biggr]\Biggr] \end{aligned}

Observing the structure of the above we define the following five quantities.

-12pt -4pt -2pt < - -12pt -4pt -2pt6pt12bp14bp0pt plus 1fil=0pt subsection2-12pt -4pt -2pt6pt12bp14bp0pt plus 1fil=0pt*Divide and Conquer Let {\mathcal{R}}_1,{\mathcal{R}}_2,{\mathcal{R}}_3,{\mathcal{R}}_4,{\mathcal{R}}_5 be such that

\begin{aligned} {\mathcal{R}}_1&=\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{1}{N_r}\sum_{i=1}^{N_r}\epsilon_i\left\langle{\bm{a}}_{1}\odot{\bm{w}}_{t},\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle\biggr)\Biggr]=\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{1}{N_r}\sum_{i=1}^{N_r}\epsilon_i\left\langlef_1({\bm{W}}),\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle\biggr)\Biggr]\\ {\mathcal{R}}_2&=\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{1}{N_r}\sum_{i=1}^{N_r}\sum_{k=1}^d\sum_{m=1}^d\epsilon_i\left\langle{\bm{a}}_{1m},\sigma({\bm{W}}{\bm{z}}_i)\right\rangle\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{{\bm{x}}_m},\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle\biggr)\Biggr]\\ &=\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{1}{N_r}\sum_{i=1}^{N_r}\sum_{m=1}^d\epsilon_i\left\langlef_{2m}({\bm{W}}),\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle\left\langle{\bm{a}}_{1m},\sigma({\bm{W}}{\bm{z}}_i)\right\rangle\biggr)\Biggr]&\\ {\mathcal{R}}_3&=\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{1}{N_r}\sum_{i=1}^{N_r}\epsilon_i\left\langle{\bm{a}}_{2}\odot{\bm{w}}_{\bm{x}},\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle\biggr)\Biggr]=\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{1}{N_r}\sum_{i=1}^{N_r}\epsilon_i\left\langlef_3({\bm{W}}),\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle\biggr)\Biggr]\\ {\mathcal{R}}_4 &=\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{-\nu}{N_r}\sum_{i=1}^{N_r}\sum_{m=1}^d\epsilon_i\left\langle{\bm{a}}_{1}\odot{\bm{w}}_{x_m}\odot{\bm{w}}_{x_m},\sigma''({\bm{W}}{\bm{z}}_i)\right\rangle\biggr)\Biggr]=\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{-\nu}{N_r}\sum_{i=1}^{N_r}\epsilon_i\left\langlef_4({\bm{W}}),\sigma''({\bm{W}}{\bm{z}}_i)\right\rangle\biggr)\Biggr]\\ {\mathcal{R}}_5 &=\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{\lambda_0}{N_r}\sum_{i=1}^{N_r}\sum_{m=1}^d\epsilon_i\left\langle{\bm{a}}_{1m}\odot{\bm{w}}_{x_m},\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle\biggr)\Biggr]=\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{\lambda_0}{N_r}\sum_{i=1}^{N_r}\epsilon_i\left\langlef_5({\bm{W}}),\sigma'({\bm{W}}{\bm{z}}_i)\right\rangle\biggr)\Biggr] \end{aligned}

Hence we can rewrite equation ? to get,

\begin{aligned} {\mathcal{R}}_{res}&\leq L_\ell\mathbb{E}_{\substack{({\bm{x}}_{ri},t_{ri}\sim D_1)\\\forall i\in[N_r]}}\Biggl[{\mathcal{R}}_1+{\mathcal{R}}_2+{\mathcal{R}}_3+{\mathcal{R}}_4+{\mathcal{R}}_5\Biggr]& \end{aligned}

Bounding Terms 1, 3 and 5

We recall the Lipschitz constants L_{\sigma'} for \sigma'(\cdot) and \sigma'(0) = c_1. Therefore, for {\mathcal{R}}_1, {\mathcal{R}}_3, and {\mathcal{R}}_5, we can invoke Lemma 6, with \phi(\cdot) = \sigma'(\cdot), and recalling the definition of B_{f_1}, B_{f_3} and B_{f_5} from Definition 6 to get

\begin{aligned} {\mathcal{R}}_{1} &\leq \frac{2 B_{f_1} L_{\sigma'}}{N_r} \mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[ \sup_{\bar{{\bm{w}}}\in\textrm{RS}({\mathcal{C}})} \sum_{i=1}^{N_r} \epsilon_i \left\langle\bar{{\bm{w}}},{\bm{z}}_i\right\rangle \right ] + \frac{B_{f_1} \abs{c_1}}{\sqrt{N_r}} \\ {\mathcal{R}}_{3} &\leq \frac{2 B_{f_3} L_{\sigma'}}{N_r} \mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[ \sup_{\bar{{\bm{w}}}\in\textrm{RS}({\mathcal{C}})} \sum_{i=1}^{N_r} \epsilon_i \left\langle\bar{{\bm{w}}},{\bm{z}}_i\right\rangle \right ] + \frac{B_{f_3} \abs{c_1}}{\sqrt{N_r}}\\ {\mathcal{R}}_{5} &\leq \frac{2 \lambda_0 B_{f_5} L_{\sigma'}}{N_r} \mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[ \sup_{\bar{{\bm{w}}}\in\textrm{RS}({\mathcal{C}})} \sum_{i=1}^{N_r} \epsilon_i \left\langle\bar{{\bm{w}}},{\bm{z}}_i\right\rangle \right] + \frac{B_{f_5} \abs{c_1}}{\sqrt{N_r}} \end{aligned}

Bounding Term 2

Since \sigma(\cdot) is L_{\sigma}-Lipschitz, \sigma(\cdot) \leq B_{\sigma}, \sigma'(\cdot) is L_{\sigma'}-Lipschitz, \sigma'(\cdot) \leq B_{\sigma'}, and \sigma(0) \cdot \sigma'(0) = c_0 c_1 , we can apply Lemma 7 with, \phi_1(\cdot) = \sigma(\cdot), \phi_2(\cdot) = \sigma'(\cdot) and k = c_0 c_1 , and recalling the definition of B_{f_2} from Definition 6 to get

\begin{aligned} {\mathcal{R}}_2 \leq \frac{4B_{f_2}(B_{\sigma} L_{\sigma'} + B_{\sigma'} L_{\sigma})}{N_r}\, \mathbb{E}_{\epsilon}\left[\sup_{\bar{w} \in RS(C)} \sum_{i=1}^{N_r} \epsilon_i \langle \bar{{\bm{w}}}, {\bm{z}}_i\rangle\right] + \frac{B_{f_2}(2B_{\sigma'}|c_0| + |c_0 c_1|)}{\sqrt{N_r}} \end{aligned}

Bounding Term 4

For the 4th term, since we are taking the expectation for \epsilon_i \sim \{\pm 1\}, we can simplify {\mathcal{R}}_4 as

\begin{aligned} {\mathcal{R}}_4 =\mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\Biggl[\sup_{{\bm{W}}\in{\mathcal{C}}}\biggl(\frac{\nu}{N_r}\sum_{i=1}^{N_r}\epsilon_i\left\langlef_4({\bm{W}}),\sigma''({\bm{W}}{\bm{z}}_i)\right\rangle\biggr)\Biggr]. \end{aligned}

We recall the Lipschitz constants L_{\sigma''} for \sigma''(\cdot) and \sigma''(0) = c_2. Therefore, for {\mathcal{R}}_4, we can invoke Lemma 6 with \phi(\cdot) = \sigma''(\cdot) and recalling the definition of B_{f_4} from Definition 6 to get

\begin{aligned} {\mathcal{R}}_{4} &\leq \frac{2\nu B_{f_4} L_{\sigma''}}{N_r} \mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[ \sup_{\bar{{\bm{w}}} \in \textrm{RS}({\mathcal{C}})} \sum_{i=1}^{N_r} \epsilon_i \left\langle\bar{{\bm{w}}},{\bm{z}}_i\right\rangle \right ] + \frac{\nu B_{f_4} \abs{c_2}}{\sqrt{N_r}}. \end{aligned}

-12pt -4pt -2pt < - -12pt -4pt -2pt6pt12bp14bp0pt plus 1fil=0pt subsection2-12pt -4pt -2pt6pt12bp14bp0pt plus 1fil=0pt*Combining the Bounds on Terms 1-5

\begin{aligned} {\mathcal{R}}_{res} &\leq L_\ell ( {\mathcal{R}}_1 + {\mathcal{R}}_2 + {\mathcal{R}}_3 + {\mathcal{R}}_4 + {\mathcal{R}}_5 )\\ &\leq L_\ell\left(\frac{2B_{f_1}L_{\sigma'}+4B_{f_2}(B_\sigma L_{\sigma'}+B_{\sigma'}L_\sigma)+2B_{f_3}L_{\sigma'}+2\nu B_{f_4}(L_{\sigma'}+L_{\sigma})+2 \lambda_0 B_{f_5}L_{\sigma'}}{N_r}\right)\times \mathbb{E}_{{\bm{\epsilon}}\sim\{\pm1\}^{N_r}}\left[\sup_{\bar{{\bm{w}}}\in\textrm{RS}({\mathcal{C}})}\sum_{i=1}^{N_r}\epsilon_i\left\langle\bar{{\bm{w}}},{\bm{z}}_i\right\rangle\right] \\ &\hspace{2.00em}+ \frac{L_\ell (B_{f_1}\abs{c_1} + B_{f_2}(2B_{\sigma'}\abs{c_0} + \abs{c_0 c_1}) + B_{f_3}\abs{c_1} + \nu B_{f_4}c_2 + \lambda_0 B_{f_5}\abs{c_1})}{\sqrt{N_r}} \end{aligned}

So, we are finally left with a Rademacher complexity of a linear model

\begin{aligned} \hat{{\mathcal{R}}}_{res} \leq \frac{L_\ell C_1}{N_r} \mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[ \sup_{\bar{{\bm{w}}}\in\textrm{RS}({\mathcal{C}})} \sum_{i=1}^{N_r} \epsilon_i \left\langle\bar{{\bm{w}}},{\bm{z}}_i\right\rangle\right] + \frac{L_\ell C_2}{\sqrt{N_r}}, \end{aligned}

where C_1 \coloneqq 2B_{f_1}L_{\sigma'}+4B_{f_2}(B_\sigma L_{\sigma'}+B_{\sigma'}L_\sigma)+2B_{f_3}L_{\sigma'}+2\nu B_{f_4}(L_{\sigma'}+L_{\sigma})+2 \lambda_0 B_{f_5}L_{\sigma'} and C_2 \coloneqq B_{f_1}\abs{c_1} + B_{f_2}(2B_{\sigma'}\abs{c_0} + \abs{c_0 c_1}) + B_{f_3}\abs{c_1} + \nu B_{f_4}\abs{c_2} + \lambda_0 B_{f_5}\abs{c_1}.

We recall that \norm{{\bm{w}}_q}_2 \leq B_{{\bm{w}}}, \forall q \in [p], where {\bm{w}}_q = {\bm{W}}_{q,:}, and \mathbb{E}_{{\bm{z}}_i}[\norm{{\bm{z}}_i}_2^2] \leq C_{{\bm{z}}}^2, \forall i \in [N_r], where {\bm{z}}_i = ({\bm{x}}_{ri}, t_{ri}) from Definition 3. Then using the standard Rademacher complexity bound for linear models (Theorem 5.5 of [tengyucs229m]) we get,

\begin{aligned} \hat{{\mathcal{R}}}_{res} \leq \frac{L_\ell B_{{\bm{w}}} C_1}{N_r} \sqrt{\sum_{i=1}^{N_r} \norm{{\bm{z}}_i}_2^2} + \frac{L_\ell C_2}{\sqrt{N_r}} \end{aligned}

and taking the expectation over the data gives us,

\begin{aligned} {\mathcal{R}}_{res} \leq \frac{L_\ell (B_{{\bm{w}}} C_{\bm{z}} C_1 + C_2)}{\sqrt{N_r}}. \end{aligned}

-12pt -4pt -2pt < - -12pt -4pt -2pt6pt12bp14bp0pt plus 1fil=0pt subsection2-12pt -4pt -2pt6pt12bp14bp0pt plus 1fil=0pt*Analysing the Initial Condition Enforcing Term

\begin{aligned} {\mathcal{R}}_0 = \lambda_1 \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_0}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \left( \frac{1}{N_0} \sum_{j=i}^{N_0} \sum_{k=1}^d\epsilon_j \ell \left ( ({\mathcal{N}}_{{\bm{w}},u}({\bm{x}}_{0j},0))_k - ({\bm{f}}_0 ({\bm{x}}_{0j}))_k \right ) \right)\right] \end{aligned}

Since the loss \ell is Lipschitz with constant L_\ell we can apply Talagrand’s contraction to get

\begin{aligned} &\leq \frac{\lambda_1 L_\ell}{N_0} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_0}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \left( \sum_{j=i}^{N_0} \sum_{k=1}^d\epsilon_j (({\mathcal{N}}_{{\bm{w}},u}({\bm{x}}_{0j},0))_k - ({\bm{f}}_0 ({\bm{x}}_{0j}))_k) \right)\right] \\ &= \frac{\lambda_1 L_\ell}{N_0} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_0}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \left( \sum_{j=i}^{N_0} \sum_{k=1}^d\epsilon_j ({\mathcal{N}}_{{\bm{w}},u}({\bm{x}}_{0j},0))_k \right)\right]. \end{aligned}

Let’s define {\bm{z}}_{0j} \coloneqq ({\bm{x}}_{0j},0), then we can simplify the above as

\begin{aligned} &= \frac{\lambda_1 L_\ell}{N_0} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_0}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \left( \sum_{j=i}^{N_0} \epsilon_j \left\langle{\bm{a}}_1,\sigma({\bm{W}}{\bm{z}}_{0j})\right\rangle \right)\right] \\ &= \frac{\lambda_1 L_\ell}{N_0} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_0}} \left[ \sup_{{\bm{w}}_q \in\textrm{RS}({\mathcal{C}})} \left( \sum_{q=1}^p ({\bm{a}}_1)_q \sum_{j=i}^{N_0} \epsilon_j \sigma(\left\langle{\bm{w}}_q,{\bm{z}}_{0j}\right\rangle)\right)\right]\\ &\leq \frac{\lambda_1 L_\ell}{N_0} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_0}} \left[ \sup_{{\bm{w}}_r \in\textrm{RS}({\mathcal{C}})} \left( \sum_{q=1}^p \abs{({\bm{a}}_1)_q} \max_{r \in [p]} \abs{\sum_{j=i}^{N_0} \epsilon_j \sigma(\left\langle{\bm{w}}_r,{\bm{z}}_{0j}\right\rangle)} \right)\right] \end{aligned}

Using our assumption that \sum_{q=1}^p \abs{({\bm{a}}_1)_q} \leq B_{\bm{a}}, we can say that,

\begin{aligned} &\leq \frac{\lambda_1 L_\ell B_{\bm{a}}}{N_0} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_0}} \left[ \sup_{{\bm{w}}_r \in\textrm{RS}({\mathcal{C}})} \left( \max_{r \in [p]} \abs{\sum_{j=i}^{N_0} \epsilon_j \sigma(\left\langle{\bm{w}}_{r},{\bm{z}}_{0j}\right\rangle)}\right)\right] \\ &\leq \frac{\lambda_1 L_\ell B_{\bm{a}}}{N_0} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_0}} \left[ \sup_{{\bm{w}}_r \in\textrm{RS}({\mathcal{C}})} \left( \abs{\sum_{j=i}^{N_0} \epsilon_j \sigma(\left\langle{\bm{w}}_r,{\bm{z}}_{0j}\right\rangle)} \right)\right] \end{aligned}

Utilizing the fact that \sigma(0) = c_0 if we set \chi(\cdot) = \phi(\cdot) - c_0, we note that for any {\bm{\epsilon}} \in \{\pm 1\}^{N_r}, \sup_{{\bm{w}}} \left\langle{\bm{\epsilon}},\chi({\bm{Z}}^\top{\bm{w}})\right\rangle \geq 0, where {\bm{Z}} = [{\bm{z}}_1,\dots,{\bm{z}}_{N_r}]. Hence, we can apply Lemma 5 to get,

\begin{aligned} &\leq \frac{ \lambda_1 L_\ell B_{\bm{a}}}{N_0} \Bigg\{2\mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_0}} \left[ \sup_{{\bm{w}}_r\in\textrm{RS}({\mathcal{C}})} \left( \sum_{j=i}^{N_0} \epsilon_j \sigma(\left\langle{\bm{w}}_r,{\bm{z}}_{0j}\right\rangle) \right)\right] + \abs{c_0} \sqrt{N_0} \Bigg\} \end{aligned}

Now, since \sigma(\cdot) is L_\sigma-Lipschitz, we can further apply Talagrand’s contraction to get,

\begin{aligned} &\leq \frac{2 \lambda_1 L_\ell L_\sigma B_{\bm{a}}}{N_0} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_0}} \left[ \sup_{{\bm{w}}_r\in\textrm{RS}({\mathcal{C}})} \left( \sum_{j=i}^{N_0} \epsilon_j \left\langle{\bm{w}}_r,{\bm{z}}_{0j}\right\rangle \right)\right] + \frac{\lambda_1 L_\ell B_{\bm{a}} \abs{c_0}}{\sqrt{N_0}}\\ \end{aligned}

We recall that \norm{{\bm{w}}_r}_2 \leq B_{{\bm{w}}}, \forall r \in [p], where {\bm{w}}_r = {\bm{W}}_{r,:}, and \mathbb{E}_{{\bm{z}}_{0j}}[\norm{{\bm{z}}_{0j}}_2^2] \leq C_{{\bm{z}}_0}^2, \forall j \in [N_0], from Definition 3. Then using the standard Rademacher complexity bound for linear models (Theorem 5.5 of [tengyucs229m]),

\begin{aligned} \hat{{\mathcal{R}}}_{0} \leq \frac{2 \lambda_1 L_\ell B_{{\bm{w}}} L_\sigma B_{\bm{a}}}{N_0}\sqrt{\sum_{j=1}^{N_0} \norm{{\bm{z}}_{0j}}^2} + \frac{\lambda_1 L_\ell B_{\bm{a}} \abs{c_0}}{\sqrt{N_0}} \end{aligned}

and taking the expectation over the data gives us,

\begin{aligned} {\mathcal{R}}_{0} \leq \frac{\lambda_1 L_\ell B_{\bm{a}} (2B_{{\bm{w}}} C_{{\bm{z}}_0} L_\sigma + \abs{c_0})}{\sqrt{N_0}} . \end{aligned}

Therefore, combining the above with the bound on {\mathcal{R}}_{\rm res} derived earlier we can bound the empirical Rademacher complexity as,

\begin{aligned} &{\mathcal{R}}_{\rm res} +{\mathcal{R}}_0 \leq \frac{L_\ell (B_{{\bm{w}}} C_{\bm{z}} C_1 + C_2)}{\sqrt{N_r}} + \frac{\lambda_1 L_\ell B_{\bm{a}} (2B_{{\bm{w}}} C_{{\bm{z}}_0} L_\sigma + \abs{c_0})}{\sqrt{N_0}} \end{aligned}

Further, by setting the loss function \ell to be the Huber loss \ell_{H,\delta} for some \delta > 0 as in Definition 1, we can simplify the above bound to

\begin{aligned} &{\mathcal{R}}_{\rm res} +{\mathcal{R}}_0 \leq \frac{\delta (B_{{\bm{w}}} C_{\bm{z}} C_1 + C_2)}{\sqrt{N_r}} + \frac{\lambda_1 \delta B_{\bm{a}} (2 B_{{\bm{w}}} C_{{\bm{z}}_0} L_\sigma + \abs{c_0})}{\sqrt{N_0}} \end{aligned}

Uses Lemma 3, Definition 5, Lemma 4, Definition 2, Lemma 6, Definition 6, Lemma 7, Definition 3, Lemma 5

Proof

MFZZ

\begin{aligned} &\sup_{h\in{\mathcal{H}}} \left( \frac{1}{N_r} \sum_{i=i}^{N_r} \ell_{res}(h,({\bm{x}}_{ri},t_{ri})) + \frac{1}{N_0} \sum_{j=i}^{N_0} \ell_{0}(h,{\bm{x}}_{0j}) - \left( \mathbb{E}_{({\bm{x}}_r,{\bm{t}}_r) \sim {\mathcal{D}}_1}[\ell_{res}(h,({\bm{x}}_{r},{\bm{t}}_{r}))] + \mathbb{E}_{{\bm{x}}_0 \sim {\mathcal{D}}_2}[\ell_{0}(h,{\bm{x}}_{0})] \right) \right)\\ &= \sup_{h\in{\mathcal{H}}} \left( \frac{1}{N_r} \sum_{i=i}^{N_r} \ell_{res}(h,({\bm{x}}_{ri},t_{ri})) + \frac{1}{N_0} \sum_{j=i}^{N_0} \ell_{0}(h,{\bm{x}}_{0j}) - \left( \mathbb{E}_{({\bm{x}}_{ri}',t_{ti}') \sim {\mathcal{D}}_1 \forall i \in [N_r]}\left[\frac{1}{N_r} \sum_{i=i}^{N_r} \ell_{res}(h,({\bm{x}}_{ri}',t_{ri}'))\right] \right.\right. \\ &\hspace{26.0em} + \left.\left. \mathbb{E}_{{\bm{x}}_{0j}' \sim {\mathcal{D}}_2 \forall j\in[N_0]}\left[\frac{1}{N_0} \sum_{j=i}^{N_0} \ell_{0}(h,{\bm{x}}_{0j}')\right] \right) \right)\\ &= \sup_{h\in{\mathcal{H}}} \left( \mathbb{E}_{({\bm{x}}_{ri}',t_{ti}') \sim {\mathcal{D}}_1 \forall i \in [N_r]} \left[\frac{1}{N_r} \sum_{i=i}^{N_r} \ell_{res}(h,({\bm{x}}_{ri},t_{ri})) - \frac{1}{N_r} \sum_{i=i}^{N_r} \ell_{res}(h,({\bm{x}}_{ri}',t_{ri}')) \right] \right. \\ &\hspace{17.0em} \left. + \mathbb{E}_{{\bm{x}}_{0j}' \sim {\mathcal{D}}_2 \forall j\in[N_0]} \left[ \frac{1}{N_0} \sum_{j=i}^{N_0} \ell_{0}(h,{\bm{x}}_{0j}) - \frac{1}{N_0} \sum_{j=i}^{N_0} \ell_{0}(h,{\bm{x}}_{0j}')\right] \right)\\ &\leq \sup_{h\in{\mathcal{H}}} \left( \mathbb{E}_{({\bm{x}}_{ri}',t_{ti}') \sim {\mathcal{D}}_1 \forall i \in [N_r]} \left[ \frac{1}{N_r} \sum_{i=i}^{N_r} \ell_{res}(h,({\bm{x}}_{ri},t_{ri})) - \frac{1}{N_r} \sum_{i=i}^{N_r} \ell_{res}(h,({\bm{x}}_{ri}',t_{ri}')) \right] \right) \\ &\hspace{17.0em} + \sup_{h\in{\mathcal{H}}} \left( \mathbb{E}_{{\bm{x}}_{0j}' \sim {\mathcal{D}}_2 \forall j\in[N_0]} \left[\frac{1}{N_0} \sum_{j=i}^{N_0} \ell_{0}(h,{\bm{x}}_{0j}) - \frac{1}{N_0} \sum_{j=i}^{N_0} \ell_{0}(h,{\bm{x}}_{0j}')\right] \right)\\ &\leq \mathbb{E}_{({\bm{x}}_{ri}',t_{ti}') \sim {\mathcal{D}}_1 \forall i \in [N_r]} \left[ \sup_{h\in{\mathcal{H}}} \left(\frac{1}{N_r} \sum_{i=i}^{N_r} \ell_{res}(h,({\bm{x}}_{ri},t_{ri})) - \frac{1}{N_r} \sum_{i=i}^{N_r} \ell_{res}(h,({\bm{x}}_{ri}',t_{ri}'))\right) \right] \\ &\hspace{17.0em} + \mathbb{E}_{{\bm{x}}_{0j}' \sim {\mathcal{D}}_2 \forall j\in[N_0]} \left[ \sup_{h\in{\mathcal{H}}} \left( \frac{1}{N_0} \sum_{j=i}^{N_0} \ell_{0}(h,{\bm{x}}_{0j}) - \frac{1}{N_0} \sum_{j=i}^{N_0} \ell_{0}(h,{\bm{x}}_{0j}') \right)\right] \end{aligned}

Now taking the expectation over ({\bm{x}}_{ri}, t_{ri}) and x_{0i} on both sides we get

\begin{aligned} &\mathbb{E}_{\substack{({\bm{x}}_{ri},t_{ri}) \sim {\mathcal{D}}_1 \forall i \in [N_r] \\ {\bm{x}}_{0j} \sim {\mathcal{D}}_2 \forall j\in[N_0]}} \left[\sup_{h\in{\mathcal{H}}} \left( \frac{1}{N_r} \sum_{i=i}^{N_r} \ell_{res}(h,({\bm{x}}_{ri},t_{ri})) + \frac{1}{N_0} \sum_{j=i}^{N_0} \ell_{0}(h,{\bm{x}}_{0j}) - \left( \mathbb{E}_{(x_r,t_r) \sim {\mathcal{D}}_1}[\ell_{res}(h,(x_{r},t_{r}))] + \mathbb{E}_{{\bm{x}}_0 \sim {\mathcal{D}}_2}[\ell_{0}(h,x_{0})] \right) \right)\right]\\ &\leq \mathbb{E}_{({\bm{x}}_{ri},t_{ri}) \sim D_1 \forall i \in [N_r]}\left[ \mathbb{E}_{({\bm{x}}_{ri}',t_{ti}') \sim {\mathcal{D}}_1 \forall i \in [N_r]} \left[ \sup_{h\in{\mathcal{H}}} \left(\frac{1}{N_r} \sum_{i=i}^{N_r} (\ell_{res}(h,({\bm{x}}_{ri},t_{ri})) - \ell_{res}(h,({\bm{x}}_{ri}',t_{ri}')))\right) \right] \right]\\ &\hspace{15.0em} + \mathbb{E}_{{\bm{x}}_{0j} \sim {\mathcal{D}}_2 \forall j\in[N_0]} \left[\mathbb{E}_{{\bm{x}}_{0j}' \sim {\mathcal{D}}_2 \forall j\in[N_0]} \left[ \sup_{h\in{\mathcal{H}}} \left( \frac{1}{N_0} \sum_{j=i}^{N_0} (\ell_{0}(h,{\bm{x}}_{0j}) - \ell_{0}(h,{\bm{x}}_{0j}')) \right)\right]\right]\\ &= \mathbb{E}_{\substack{({\bm{x}}_{ri},t_{ri}) \sim D_1 \forall i \in [N_r] \\ ({\bm{x}}_{ri}',t_{ti}') \sim {\mathcal{D}}_1 \forall i \in [N_r]}}\left[ \mathbb{E}_{\epsilon_i\sim \{\pm 1\}} \left[ \sup_{h\in{\mathcal{H}}} \left(\frac{1}{N_r} \sum_{i=i}^{N_r} \epsilon_i(\ell_{res}(h,({\bm{x}}_{ri},t_{ri})) - \ell_{res}(h,({\bm{x}}_{ri}',t_{ri}')))\right) \right] \right]\\ &\hspace{15.0em} + \mathbb{E}_{\substack{{\bm{x}}_{0j} \sim {\mathcal{D}}_2 \forall j\in[N_0] \\ {\bm{x}}_{0j}' \sim {\mathcal{D}}_2 \forall j\in[N_0]}} \left[\mathbb{E}_{\epsilon_j \sim \{\pm 1\}} \left[ \sup_{h\in{\mathcal{H}}} \left( \frac{1}{N_0} \sum_{j=i}^{N_0} \epsilon_j (\ell_{0}(h,{\bm{x}}_{0j}) - \ell_{0}(h,{\bm{x}}_{0j}')) \right)\right]\right]\\ &\leq \mathbb{E}_{\substack{({\bm{x}}_{ri},t_{ri}) \sim D_1 \forall i \in [N_r] \\ ({\bm{x}}_{ri}',t_{ti}') \sim {\mathcal{D}}_1 \forall i \in [N_r]}}\left[ \mathbb{E}_{\epsilon_i\sim \{\pm 1\}} \left[ \sup_{h\in{\mathcal{H}}} \left(\frac{1}{N_r} \sum_{i=i}^{N_r} \epsilon_i\ell_{res}(h,({\bm{x}}_{ri},t_{ri})) \right) + \sup_{h\in{\mathcal{H}}} \left( \frac{1}{N_r} \sum_{i=i}^{N_r} - \epsilon_i\ell_{res}(h,({\bm{x}}_{ri}',t_{ri}')) \right) \right] \right]\\ &\hspace{15.0em} + \mathbb{E}_{\substack{{\bm{x}}_{0j} \sim {\mathcal{D}}_2 \forall j\in[N_0] \\ {\bm{x}}_{0j}' \sim {\mathcal{D}}_2 \forall j\in[N_0]}} \left[\mathbb{E}_{\epsilon_j \sim \{\pm 1\}} \left[ \sup_{h\in{\mathcal{H}}} \left( \frac{1}{N_0} \sum_{j=i}^{N_0} \epsilon_j \ell_{0}(h,{\bm{x}}_{0j}) \right) + \sup_{h\in{\mathcal{H}}} \left( \frac{1}{N_0} \sum_{j=i}^{N_0} - \epsilon_j \ell_{0}(h,{\bm{x}}_{0j}') \right)\right]\right]\\ &= 2{\mathcal{R}}_{res} ({\mathcal{H}}) + 2{\mathcal{R}}_{0} ({\mathcal{H}}) \end{aligned}

Proof

CXZ0

The net’s full output is given by:

({\mathcal{N}}_{{\bm{w}},u}({\bm{z}}))_i=\sum_{j=1}^pa_{1ij}\sigma\left(\sum_{k=1}^dw_{jk}x_k+w_{j,d+1}t\right),

where {\bm{z}} = ({\bm{x}}, t). The first order partial derivatives of {\mathcal{N}}_{{\bm{w}}} are

\begin{aligned} \partial_{x_k} {\mathcal{N}}_{{\bm{w}},p}({\bm{z}}) &= \sum_{j=1}^p a_{2j} \sigma'\left(\sum_{q=1}^dw_{j,q}x_q+w_{j,d+1}t\right)w_{j,k}=\left\langle{\bm{a}}_2\odot{\bm{w}}_{x_k},\sigma'({\bm{W}}{\bm{z}})\right\rangle\\ (\partial_t{\mathcal{N}}_{{\bm{w}},{\bm{u}}}({\bm{z}}))_k&=\sum_{j=1}^pa_{1k,j}\sigma'\left(\sum_{q=1}^dw_{j,q}x_q+w_{j,d+1}t\right)w_{j,d+1}=\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_t,\sigma'({\bm{W}}{\bm{z}})\right\rangle\\ (\partial_{x_m}{\mathcal{N}}_{{\bm{w}},{\bm{u}}}({\bm{z}}))_k&=\sum_{j=1}^pa_{1k,j}\sigma'\left(\sum_{q=1}^dw_{j,q}x_q+w_{j,d+1}t\right)w_{j,m}=\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{x_m},\sigma'({\bm{W}}{\bm{z}})\right\rangle\\ \end{aligned}

Using these we can show the non-linear term can be expressed as

\begin{aligned} \nabla\cdot{\mathcal{N}}_{{\bm{w}},u}({\bm{z}})&=\sum_{m=1}^d\sum_{j=1}^pa_{1m,j}\sigma'\left(\sum_{q=1}^dw_{j,q}x_q+w_{j,d+1}t\right)w_{j,m}=\sum_{m=1}^d\left\langle{\bm{a}}_{1m}\odot{\bm{w}}_{x_m},\sigma'({\bm{W}}{\bm{z}})\right\rangle\\ \implies(({\mathcal{N}}_{{\bm{w}},{\bm{u}}}\cdot\nabla){\mathcal{N}}_{{\bm{w}},{\bm{u}}})_k&=\sum_{m=1}^d\left(\sum_{j=1}^pa_{1m,j}\sigma\left(\sum_{q=1}^dw_{j,q}x_q+w_{j,d+1}t\right)\right) \left(\sum_{j=1}^pa_{1k,j}\sigma'\left(\sum_{q=1}^dw_{j,q}x_q+w_{j,d+1}t\right)w_{j,m}\right)\\ &=\sum_{m=1}^d(\left\langle{\bm{a}}_{1m},\sigma({\bm{W}}{\bm{z}})\right\rangle)(\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{{\bm{x}}_m},\sigma'({\bm{W}}{\bm{z}})\right\rangle)\\ \end{aligned}

and that the Laplacian is

\begin{aligned} (\partial_{x_m}^2{\mathcal{N}}_{{\bm{w}},{\bm{u}}}({\bm{z}}))_k&=\sum_{j=1}^pa_{1k,j}\sigma''\left(\sum_{q=1}^dw_{j,q}x_q+w_{j,d+1}t\right)w_{j,m}^2=\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{x_m}\odot{\bm{w}}_{x_m},\sigma''({\bm{W}}{\bm{z}})\right\rangle\\ \implies(\nabla^2{\mathcal{N}}_{{\bm{w}},{\bm{u}}}({\bm{z}}))_k&=\sum_{m=1}^d\sum_{j=1}^pa_{1k,j}\sigma''\left(\sum_{q=1}^dw_{j,q}x_q+w_{j,d+1}t\right)w_{j,m}^2=\sum_{m=1}^d\left\langle{\bm{a}}_{1k}\odot{\bm{w}}_{x_m}\odot{\bm{w}}_{x_m},\sigma''({\bm{W}}{\bm{z}})\right\rangle \end{aligned}

Proof

GR61

\begin{aligned} \sup_{{\bm{w}}}\abs{\sum_{i=1}^n \epsilon_i f_{\bm{w}}({\bm{z}}_i)} &= \sup_{{\bm{w}}}\abs{\sum_{i=1}^n \epsilon_i (g_{\bm{w}}({\bm{z}}_i) + c)}\leq\sup_{\bm{w}} \left( \abs{\sum_{i=1}^n \epsilon_i g_{\bm{w}}({\bm{z}}_i)} + \abs{\sum_{i=1}^n c\epsilon_i}\right) = \sup_{\bm{w}} \left( \abs{\sum_{i=1}^n \epsilon_i g_{\bm{w}}({\bm{z}}_i)}\right) + \abs{\sum_{i=1}^n c\epsilon_i} \end{aligned}

Now, taking the expectation over {\bm{\epsilon}} on both sides we get,

\begin{aligned} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^n} \left[ \sup_{{\bm{w}}}\abs{\sum_{i=1}^n \epsilon_i f_{\bm{w}}({\bm{z}}_i)} \right] &\leq \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^n} \left[ \sup_{\bm{w}} \left( \abs{\sum_{i=1}^n \epsilon_i g_{\bm{w}}({\bm{z}}_i)}\right) \right] + \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^n} \left[ \abs{\sum_{i=1}^n c\epsilon_i} \right]\\ &\leq \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^n} \left[ \sup_{\bm{w}} \left( \abs{\sum_{i=1}^n \epsilon_i g_{\bm{w}}({\bm{z}}_i)}\right) \right] + \abs{c}\sqrt{n} \end{aligned}

Let \phi be the ReLU function, then the lemma’s assumption implies that \sup_{{\bm{w}}} \phi\left(\left\langle{\bm{\epsilon}},g_{{\bm{w}}}({\bm{Z}})\right\rangle\right)=\sup_{{\bm{w}}}\left\langle{\bm{\epsilon}},g_{{\bm{w}}}({\bm{Z}})\right\rangle for any {\bm{\epsilon}} \in\{\pm 1\}^{n}. Observing that |x|=\phi(x)+\phi(-x),

\begin{aligned} \sup_{{\bm{w}}}\abs{\left\langle{\bm{\epsilon}},g_{{\bm{w}}}({\bm{Z}})\right\rangle} &=\sup_{{\bm{w}}}\left[\phi\left(\left\langle{\bm{\epsilon}},g_{{\bm{w}}}({\bm{Z}})\right\rangle\right)+\phi\left(\left\langle-{\bm{\epsilon}},g_{{\bm{w}}}({\bm{Z}})\right\rangle\right)\right] \\ & \leq \sup _{{\bm{w}}} \phi\left(\left\langle{\bm{\epsilon}},g_{{\bm{w}}}({\bm{Z}})\right\rangle\right)+\sup _{{\bm{w}}} \phi\left(\left\langle-{\bm{\epsilon}},g_{{\bm{w}}}({\bm{Z}})\right\rangle\right) \\ &=\sup_{{\bm{w}}}\left\langle{\bm{\epsilon}},g_{{\bm{w}}}({\bm{Z}})\right\rangle + \sup_{{\bm{w}}}\left\langle-{\bm{\epsilon}},g_{{\bm{w}}}({\bm{Z}})\right\rangle. \end{aligned}

Taking the expectation over {\bm{\epsilon}} (and noting that {\bm{\epsilon}} and -{\bm{\epsilon}} have the same distribution) we get,

\begin{aligned} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^n} \left[ \sup_{\bm{w}} \abs{\sum_{i=1}^n \epsilon_i g_{\bm{w}}({\bm{z}}_i)} \right] \leq 2\mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^n} \left[ \sup_{\bm{w}} \left(\sum_{i=1}^n \epsilon_i g_{\bm{w}}({\bm{z}}_i)\right) \right] \end{aligned}

Replacing the above into equation Proof we get,

\begin{aligned} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^n} \left[ \sup_{{\bm{w}}}\abs{\sum_{i=1}^n \epsilon_i f_{\bm{w}}({\bm{z}}_i)} \right] &\leq 2\mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^n} \left[ \sup_{\bm{w}} \left( \sum_{i=1}^n \epsilon_i g_{\bm{w}}({\bm{z}}_i)\right) \right] + \abs{c}\sqrt{n} \end{aligned}

Proof

IAWM

\begin{aligned} &\mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \left( \frac{1}{N_r} \sum_{i=1}^{N_r} \epsilon_i \left\langlef({\bm{W}}),\phi({\bm{W}} {\bm{z}}_i)\right\rangle \right) \right] \\ &= \frac{1}{N_r} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \left( \sum_{i=1}^{N_r} \epsilon_i \sum_{m=1}^{p} f({\bm{W}})_m \phi(\left\langle{\bm{w}}_m,{\bm{z}}_i\right\rangle) \right) \right] \\ &= \frac{1}{N_r} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \left( \sum_{m=1}^{p} f({\bm{W}})_m \sum_{i=1}^{N_r} \epsilon_i \phi(\left\langle{\bm{w}}_m,{\bm{z}}_i\right\rangle) \right) \right] \\ &\leq \frac{1}{N_r} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \left( \sum_{m=1}^{p} \abs{f({\bm{W}})_m} \max_{k\in[p]} \abs{\sum_{i=1}^{N_r} \epsilon_i \phi(\left\langle{\bm{w}}_k,{\bm{z}}_i\right\rangle)} \right) \right] &&\left( \text{since } \sum_j \alpha_j \beta_j \leq \sum_j \abs{\alpha_j} \max_k \abs{\beta_k} \right)\\ &\leq \frac{B}{N_r} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \max_{k\in[p]} \abs{\sum_{i=1}^{N_r} \epsilon_i \phi(\left\langle{\bm{w}}_k,{\bm{z}}_i\right\rangle)} \right] &&\left( \text{since } \sum_{m=1}^{p} \abs{f({\bm{W}})_m} \leq B \right)\\ &= \frac{B}{N_r} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}} \left[ \sup_{\bar{{\bm{w}}}\in\textrm{RS}({\mathcal{C}})} \abs{\sum_{i=1}^{N_r} \epsilon_i \phi(\left\langle\bar{{\bm{w}}},{\bm{z}}_i\right\rangle)} \right] \end{aligned}

We then apply Lemma 5 with f_{\bar{{\bm{w}}}}({\bm{z}}_i) = \phi(\langle \bar{{\bm{w}}}, {\bm{z}}_i\rangle) and constant c = \phi_1(0), so that g_{\bar{{\bm{w}}}}({\bm{z}}_i) = \phi(\langle \bar{{\bm{w}}}, {\bm{z}}_i\rangle)-c vanishes at \bar{{\bm{w}}} = 0. The condition \sup_{\bar{{\bm{w}}}} \langle \epsilon, g_{\bar{{\bm{w}}}}({\bm{Z}})\rangle \geq 0 holds since the supremum is taken over the zero vector as a feasible point. Lemma 5 then gives:

\begin{aligned} &\leq \frac{B}{N_r} \left\{2\mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}} \left[ \sup_{\bar{{\bm{w}}}\in\textrm{RS}({\mathcal{C}})} \sum_{i=1}^{N_r} \epsilon_i g(\left\langle\bar{{\bm{w}}},{\bm{z}}_i\right\rangle) \right] + \abs{c}\sqrt{N_r}\right\} \end{aligned}

Now, \chi(\cdot) has the same Lipschitz constant as \phi(\cdot), we can apply Talagrand’s contraction lemma to get,

\begin{aligned} &\leq \frac{2B L_{\phi}}{N_r} \mathbb{E}_{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}} \left[ \sup_{\bar{{\bm{w}}}\in\textrm{RS}({\mathcal{C}})} \sum_{i=1}^{N_r} \epsilon_i \left\langle\bar{{\bm{w}}},{\bm{z}}_i\right\rangle \right] + \frac{B\abs{c}}{\sqrt{N_r}} \end{aligned}

Uses Lemma 5

Proof

7ATP

\begin{aligned} &\mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \left(\frac{1}{N_r} \sum_{i=1}^{N_r}\sum_{m=1}^d \epsilon_i \left\langlef_m({\bm{W}}),\phi_1 ({\bm{W}} {\bm{z}}_i)\right\rangle \left\langle{\bm{a}}_m,\phi_2({\bm{W}} {\bm{z}}_i)\right\rangle \right)\right] \\ &=\frac{1}{N_r} \mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \left( \sum_{i=1}^{N_r}\sum_{m=1}^d \epsilon_i \sum_{q_1,q_2 = 1}^p f_m({\bm{W}})_{q_1} \phi_1(\left\langle{\bm{w}}_{q_1},{\bm{z}}_i\right\rangle) {\bm{a}}_{m,q_2}\phi_2(\left\langle{\bm{w}}_{q_2},{\bm{z}}_i\right\rangle) \right)\right] \\ &=\frac{1}{N_r}\mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \left( \sum_{m=1}^d\sum_{q_1,q_2 = 1}^p f_m({\bm{W}})_{q_1} {\bm{a}}_{m,q_2} \sum_{i=1}^{N_r} \epsilon_i \phi_1(\left\langle{\bm{w}}_{q_1},{\bm{z}}_i\right\rangle)\phi_2(\left\langle{\bm{w}}_{q_2},{\bm{z}}_i\right\rangle) \right)\right] \end{aligned}

Using \sum_j \alpha_j \beta_j \leq \sum_j \abs{\alpha_j} \max_k \abs{\beta_k},

\begin{aligned} &\leq\frac{1}{N_r}\mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \left( \sum_{m=1}^d\sum_{q_1,q_2 = 1}^p \abs{f_m({\bm{W}})_{q_1}} \abs{{\bm{a}}_{m,q_2}} \max_{q_1,q_2 \in [p]} \abs{\sum_{i=1}^{N_r} \epsilon_i \phi_1(\left\langle{\bm{w}}_{q_1},{\bm{z}}_i\right\rangle)\phi_2(\left\langle{\bm{w}}_{q_2},{\bm{z}}_i\right\rangle)} \right)\right] \end{aligned}

Since \sum_{m=1}^d\sum_{q_1,q_2 = 1}^p \abs{f_m({\bm{W}})_{q_1}} \abs{{\bm{a}}_{m,q_2}} \leq B,

\begin{aligned} &\leq\frac{B}{N_r}\mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[ \sup_{{\bm{W}}\in{\mathcal{C}}} \max_{q_1,q_2 \in [p]} \abs{\sum_{i=1}^{N_r} \epsilon_i \phi_1(\left\langle{\bm{w}}_{q_1},{\bm{z}}_i\right\rangle)\phi_2(\left\langle{\bm{w}}_{q_2},{\bm{z}}_i\right\rangle)} \right]\\ &= \frac{B}{N_r}\mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[ \sup_{\bar{{\bm{w}}_1},\bar{{\bm{w}}_2}\in\textrm{RS}({\mathcal{C}})} \abs{\sum_{i=1}^{N_r} \epsilon_i \phi_1(\left\langle\bar{{\bm{w}}}_1,{\bm{z}}_i\right\rangle)\phi_2(\left\langle\bar{{\bm{w}}}_2,{\bm{z}}_i\right\rangle)} \right] \end{aligned}

We then apply Lemma 5 with f_{\bm{w}}({\bm{z}}_i) = \phi_1(\langle {\bm{w}}_1, {\bm{z}}_i\rangle)\phi_2(\langle {\bm{w}}_2, {\bm{z}}_i\rangle) and constant c = \phi_1(0)\cdot\phi_2(0) = k, so that g_{\bm{w}}(z_i) = \phi_1(\langle {\bm{w}}_1, {\bm{z}}_i\rangle)\phi_2(\langle {\bm{w}}_2, {\bm{z}}_i\rangle) - k vanishes at {\bm{w}} = 0. The condition \sup_{\bm{w}} \langle \epsilon, g_{\bm{w}}({\bm{Z}})\rangle \geq 0 holds since the supremum is taken over the zero vector as a feasible point. Lemma 5 then gives:

\begin{aligned} &\leq\frac{B}{N_r} \left\{2\mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[ \sup_{\bar{{\bm{w}}}_1\bar{{\bm{w}}}_2\in\textrm{RS}({\mathcal{C}})} \sum_{i=1}^{N_r} \epsilon_i (\phi_1(\left\langle\bar{{\bm{w}}}_{1},{\bm{z}}_i\right\rangle)\phi_2(\left\langle\bar{{\bm{w}}}_2,{\bm{z}}_i\right\rangle) - k)\right] + \abs{k}\sqrt{N_r}\right\}\\ &=\frac{B}{N_r} \left\{2\mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[ \sup_{\bar{{\bm{w}}}_1\bar{{\bm{w}}}_2\in\textrm{RS}({\mathcal{C}})} \sum_{i=1}^{N_r} \epsilon_i \phi_1(\left\langle\bar{{\bm{w}}}_{1},{\bm{z}}_i\right\rangle)\phi_2(\left\langle\bar{{\bm{w}}}_2,{\bm{z}}_i\right\rangle)\right] -2\mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[\sum_{i=1}^{N_r} \epsilon_i k\right]+ \abs{k}\sqrt{N_r}\right\}\\ &=\frac{B}{N_r} \left\{2\mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[ \sup_{\bar{{\bm{w}}}_1\bar{{\bm{w}}}_2\in\textrm{RS}({\mathcal{C}})} \sum_{i=1}^{N_r} \epsilon_i \phi_1(\left\langle\bar{{\bm{w}}}_{1},{\bm{z}}_i\right\rangle)\phi_2(\left\langle\bar{{\bm{w}}}_2,{\bm{z}}_i\right\rangle)\right] + \abs{k}\sqrt{N_r}\right\} \end{aligned}

where the last step comes from the fact that \mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[\sum_{i=1}^{N_r} \epsilon_i \right]=\sum_{i=1}^{N_r}\mathbb{E}_{\substack{{\bm{\epsilon}} \sim \{\pm 1\}^{N_r}}} \left[ \epsilon_i \right]=0.

It remains to bound the Rademacher complexity of the function class \mathcal{G} = \{\psi_{{\bm{w}}_1,{\bm{w}}_2} : {\bm{w}}_1, {\bm{w}}_2 \in \mathrm{RS}(C)\}, where \psi_{{\bm{w}}_1,{\bm{w}}_2}(z) := \phi_1(\langle {\bm{w}}_1, z\rangle)\phi_2(\langle {\bm{w}}_2, z\rangle). We proceed in two sequential contraction steps, freezing one supremum variable at a time.

\bullet (Splitting across a reference weight {{\bm{w}}}_2^*) Fix any reference vector {\bm{w}}_2^* \in \mathrm{RS}(C). For any fixed \bar{{\bm{w}}}_1, the scalars \phi_1(\langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle) are deterministic, and since |\phi_1(\cdot)| \leq B_{\phi_1}, we have:

\begin{aligned} &\mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}}_1, \bar{{\bm{w}}}_2 \in \mathrm{RS}(C)} \sum_{i=1}^{N_r} \epsilon_i\, \phi_1(\langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle)\phi_2(\langle \bar{{\bm{w}}}_2, {\bm{z}}_i\rangle)\right] \\ &\leq \mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}}_1 \in \mathrm{RS}(C)}\left\{ \sup_{\bar{{\bm{w}}}_2 \in \mathrm{RS}(C)} \sum_{i=1}^{N_r} \epsilon_i\, \phi_1(\langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle) \left[\phi_2(\langle \bar{{\bm{w}}}_2, {\bm{z}}_i\rangle) - \phi_2(\langle {\bm{w}}_2^*, {\bm{z}}_i\rangle)\right] + \sum_{i=1}^{N_r} \epsilon_i\, \phi_1(\langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle)\phi_2(\langle {\bm{w}}_2^*, {\bm{z}}_i\rangle)\right\}\right] \\ &\leq \mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}}_1, \bar{{\bm{w}}}_2 \in \mathrm{RS}(C)} \sum_{i=1}^{N_r} \epsilon_i\, \phi_1(\langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle) \left[\phi_2(\langle \bar{{\bm{w}}}_2, {\bm{z}}_i\rangle) - \phi_2(\langle {\bm{w}}_2^*, {\bm{z}}_i\rangle)\right]\right] + \mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}}_1 \in \mathrm{RS}(C)} \sum_{i=1}^{N_r} \epsilon_i\, \phi_1(\langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle)\phi_2(\langle {\bm{w}}_2^*, {\bm{z}}_i\rangle)\right]. \end{aligned}

For the first term in (?), since |\phi_1(\langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle)| \leq B_{\phi_1} we get,

We apply Lemma 5 with f_{\bm{w}}({\bm{z}}_i) = \phi_2(\langle \bar{{\bm{w}}}_2, {\bm{z}}_i\rangle) - \phi_2(\langle {\bm{w}}_2^*, {\bm{z}}_i\rangle) and constant c = 0, so that g_w({\bm{z}}_i) = \phi_2(\langle \bar{{\bm{w}}}_2, {\bm{z}}_i\rangle) - \phi_2(\langle {\bm{w}}_2^*, {\bm{z}}_i\rangle) vanishes at \bar{{\bm{w}}}_2 = {\bm{w}}_2^*. The condition \sup_{\bm{w}} \langle \epsilon, g_w({\bm{Z}})\rangle \geq 0 holds since the supremum is taken over the zero vector as a feasible point. Therefore, for the first term in the RHS of (?) we get,

\begin{aligned} \mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}}_1, \bar{{\bm{w}}}_2 \in \mathrm{RS}(C)} \sum_{i=1}^{N_r} \epsilon_i\, \phi_1(\langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle) \left[\phi_2(\langle \bar{{\bm{w}}}_2, {\bm{z}}_i\rangle) - \phi_2(\langle {\bm{w}}_2^*, {\bm{z}}_i\rangle)\right]\right] &\leq 2 B_{\phi_1}\, \mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}}_2 \in \mathrm{RS}(C)} \sum_{i=1}^{N_r} \epsilon_i \left[\phi_2(\langle \bar{{\bm{w}}}_2, {\bm{z}}_i\rangle) - \phi_2(\langle {\bm{w}}_2^*, {\bm{z}}_i\rangle)\right]\right]\\ &\leq 2B_{\phi_1} L_{\phi_2}\, \mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}} \in \mathrm{RS}(C)} \sum_{i=1}^{N_r} \epsilon_i \langle \bar{{\bm{w}}}, {\bm{z}}_i\rangle\right], \end{aligned}

where the last step uses Talagrand’s contraction on \phi_2(\langle \bar{{\bm{w}}}_2, {\bm{z}}_i\rangle) - \phi_2(\langle {\bm{w}}_2^*, {\bm{z}}_i\rangle), which is L_{\phi_2}-Lipschitz in \langle \bar{{\bm{w}}}_2, {\bm{z}}_i\rangle.

\bullet (Contract over \bar{{\bm{w}}}_1 with {\bm{w}}_2^* fixed) We now handle the second term in (?). Since {\bm{w}}_2^* is a fixed reference vector, \phi_2(\langle {\bm{w}}_2^*, {\bm{z}}_i\rangle) is a fixed scalar for each i. Using |\phi_2(\langle {\bm{w}}_2^*, {\bm{z}}_i\rangle)| \leq B_{\phi_2}:

\begin{aligned} &\mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}}_1 \in \mathrm{RS}(C)} \sum_{i=1}^{N_r} \epsilon_i\, \phi_1(\langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle)\phi_2(\langle {\bm{w}}_2^*, {\bm{z}}_i\rangle)\right]\leq \mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}}_1 \in \mathrm{RS}(C)} \abs{\sum_{i=1}^{N_r} \epsilon_i\, \phi_1(\langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle)\phi_2(\langle {\bm{w}}_2^*, {\bm{z}}_i\rangle)}\right]\\ &\leq B_{\phi_2}\, \mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}}_1 \in \mathrm{RS}(C)} \left|\sum_{i=1}^{N_r} \epsilon_i\, \phi_1(\langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle)\right|\right]. \end{aligned}

We apply Lemma 5 with f_{\bm{w}}({\bm{z}}_i) = \phi_1(\langle {\bm{w}}, {\bm{z}}_i\rangle) and constant c = \phi_1(0) = c_0, so that g_{\bm{w}}({\bm{z}}_i) = \phi_1(\langle {\bm{w}}, {\bm{z}}_i\rangle) - c_0 vanishes at {\bm{w}} = 0. The condition \sup_w \langle \epsilon, g_{\bm{w}}(Z)\rangle \geq 0 holds since the supremum is taken over the zero vector as a feasible point. Hence for the second term in the RHS of (?) we get,

\begin{aligned} \mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}}_1 \in \mathrm{RS}(C)} \sum_{i=1}^{N_r} \epsilon_i\, \phi_1(\langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle)\phi_2(\langle {\bm{w}}_2^*, {\bm{z}}_i\rangle)\right] \leq 2 B_{\phi_2}\,\mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}}_1 \in \mathrm{RS}(C)} \sum_{i=1}^{N_r} \epsilon_i \left[\phi_1(\langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle) - c_0\right]\right] + B_{\phi_2}|c_0|\sqrt{N_r}. \end{aligned}

Now \phi_1(\langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle) - c_0 is an L_{\phi_1}-Lipschitz function of \langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle, so Talagrand’s contraction applies to give,

\begin{aligned} \mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}}_1 \in \mathrm{RS}(C)} \sum_{i=1}^{N_r} \epsilon_i\, \phi_1(\langle \bar{{\bm{w}}}_1, {\bm{z}}_i\rangle)\phi_2(\langle {\bm{w}}_2^*, {\bm{z}}_i\rangle)\right] \leq 2 B_{\phi_2} L_{\phi_1}\,\mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}} \in \mathrm{RS}(C)} \sum_{i=1}^{N_r} \epsilon_i \langle \bar{{\bm{w}}}, {\bm{z}}_i\rangle\right] + B_{\phi_2}|c_0|\sqrt{N_r}. \end{aligned}

\bullet (Combining) Substituting into (?) the end results of the above two analysis blocks we get,

Substituting the above back into (?) we get the desired bound,

\begin{aligned} &\mathbb{E}_{\epsilon \sim \{\pm 1\}^{N_r}}\left[\sup_{W \in C} \left(\frac{1}{N_r}\sum_{i=1}^{N_r} \sum_{m=1}^{d} \epsilon_i \langle f_m(W), \phi_1(W{\bm{z}}_i)\rangle \langle a_m, \phi_2(W{\bm{z}}_i)\rangle \right)\right] \\ &\leq \frac{B}{N_r}\left\{4(B_{\phi_1} L_{\phi_2} + B_{\phi_2} L_{\phi_1})\, \mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}} \in \mathrm{RS}(C)} \sum_{i=1}^{N_r} \epsilon_i \langle \bar{{\bm{w}}}, {\bm{z}}_i\rangle\right] + 2 B_{\phi_2}|c_0|\sqrt{N_r} + |k|\sqrt{N_r}\right\}\\ &=\frac{4B(B_{\phi_1} L_{\phi_2} + B_{\phi_2} L_{\phi_1})}{N_r}\, \mathbb{E}_{\epsilon}\left[\sup_{\bar{{\bm{w}}} \in \mathrm{RS}(C)} \sum_{i=1}^{N_r} \epsilon_i \langle \bar{{\bm{w}}}, {\bm{z}}_i\rangle\right] + \frac{B(2B_{\phi_2}|c_0| + |k|)}{\sqrt{N_r}}. \end{aligned}

Uses Lemma 5

Source and licence

The text on this page is from arXiv:2603.23072v1 by Sebastien Andre-Sloan, Dibyakanti Kumar, Alejandro F Frangi and Anirbit Mukherjee, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.

The record of each item is in analysis/objects/2603/2603.23072.jsonl.