Bayesian Interpolation with Deep Linear Networks

Fix P,L,N_0,\ldots, N_L\geq 1,\, \sigma^2>0 as well as training data X_{N_0},Y_{N_0}. Fix {\bf t}\in \mathbb R^{N_0} and write

Define 4M = \prod_{\ell=0}^L 2\sigma^2/N_\ell

and introduce the following shorthand for the Meijer-G functions (see § ?) with parameters given by layer widths:

The partition function Z_\infty({\bf t}) of the predictive posterior defined in (?) is

In particular, the Bayesian model evidence Z_\infty({\bf 0}) equals

Further, given x\in \mathbb R^{N_0}, the mean of the predictive posterior is

while the posterior variance \mathrm{Var}_{\mathrm{post}}\left[f(x)\right] is

where x_\perp is the projection of x onto the orthogonal complement of the span \mathrm{col}(X_{N_0}) of the training data.

Set N_1,\ldots,N_L = N and define \mathbf{N} = (N_1, \dots, N_L). Suppose that the training data satisfies (?). In different limiting cases such that \{P, N\} \to \infty with fixed P/N_0 = \alpha_0, we evaluate the quantities

We will see in each case that to leading order \log G does not depend on k while \Delta(\log G)[k] does.

1. Fix L < \infty,\, \alpha,\sigma^2>0. Suppose P,N\rightarrow \infty with P/N\rightarrow \alpha. Then,

   \begin{aligned} \log G &= \frac{N\alpha}{2}\left[\log\left(\frac{N\alpha}{2}\right) + \log\left(1+\frac{z_*}{\alpha}\right) - \left(1+\frac{z_*}{\alpha}\right)\right] \\ &\quad + \frac{NL}{2}\left[\log\left(\frac{N}{2}\right) + \log\left(1+z_*\right) - (1+z_*)\right] \end{aligned}

   plus an error of size \tilde O(1) and

   \begin{aligned} \Delta(\log G)[k] &= kL\left[\log\left(\frac{N}{2}\right) + \log(1+z_*)\right], \end{aligned}

   plus an error of size \tilde O\left(1/N\right), where z_*> \min\left\{-\alpha,-1\right\} is the unique solution to

   \left(1+\frac{z_*}{\alpha}\right)\left(1+z_*\right)^L = \frac{\left|\left|\theta_*\right|\right|^2}{\sigma^{2(L+1)}\alpha_0}.

2. Fix \lambda_{\mathrm{prior}},\alpha>0 and suppose L=\lambda_{\mathrm{prior}} N, \, P,N\rightarrow \infty,\, P/N\rightarrow \alpha, \, \sigma^2=1. Then,

   \begin{aligned} \log G &= \frac{\lambda_{\mathrm{prior}} N^2}{2}\left[\log\left(\frac{N}{2}\right) - 1\right] + \frac{N\alpha}{2}\left[\log\left(\frac{N\alpha}{2}\right) - 1\right]\\ &\quad + \frac{\lambda_{\mathrm{prior}} N}{2}\left[-\log\left(\frac{N}{2}\right) + \log(2\pi)\right] + \tilde O(1) \end{aligned}

   and, up to an error of size \tilde O\left(1/N\right),

   \begin{aligned} \Delta(\log G)[k] &= k\left[\lambda_{\mathrm{prior}} N \log\left(\frac{N}{2}\right) + \log \left(\frac{\left|\left|\theta_*\right|\right|^2}{\alpha_0}\right)\right]. \end{aligned}

3. Fix \lambda_{\mathrm{post}} >0. Suppose N,P,L\rightarrow \infty with LP/N\rightarrow \lambda_{\mathrm{post}}, \, \sigma^2=1 and L/N\to 0. Then,

   \begin{aligned} \log G &= \frac{P}{2}\left[\log\left(\frac{P}{2}\right)-1\right] + \frac{\lambda_{\mathrm{post}} N}{2}\frac{N}{P}\left[\log\left(\frac{N}{2}\right)-1\right] \\ &\quad + \frac{P}{2}\left[\log(1+t_*)-t_*\left(1+\frac{\lambda_{\mathrm{post}} t_*}{2}\right)\right] \\ &\quad + \frac{\lambda_{\mathrm{post}} N}{2}\left[\frac{N}{P}\left(1+\frac{P}{N}t_*\right)\log\left(1+\frac{P}{N}t_*\right) - t_*\right] \\ &\quad + \tilde O(1), \end{aligned}

   and

   \begin{aligned} \Delta(\log G)[k] &= k\left[L \log \left(\frac{N}{2}\right) + \lambda_{\mathrm{post}} t_*\right] + \tilde{O}\left(\frac{1}{N}\right), \end{aligned}

   where t_* is the unique solution to

   \begin{aligned} e^{\lambda_{\mathrm{post}} t_*}(1+t_*) &= \left|\left|\theta_*\right|\right|^2/\alpha_0. \end{aligned}

In all the estimates above, \tilde O(1) = O(\max\{\log P, \log N\}) suppresses lower-order terms of order 1, up to logarithmic factors; similarly, \tilde O(1/N) = O(\max\{\log P, \log N\}/N). Suppressed terms are included in § ?

Fix L, N_1\dots, N_L \geq 1, \sigma^2 > 0, \alpha_0\in (0,1), and consider sequences of training data sets X_{N_0}\in \mathbb R^{N_0\times P}, Y_{N_0}\in \mathbb R^{1\times P} such that

and (?) holds. Let Z_\infty({\bf 0}) denote the limiting Bayesian model evidence. The following statements are equivalent:

1. \sigma^2 maximizes Z_\infty({\bf 0}), and

2. the posterior over predictions f(x) converges weakly to a Gaussian \mathcal{N}\left(\mu_*, \nu \Sigma_\perp\right).

For each N_0,P\geq 1, consider training data X_{N_0},Y_{N_0} satisfying (?). Fix constants L\geq 0,\,\sigma^2 > 0 and suppose that

In this limit, the posterior over predictions f(x) converges weakly to a Gaussian

where z_* is the unique solution to

Additionally, we have the following large-P expansion of the Bayesian model evidence

In the setting of Theorem 3.4 the Bayesian evidence Z_\infty({\bf 0}) satisfies:

In particular, given a prior variance with \mathrm{sgn}\left(\nu-1\right)=\mathrm{sgn}\left(\sigma^2-1\right) satisfying |\sigma^2-1| \leq \epsilon, the optimal depth network satisfies L_* \geq |\log(\nu)|/\epsilon.

For each N_0,P\geq 1, consider training data X_{N_0},Y_{N_0} satisfying (?). Moreover, fix \lambda_{\mathrm{prior}},\alpha \in (0,\infty),\, \alpha_0\in (0,1). Suppose that N_1=\cdots=N_L=N and that

In this limit, the posterior over predictions f(x) converges weakly to a Gaussian

which is independent of \alpha,\lambda_{\mathrm{prior}}. Additionally, the evidence admits the following asymptotic expansion:

In the setting of Theorem 3.6, the posterior distribution over regression weights \theta is the same in the following two settings:

• We fix L\geq 0, take the data-dependent prior variance \sigma_* that maximizes the Bayesian model evidence as a function of the training data and network depth as in (3.5), and send the network width N to infinity.

• We fix \lambda_{\mathrm{prior}} >0, take a data-agnostic prior \sigma^2=1, and send both the network depth L:=\lambda_{\mathrm{prior}} \cdot N and network width N to infinity together.

For each N_0,P\geq 1, consider training data X_{N_0},Y_{N_0} satisfying (?). Moreover, fix constants \lambda_{\mathrm{post}} > 0,\, \alpha_0\in (0,1), \, \sigma^2=1. Suppose that N_1=\cdots = N_L=N and suppose that

In this limit, the posterior over predictions f(x) converges weakly to a Gaussian \mathcal{N}\left(\mu_*,~\nu\Sigma_\perp(1+t_*)^{-1}\right),

where t_* is the unique solution to

Additionally, we have the following asymptotic expansion for the Bayesian model evidence

In the setting of Theorem 3.8, the Bayesian evidence Z_\infty({\bf 0}) is monotonically increasing in \lambda_{\mathrm{post}}. Specifically,

For each N_0\geq 1 consider training data X_{N_0},Y_{N_0} satisfying (?). Fix constants \lambda_{\mathrm{prior}} > 0,\alpha>0, \, \sigma^2=1. Suppose that N_1=\cdots=N_L= N, that L=\lambda_{\mathrm{prior}} N and number of data points satisfies P=N\alpha. Then, as N\rightarrow \infty,

where C\in \mathbb R is a universal constant.

Consider generative data model

and posterior distribution \mathcal{N}\left(\mu_*, \nu\Sigma_\perp\right). We have error

which diverges at \alpha_0 = 1.

We have the following analytic asymptotic expansion for the Gamma function

In particular, we also have

Both expansions hold uniformly on sets of the form \left|z\right|< \pi - \delta for a fixed \delta >0.

Let k,\theta >0 and suppose \phi \sim \Gamma(k,\theta). Then for any t\in \mathbb R we have

In particular

is a meromorphic function of t with poles on the negative imaginary axis:

For \eta,\omega>0, We have

For (4.3) we require

and

Finally, fix L\geq 1 and

and let

We have for every y>0

For each N_0, consider a collection of k\geq 1 test points

Suppose that

• For each \alpha_0\in (0,1) there exists a vector \mu_{*}=\left(\mu_{*,j},\, j=1,\ldots, k\right)\in \mathbb R^k such that

  \lim_{\substack{P,N_0\rightarrow \infty \\ P/N_0\rightarrow \alpha_0\in (0,1)}}\left \langle \theta_{*,N_0},x_{j,N_0}\right\rangle = \mu_{*,j}\qquad \text{ almost surely}.

• For each \alpha_0\in (0,1) there exists a positive semi-definite k\times k matrix \Sigma_\perp such that

  \lim_{\substack{P,N_0\rightarrow \infty \\ P/N_0\rightarrow \alpha_0\in (0,1)}} \left(\frac{1}{N_0-P}\left \langle x_{i,N_0}^\perp,x_{j,N_0}^\perp\right\rangle \right)_{1\leq i,j\leq k} = \Sigma_\perp\qquad \text{ almost surely},

  where x_{j,N_0}^\perp denotes the projection of x_j onto the orthogonal complement of the column space of X_{N_0}.

Assume that the convergence of the posterior for \left|\left|\theta_\perp\right|\right|^2 in the large-N limit satisfies

for some constant c. Then the distribution over posterior predictions evaluated on {\bf x}_{N_0}

converges weakly to a k-dimensional Gaussian:

For any \beta > 0, the partition function Z_\beta(x) equals

where X^{\dagger} is the pseudo-inverse of X and

Fix L, N, N_0,\ldots, N_L\geq 1 as well as \left|\left|\theta_*\right|\right|>0 and define M by

For any N\geq 1 we have

where \mathcal C\subseteq\mathbb C is the contour that runs along the real line from -\infty to \infty and

There exist c,T_0>0 such that

Moreover, for any T,\delta>0, writing

there exists C>0 such that

Fix \delta, T>0 and define S_{N,\delta, T} as in (4.9). Then, uniformly over S_{N,\delta, T}, we have

where

There exist c,T_0>0 such that

Moreover, for any T,\delta>0, writing

there exists C>0 such that

As in the setting of Theorem 6, fix \lambda_{\mathrm{prior}}>0. For each fixed L\geq 0

where

satisfies c \leq 0, and z_* is defined as in (?) of Theorem 3.2, i.e.,

In the setting of Theorem 6, we have

Moreover, for any \lambda>0, there exists c>0 such that we have

i.e., the ratio of evidences is lower-bounded by a constant.