On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case

t follows from (?) – (?) and (?) that

Now the first and the third relations (I) yield m_{K}(-h(z^{-1}))=m_{\mathcal{M}}(z), hence z_{K}(m)=-h(z_{\mathcal{M} }^{-1}(m)), and then the second and the third relations yield m_{R}(k(z^{-1})z)=m_{\mathcal{M}}(z), hence z_{R}(\mu )=k(z_{\mathcal{M} }^{-1}(m))z_{\mathcal{M}}(m). Multiplying these two relations and using once more the third relation in (I), we obtain

and (2.3) – (2.3) follows.

e prove the theorem by induction in L. We have from (?) – (?) and (?) with L=1 the following n\times n random matrix

It is convenient to pass from M_{n}^{1} to the n\times n matrix (see Remark 3.2)

which has the same spectrum, hence the same Normalized Counting Measure as M_{n}^{1}. The matrix \mathcal{M}_{n}^{1} is a particular case with S_{n}=\mathbf{1}_{n} of matrix (2.1) treated in Theorem 2.1 above. Since the NCM of \mathbf{1}_{n} is the Dirac measure \delta _{1}, conditions (?) – (?) of the theorem are evident. Conditions ( ?) and (?) of the theorem are just (2.5) and (2.5). It follows then from Corollary 2.3 that the assertion of our theorem, i.e., formula (2.5) with q^{1} of (2.5) is valid for L=1.

Consider now the case where L=2 of (?) – (?) and (?):

Since M_{n}^{1} is positive definite, we write

with a positive definite S_{n}^{1}, hence,

and the corresponding \mathcal{M}_{n}^{2} is (see Remark 3.2)

We observe that \mathcal{M}_{n}^{2} is a particular case of matrix (2.1) of Theorem 2.1 with M_{n}^{1} of (?) as R_{n}=(S_{n})^{2}, X_{n}^{2} as X_{n}, K_{n}^{2} as K_{n}, \{x_{j_{1}}^{1}\}_{j_{1}=1}^{n} as \{x_{\alpha n}\}_{\alpha =1}^{n}, \Omega _{1}=\Omega ^{1} of (?) as \Omega _{R,x} and \Omega ^{2} of (?) as \Omega _{X,b}, i.e., the case of the random but \{X_{n}^{2},b^{2}_{n}\} -independent R_{n} and \{x_{\alpha n}\}_{\alpha =1}^{n} in (2.1) as described in Remark 2.2 (i). Let us check that conditions (?) – (?) and (?) – (?) of Theorem 2.1 are satisfied for \mathcal{M}_{n}^{2} of ( ?) with probability 1 in the probability space \Omega _{1}=\Omega^1 generated by \{X_{n}^{1},b_{n}^{1}\} for all n and independent of the space \Omega ^{2} generated by \{X_{n}^{2},b_{n}^{2}\} for all n.

To this end we use an important fact on the operator norm of n\times n random matrices with i.i.d. entries satisfying (?). Namely, if X_{n} is a such n\times n matrix, then we have with probability 1

thus, with the same probability

if n_0 is large enough.

For the Gaussian matrices relation (?) has already been known in the Wigner school of the early 1960th, see [Pa-Sh:11]. It follows in this case from the orthogonal polynomial representation of the density of the NCM of n^{-1}X_{n}X_{n}^{T} and the asymptotic formula for the corresponding orthogonal polynomials. For the modern form of (?), in particular its validity for random matrices with i.i.d entries of zero mean and finite fourth moment, see [Ba-Si:00, Ve:18] and references therein.

We will also need the bound

following from (?), (?) and (2.5) and valid everywhere in \Omega _{1} of (?).

Now, by using (W), (?), (?) and the inequality

valid for any matrix A and a positive definite matrix B, we obtain with probability 1 in \Omega _{1} and for sufficiently large n_{0} of (?)

We conclude that M_{n}^{1}, which plays here the role of R_{n} of Theorem 2.1 and Remark 2.2 (i) according to (?), satisfies condition (?) with r_{2}=(C\Phi _{1})^{4} and with probability 1 in our case, i.e., on a certain \Omega _{11}\subset \Omega _{1},\;\mathbf{P}(\Omega _{11})=1.

Next, it follows from the above proof of the theorem for L=1, i.e., in fact, from Theorem 2.1, that there exists \Omega _{12}\subset \Omega _{1},\;\mathbf{P}(\Omega _{12})=1 on which the NCM \nu _{M_{n}^{1}} converges weakly to a non-random limit \nu _{M^{1}}, hence condition (?) is also satisfied with probability 1, i.e., on \Omega _{12}.

At last, according to Lemma 3.6 and (2.5), there exists \Omega _{13}\subset \Omega _{1},\;\mathbf{P}(\Omega _{13})=1 on which there exists

and according to (?) and (2.5) we have uniformly in n: |x_{j_{1}}^{1}|\leq \Phi _{0},\;j_{1}=1,...,n, i.e., conditions (?) and (2.5) are also satisfied.

Hence, we can apply Theorem 2.1 on the subspace \overline{\Omega } _{1}=\Omega _{11}\cap \Omega _{12}\cap \Omega _{13}\subset \Omega _{1},\; \mathbf{P}(\overline{\Omega }_{1})=1 where all the conditions of the theorem are valid, i.e., \overline{\Omega }_{1} plays the role of \Omega _{R,x} of Remark 2.2 (i). Then, the theorem implies that for every \omega _{1}\in \overline{\Omega }_1 there exists subspace \overline{ \Omega^2}(\omega _{1}) of the space \Omega ^{2} generated by \{X_{n}^{2},b_{n}^{2}\} for all n and such that \mathbf{P}(\overline{ \Omega^2}(\omega _{1}))=1 and formulas (2.5) – (?) are valid for L=2. It follows then from the Fubini theorem that the same is true on a certain \overline{\Omega }_{2}\subset \Omega _{2},\;\mathbf{P}( \overline{\Omega }_{2})=1 where \Omega _{2} is defined by (?) with L=2.

This proves the theorem for L=2. The proof for L=3,4,\dots is analogous, since (cf. (?))

In particular, we have with probability 1 on \Omega _{l} of (?) for M_{n}^{l} playing the role of R_{n} of Theorem 2.1 on the l th step of the inductive procedure (cf. (?))

If x_0 is random, then it is necessary to follow the argument given in Remark 2.6 (iii).

ssertions (i) and (ii) are elementary.

(iii). To obtain (?) we use (?) with A and B of (?) to write the formula

Multiplying it by L of (3.1) from the right and using LG_{B}(z)L=(G_{B}(z)Y,Y)L, we get

Plugging this into the r.h.s. of (A), we obtain (?) and then (?).

Note that the r.h.s. of (?) and (?) are well defined for \Im z\neq 0. Indeed, it follows from (?) with A-z as A and A-z^{\ast }as B that

thus

To get (?) we take into account that K\geq 0 and (G_{B}(-\xi )Y,Y)\geq 0, since B, hence, G_{B}(-\xi ) is positive definite. This yields the following bound for the r.h.s. of (?)

implying

(iv) We use the formulas (see (?))

The first line above, (3.1) and (?) yields (?) and (?), while the second line implies

The sums of the r.h.s. are bounded by

and m_{4}\geq 1 in view of (?). This leads (?).

e begin with using Lemma 3.4 (i) below implying that the fluctuations (?) of \nu _{ \mathcal{M}_{n}} vanish sufficiently fast as n\rightarrow \infty. This and the Borel-Cantelli lemma imply that

with probability 1, hence reduce the proof of the theorem to the proof of the weak convergence of the expectation

of \nu _{\mathcal{M}_{n}} to the limit \nu _{\mathcal{M}} whose Stieltjes transform is given by (?) – (?). Since \mathcal{M} _{n} is positive definite, hence, its spectrum belongs to the closed positive semiaxis \mathbb{R}_{+} for all n, it suffices to prove the tightness of the sequence of measures \{\overline{\nu }_{\mathcal{M}_{n}}\}_{n} and the pointwise convergence on a set of positive Lebesgue measure in \mathbb{C} \setminus \mathbb{R}_{+} of their Stieltjes transforms (cf. (?))

to the limit satisfying (?) – (?), see, e.g. [Pa-Sh:11], Proposition 2.1.2.

The tightness is guaranteed by the uniform in n bound for

since for any T>0 we have for the tail of \overline{\nu }_{\mathcal{M} _{n}}

According to the definition of the NCM (see, e.g. (?)), spectral theorem and (2.1), we have

and then (?), (?) – (?) and (?) – (?) yield

where we used the inequality n^{-1}\mathrm{Tr}\,R_{n} \le (n^{-1}\mathrm{Tr}\,R_{n}^2)^{1/2} to obtain the r.h.s. bound. This implies the tightness of \{\overline{\nu }_{\mathcal{M}_{n}}\}_{n} and reduces the proof of the theorem to the proof of pointwise in \mathbb{C} \setminus \mathbb{R}_{+} convergence of (W) to the limit determined by (?) – (?).

The above argument, reducing the analysis of the large size behavior of the eigenvalue distribution of random matrices to that of the expectation of the Stieltjes transform of the distribution, is widely used in random matrix theory (see [Pa-Sh:11], Chapters 3, 7, 18 and 19), in particular, while dealing with the sample covariance matrices. However, the matrix \mathcal{M} _{n} of (2.1) differs essentially from the sample covariance matrices, since the "central" matrix K_{n} of (?) is random and dependent on X_{n} (the data matrix according to statistics), while in the sample covariance matrix the analog of K_{n} is either deterministic or random but independent of X_{n}. Nevertheless, we show that in our case of the X_{n}-dependent K_{n} of (?) it suffices to follow essentially the proof for X_{n}-independent analogs of K_{n}, which dates back to [Ma-Pa:67, Pa:72] and has been largely extended and used afterwards, see, e.g. [Pa-Sh:11, Ak-Co:11, Ba-Si:00, Gi:12].

We outline first the scheme of the proof of the theorem. Write (2.1) as

where K_{jn} are given by (?) and

is the rank-one matrix (3.1) corresponding to the random vector (cf. ( ?))

i.e., X_{j} is the jth row of the random matrix (?), thus the collection \{X_{j}\}_{j=1}^{n} consists of i.i.d. random vectors satisfying (?).

It follows from the definition of the Normalized Counting Measure (see (?)) and the spectral theorem for the resolvent

of \mathcal{M}_{n} that

Hence, we have to deal with G_{\mathcal{M}_{n}}(z).

By using the resolvent identity (?) for A=\mathcal{M}_{n} and B=0, we obtain for (?) in view of (?)

and we omit here and below the subindex \mathcal{M}_{n} in the resolvent ( ?) as well as the subindex n in many instances below where this does no lead to confusion.

Next, we choose in (?)

and use (?)

to obtain

where

and Y_{j} is defined in (?). Plugging (?) into (?), we obtain our basic starting formula

which we are going to convert into the "prelimit" version of the system (?) – (?)

plus error terms vanishing as n\rightarrow \infty.

It follows from (?) that we are allowed to make any modification of (?) provided that the corresponding error term \mathcal{E}_n satisfies

Denote \mathbf{E}_{j}\{...\} the operation of expectation conditioned on \{X_k\}_{k \neq j}

and use:

(i) (?) – (?) and (?) to replace the random quadratic form (C Y_{j},Y_{j})

with a Y_{j} -independent matrix C_{} by

(ii) (?) – (?) to replace n^{-1}\mathrm{Tr}G_{j}(z)C with a Y_{j} -independent C by n^{-1}\mathrm{Tr}\,G(z)C;

(iii) Lemma 3.4 to replace the random variable n^{-1}\mathrm{Tr}\, G(z)C with a Y_{j} -independent matrix C by the expectation \mathbf{E}\{n^{-1} \mathrm{Tr}\,G(z)C\}.

We will apply then: (i) with C=G_{j} to replace a_{j}(z) of (?) by

(ii) with C=R to replace h_{jn}(z) by h_{n}(z) and then (iii) with C=R to replace h_{n}(z) by \overline{h_{n}}(z), where

As a result, we can replace (1+K_{j}a_{j}(z)) by (1+K_{j}\overline{h_{n}} (z)) in the r.h.s. of (?), see Lemma 3.3 for details.

Likewise, we can replace L_{j} in (?) by its expectation n^{-1}R by using (i) and then replace G_{j}(z) by G(z) by using (ii) to convert (?) into

where

and

is the error term.

Applying to (?) the operation \mathbf{E}\{n^{-1}\mathrm{Tr\;}\dots\} and taking into account (?) and (?), we get

where

i.e., a "prelimit" version of (?) with the error term t_{1n}, cf. ( ?).

Next, we have from (?)

Multiplying the formula by R, applying to the result the operation \mathbf{E}\{n^{-1}\mathrm{Tr\;}...\} and using the fact that R of (?), hence, \mathcal{G} are independent of X_n of (?), we obtain in view of (?)

where \nu _{R_{n}} is the Normalized Counting Measure of R_{n} defined in (?). This is a "prelimit" version of (?) with the error term (cf. (?) and (?))

At last, observing that according to the conditions of the theorem (see (?), (?) and (?)) \{K_{j}\}_{j=1}^{n} are independent identically distributed (and, possibly, n-dependent) random variables, we obtain from (?) the "prelimit version"

of (?) in which \overline{\nu }_{K_{n}} is the probability law of (see (?))

Having obtained semi-heuristically relations (?), (?) and ( ?), we pass now to their rigorous derivation, i.e., to the proof that the remainders t_{1n} of (?) and t_{2n} of (?) vanish in the limit n\rightarrow \infty and that f_{\mathcal{M}_{n}}, \overline{h}_{n} and \overline{k}_{n} converge to a solution of (? ) – (?).

We will deal first with the n \to \infty limit in (?), (?) and (?) assuming that t_{1n} and t_{2n} vanish as n\rightarrow \infty. In fact, the limit is a version of that widely used in random matrix theory, see, e.g. [Pa-Sh:11]. Thus, we just outline the procedure.

According to (W) f_{\mathcal{M}_{n}} is analytic in \mathbb{C} \setminus \mathbb{R}_{+} for every n. Thus, by Vitali’s theorem on the convergence of analytic functions, it suffices to study the limiting properties of the sequence \{f_{\mathcal{M}_{n}}\}_{n} for z varying in a closed interval of the open negative semiaxis

where \xi_\pm do not depend on n.

Furthermore, since \mathcal{M}_{n} of (2.1) and \mathcal{M} _{n}^{(j)} of (?) are positive definite, their resolvents G(z) and G_{j}(z ) for z=-\xi \in I_{-} are also positive definite, thus

and we have for a_{j} of (?)

Besides, we have from (?) and (?)

Thus, 1+K_{j}a_{j}(-\xi )\geq 1 and (?) is well defined for \xi >0.

It follows from (?), spectral theorem for \mathcal{M}_{n} and (?) that if \{\lambda_\alpha\}_\alpha and \{\psi_\alpha\}_\alpha are the eigenvalues and the eigenvectors of \mathcal{M}_n, then

where we took into account that R is positive defined and used Schwarz inequality for traces. This implies that h_n is analytic in \mathbb{C} \setminus \mathbb{R}_+ and

In particular, the function \overline{k}_n of (?) is also analytic in \mathbb{C} \setminus \mathbb{R}_+.

It follows also from the above and (W) that

Moreover, since the sequences \{f_{\mathcal{M}_{n}}\}_n, \; \{\overline{h}_{n}\}_n and \{\overline{k}_{n}\}_n are real analytic on I_- of (?), there exists a subsequence n_{j}\rightarrow \infty such that \{f_{\mathcal{M} _{n_{j}}}\},\{\overline{h}_{n_{j}}\} and \{\overline{k}_{n_{j}}\} converge uniformly on (?) to certain limits f_{\mathcal{M}},h and k analytic in \mathbb{C}\setminus \mathbb{R}_{+}. This allows us to carry out the limit along n_{j}\rightarrow \infty in the second term in the r.h.s. of (?) and to obtain (?) provided that t_{1n_{j}} vanishes as n_{j}\rightarrow \infty.

Next, write the first term in the r.h.s. of (?) for z=-\xi \in I_{-} as

It follows then from (?), (?) and (?) that the first term tends to the r.h.s. of (?) as n_{j}\rightarrow \infty. The integral in the second term is bounded by r_{2}/\xi ^{2} in view of (?) and (?), hence, the second term vanishes as n_{j}\rightarrow \infty. Thus, we obtain (?) provided that t_{2n_{j}} vanishes as n_{j}\rightarrow \infty.

An analogous argument applies to (?). However, to obtain (?) and (?), we have to find the limiting probability law of the random variable \eta _{jn} of (?). It follows from the standard facts on the Central Limit Theorem (see, e.g. [Sh:96], Section III.4), (?) and (?) that the law is Gaussian of zero mean and variance q-\sigma _{b}^{2}, see Lemma 3.5 for details. This proves (?).

Thus, we have proved the validity of (?) – (?) for z\in I_{-} of (?) provided that t_{1n} and t_{2n} of (?) and (?) vanish uniformly in z\in I_{-}. It is shown in Lemma 3.7 that the system (?) – (?) is well defined and uniquely solvable everywhere in \mathbb{C}\setminus \mathbb{R}_{+}. This implies that the whole sequences \{f_{\mathcal{M}_{n}}\},\{\overline{h} _{n}\} and \{\overline{k}_{n}\} converge uniformly on any compact set of \mathbb{C}\setminus \mathbb{R}_{+}, that their limits f_{\mathcal{M}},h and k are not identically zero and can be found from relations (?) – (?) which are valid everywhere in \mathbb{C}\setminus \mathbb{R} _{+}. Indeed, if h is identically zero, then it follows from (?) with z=-\xi <0 that k(-\xi )=k_{1}>0\,, where k_{1} is the first moment of measure \nu _{K} of (?). Then (?) implies that \nu _{R} is concentrated at zero. This contradicts condition (a) of the theorem. Analogously, assuming that k is identically zero, we conclude that \nu _{K} of (?) is concentrated at zero and this is impossible if \varphi is not identically constant.

Besides, it follows from (?)

that k(-\xi ) and \xi h(-\xi ) are nonnegative and bounded. Thus, the limit \xi \rightarrow \infty in (?) yields (?). Conditions (?) follow from the n \to \infty versions of (?) – (?).

We pass now to the most technical part of the proof in which we establish that the error terms (?) and (?) vanish as n \to \infty uniformly on z\in I_{- } of (?).

Note first that it suffices to assume that the sequence \{R_{n}\} of (2.1) – (?) is uniformly bounded, i.e.,

instead of (?). This is also a standard and technically convenient trick of random matrix theory where it is shown that once the limiting Normalized Counting Measure is found under condition (?), it can be also found under condition (?), see e.g. [Pa-Sh:11], Section 19, in particular, Theorem 19.1.8 for the case where K_{n} is independent of X_{n} of (?). In our case of (?) – (?) the proof of this fact is given in [Pa:20].

By using (?) and (?), we have from (?), (? ) and (?)

It follows then from (?) – (?) that

These bounds, (?) and (?) imply

According to Lemma 3.3, d_{1j}\leq C^{\prime }/n^{1/2} if n is large enough and we obtain

This and (?) justify (?).

Consider now t_{2n} of (?). Using an argument similar to that leading to (?), we obtain

where (cf. (?) and (?))

and (cf. ?)

Since R is positive definite, (?) implies for \mathcal{G} of ( ?) (cf. (?))

It follows then from (?), (?), (?) and (?) that

This, (?) and (?) yield

with d_{1} of (?). Thus, Lemma 3.3 implies

for a certain n-independent C_{21}.

Likewise, by using (?), (?) and (?), we obtain

and then Lemma 3.3 implies |t_{2n}''(-\xi )|\leq C_{22}/n^{1/2} for a certain n-independent C_{22}.

Combining this bound, (?) and ( ?), we get (cf. (?))

This and (?) justifies (?).

e have by Schwarz inequality,

and then the inequality \left( a_{1}+a_{2}+a_{3}\right) ^{2}\leq 3(a_{1}^{2}+a_{2}^{2}+a_{3}^{2})

yields

where h_{jn} is defined in (?). It follows from (?) and (?) that

Denote by \mathbf{E}_{j}\{\dots\} the (conditional) expectation with respect to X_{j} and \mathbf{Var}_{j}\{\dots\} the corresponding variance (recall that according to (?) \{X_{j}\}_{j=1}^{n} are the n-component i.i.d. vectors with i.i.d. components). Since G_{j} is independent of X_{j} by (?), we have from the above and (?) \ \mathbf{E}_{j}\{a_{j}\}=h_{jn} (see (?)), thus the first term on the r.h.s. of (?) is

Next, (?), (?) and (?) imply

since

and we obtain for the first term of (?)

Consider the second term of the r.h.s. of (?). Since G(-\xi ) and G_{j}(-\xi ) in the definitions (?) of h_{n} and (?) of h_{jn} are the resolvents of positive definite \mathcal{M}_{n} and \mathcal{M}_{n}-K_{j}L_{j}, we use (?) with A=\mathcal{M}_{n} and C=R and (?) to obtain

that |h_{jn}-h_{n}|\leq \rho /n\xi. Hence, we have for the second term of the r.h.s. of (?)

As for the third term in the r.h.s. of (?), its bound follows from Lemma 3.4 (ii) with A=R, yielding in view of (?)

Combining this bound with (?) and (?) and using then (W), we get the first bound in (3.3).

To prove the second bound in (3.3) we apply an analogous argument to the r.h.s. of

where c_{jn}=n^{-1}\mathrm{Tr}\,R^{2}\mathcal{G}G_{j} (cf. (?)).

It follows from (?) and (?) that \mathbf{E} _{j}\{b_{j}\}=c_{jn}, hence, (cf. (?))

Using (?) with C=R\mathcal{G}G_{j} and taking into account that S^{2}=R and that R and \mathcal{G} commute (see (?)), we have by (?), (?) and (?)

hence, the bound for the first term of (?)

Next, we have

(cf. (?)) for the second term of (?) and

by Lemma 3.4 with A=R^{2}\mathcal{G} for the third term of (?). Plugging the above three bound into (?), we obtain the second bound in (3.3).

t follows from a general martingale difference argument (see [Pa-Sh:11], Proposition 18.1.1) that if \psi :\mathbb{R}^{n^{2}}\rightarrow \mathbb{C},

\{X_{j}\}_{j=1}^{n} are i.i.d. random vectors, \Psi =\psi (X_{1},\dots,X_{n}), \mathbf{E}_{j}\{\dots\} is the expectation conditioned on \{X_k\}_{k \neq j}

and

then

where C_{p} depends only on p.

Choose \Psi =\nu _{\mathcal{M}_{n}}(\Delta ) and the rows \{X_{j\alpha }\}_{\alpha =1}^{n} of X_{n} of (?) as X_{j} and write \nu _{ \mathcal{M}_{n}}(\Delta )=\nu _{\mathcal{M}_{n}^{(j)}}(\Delta )+\mu _{jn}(\Delta ), where \mathcal{M}_{n}^{(j)} is defined in (?). Since \mathcal{M}_{n}-\mathcal{M}_{n}^{(j)}=K_{j}L_{j} is a rank-one matrix, we can use the interlacing property of eigenvalues of a hermitian matrix and its rank-one perturbation (see [Ho-Jo:13], Section 4.3 and formula (?) of this paper) to show that |\mu _{jn}(\Delta )|\leq 1/n for any \Delta \in \mathbb{R}_{+} and any realization of random parameters. Hence, taking into account that \mathcal{M}_{n}^{(j)} does not depend on X_{j}, we obtain

This and (?) with p=2 imply assertion (i) of the lemma with C^{(1)}=2^{4}C_{2}.

To prove assertion (ii) we choose p=1 in (?) ,

and the same X_{j}’s. If G_{j} is given by (?), then we have by ( ?) with A and B as in (?) and C=A:

Hence, in this case (cf. (?))

and we obtain the bound

implying assertion (ii) of the lemma.

e will use the Central Limit Theorem for independent and not necessarily identically distributed random variables \{\xi _{\alpha n}\}_{\alpha =1}^{n} with

In this case \Sigma _{n}^{-1}\Xi _{n} converges in distribution to the standard Gaussian variable \gamma if for any \tau >0

where I is the indicator of \mathbb{R}_{+} (see [Sh:96], Section III.4). Choosing X_{\alpha }x_{\alpha n} as \xi _{\alpha n}, it is easy to find from (?) that (W) is equivalent to

It follows from (3.5) that \mathbf{E}\{X_{\alpha }^{2}I(|X_{\alpha }|-\tau \sqrt{n}/x_{\alpha n})\}\leq m_{4}x_{\alpha n}^{2}/\tau ^{2}n and then the l.h.s. of (W) is bounded by

which is zero in view of (?).

Likewise, \Sigma _{n}^{-1}\Xi _{n} is equivalent to (q_{n}-\sigma _{b}^{2})^{-1/2}\eta _{n}, hence, converges in distribution to \gamma.

et l=1 in (3.6) Since \{b_{j_{1}}^{1}\}_{j_{1}=1}^{n} and \{X_{j_{1},j_{0}}^{1}\}_{j_{1},j_{0}=1}^{n} are i.i.d. random variables satisfying (?) – (?), it follows from (?) that the components \{y_{j_{1}}^{1}\}_{j_{1}=1}^{n} of y^{1} are also i.i.d. random variables of zero mean and variance q_{n}^{1} of (2.5). Since \chi is bounded, the collection \{\chi (y_{j_{1}}^{1})\}_{j_{1}=1}^{n} consists of bounded i.i.d random variables defined for all n on the same probability space \Omega _{1} generated by (?) and (?) with l=1. This allows us to apply to \{\chi (y_{j_{1}}^{1})\}_{j_{1}}^{n} the strong Law of Large Numbers implying (3.6) for l=1 together with the formula

valid on a certain \overline{\Omega }_{1}\subset \Omega _{1}=\Omega ^{1}, \mathbf{P}(\overline{\Omega} _{1})=1, see (?).

To get (3.6) for l=1 recall that according to (?) and (?) – (?)

and \eta _{1}^{1} and b_{1}^{1} are independent. Hence,

where \Gamma_{n} is the probability law of \eta _{1}^{1}. Passing here to the limit n\rightarrow \infty and using Lemma 3.5, and (2.5) – (2.5), we obtain (3.6) for l=1.

Consider now the case l=2. Since \{X^{1},b^{1}\} and \{X^{2},b^{2}\} are independent, we can fix \omega _{1}\in \overline{\Omega }_{1},\ \mathbf{ P}(\overline{\Omega }_{1})=1 (a realization of \{X^{1},b^{1}\}) and apply to \chi _{n}^{2} of (3.6) the same argument as that for the case l=1 above to prove that for every \omega _{1}\in \overline{\Omega }_{1} there exists \overline{\Omega }^{2}(\omega ^{1})\subset \Omega ^{2},\; \mathbf{P}(\overline{\Omega }^{2}(\omega ^{1}))=1\, on which we have the analog of (S)

where \mathbf{E}_{\{X^{2},b^{2}\}}\{...\} denotes the expectation with respect to \{X^{2},b^{2}\} only. Now, by using again Lemma 3.5 and the Fubini theorem we obtain that there exists \overline{\Omega } _{2}\subset \Omega _{2}=\Omega ^{1}\otimes \Omega ^{2},\; \mathbf{P}( \overline{\Omega }_{2})=1 on which we have (3.6) for l=2 with

The limit in the r.h.s. above exists with probability 1 on \Omega ^{1} and equals the r.h.s. of (3.6) for l=2 just because it is a particular case of (3.6) for \chi =\varphi ^{2} and l=2.

This proves the validity (3.6) – (3.6) for l=2 with probability 1. Analogous argument applies for l=3,4,\dots.

To prove (?) it suffices to prove the validity with probability 1 of

for any bounded and continuous \psi :\mathbb{R\rightarrow R}.

In view of (?), (?) and (?) the relation can be written as

The l.h.s. here is a particular case of (3.6) – (3.6) for \chi =\psi \circ \varphi ^{\prime 2}, thus, it equals the r.h.s. of (3.6) for this \chi.

ote that in the course of proving Theorem 2.1 it was proved that the system has at least one solution satisfying the conditions of the lemma.

Let us prove assertion (i) of the lemma. It follows from (?), ( 3.7) and the analyticity of h in \mathbb{C}\setminus \mathbb{R} _{+} that k is also analytic in \mathbb{C}\setminus \mathbb{R}_{+}. Next, for any solution of (?) – (?) we have from (?) with \Im z\neq 0

and then (?) yields (?) for \Im z\neq 0, while (?) with z=-\xi <0, i.e.,

the positivity of h(-\xi ) (see (?)), (3.7) and Schwarz inequality yield (?) for z=-\xi.

Let us prove now that the system (?) – (?) is uniquely solvable in the class of pairs of functions (h,k) analytic in \mathbb{C} \setminus \mathbb{R}_{+} and satisfying (?) and (?). The argument below is a version of that used in [Pa-Sh:11], Lemma 2.2.6 for the deformed Wigner ensemble.

Assume that there exist two different solutions (h_{1},k_{1}) and (h_{2},k_{2}) of (?) – (?), i.e., there exists z_{0} \in \mathbb{C}\setminus \mathbb{R}_{+}, where at least one of two functions \delta h=h_{1}-h_{2}, \; \delta k=k_{1}-k_{2} is not zero. Since \delta h and \delta k are analytic in \mathbb{C}\setminus \mathbb{R}_{+}, we can assume without loss of generality that \Im z_{0} \neq 0. It follows then from (? ) – (?) that

where

Viewing (?) as a system of linear equations for \delta h and \delta k that has a non-trivial solution, we conclude that

On the other hand, we have by Schwarz inequality

where

In addition, the imaginary parts of (?) and (?) yield

hence, by (?),

This and (?) lead to the strict inequality

which contradicts (?).

Let us prove assertion (ii) of the lemma. Since h^{(p)} and k^{(p)} are analytic and uniformly in p bounded outside the closed positive semiaxis, there exist subsequences \{h^{(p_{j})},k_{{}}^{(p_{j})}\}_{j} converging pointwise in \mathbb{C}\setminus \mathbb{R}_{+} to a certain analytic pair (\widetilde{h},\widetilde{k}). Let us show that (\widetilde{h},\widetilde{ k})=(h,k). It suffices to consider real negative z=-\xi >0 (see (?)). Write for the analog of (?) for \nu _{K^{(p)}}:

Putting here p=p_{j}\rightarrow \infty, we see that the l.h.s. converges to \widetilde{k}, the first integral on the right converges to the r.h.s of (?) with \widetilde{h} instead of h since \nu _{K^{(p)}} converges weakly to \nu _{K}, the integrand is bounded and continuous and the second integral is bounded in p since h^{(p)}(-\xi )>0, \widetilde{h}(-\xi )>0 and the second moment of \nu _{K^{(p)}} is bounded in p according to (3.7), hence, the second term vanishes as p=p_{j}\rightarrow \infty. An analogous argument applied to (? ) show (\widetilde{h},\widetilde{k}) is a solution of (?) – (?) and then the unique solvability of the system implies that ( \widetilde{h},\widetilde{k})=(h,k).