A deep first-order system least squares method for solving elliptic PDEs

Let us first focus on a generic neural network v_{{{\bm \Theta}}}: \mathbb{R}^d \to \mathbb{R} with one hidden layer,

Above, we assume \sigma is a Lipschitz continuous activation function, and the parameters B \in \mathbb{R}^{1\times n}, A \in \mathbb{R}^{n\times d} and c \in \mathbb{R}^{n \times 1} are collected in {{\bm \Theta}} \in \mathbb{R}^m, m = n (d+2). Using the fact that v_{{{\bm \Theta}}} and its derivatives depend continuously on the parameters, one can verify easily that the map \mathbb{R}^m \mapsto W^{1,\infty}(\Omega) such that {{\bm \Theta}} \mapsto v_{{{\bm \Theta}}} is continuous. Moreover, the function G: \mathbb{R}^m \times \Omega \to \mathbb{R}, defined as G({{\bm \Theta}},x) := v_{{{\bm \Theta}}}(x) is Lipschitz continuous, and therefore it is bounded on B(0,R) \times \Omega and its (weak) derivatives are essentially bounded on the same set as well. Furthermore, if f \in L^2(\Omega) then |G({{\bm \Theta}},x) + f(x)|^2 \leq 2|G({{\bm \Theta}},x)|^2 + 2|f(x)|^2 \leq 2M + 2|f(x)|^2 =: s({{\bm \Theta}},x), with s \in L^1(B(0,R) \times \Omega).

For arbitrary neural network functions (u_{{{\bm \Theta}}},{{\bm \phi}}_{ {{\bm \Theta}}}) in the space \mathcal{A}_m, defined by (?), we exploit the idea above together with the fact that the auxiliary functions d_\mathcal{D}, d_\mathcal{N} and {{\bm n}} are smooth to conclude the desired result.

From [Cai94], we know that \mathcal{L} is elliptic with respect to the H^1(\Omega) \times H(\operatorname{div}; \Omega) norm. Namely, there exist positive constants \alpha and \beta such that

for all (u,{{\bm \phi}}) \in H^1(\Omega) \times H(\operatorname{div}; \Omega).

Let \varepsilon > 0. By (?), we consider m_0 > 0 such that d({{\bm q}}_0,\mathcal{A}_m)<\varepsilon and 1/m < \varepsilon for all m>m_0. For every m > 0, there exists {{\bm q}}^*_{m} = (u^*_{m}, {{\bm \phi}}^*_{m}) \in \mathcal{A}_{m} with d({{\bm q}}_0, \mathcal{A}_{m}) \ge \|{{\bm q}}^*_m-{{\bm q}}_0\|_{H^1(\Omega) \times H(\operatorname{div}; \Omega)} - \varepsilon. Then, for all m>m_0 and every neural network quasi-minimizer {{\bm q}}_m = (u_m, {{\bm \phi}}_m) \in \mathcal{I}_m, using that the solution {{\bm q}}_0 = (u_0,{{\bm \phi}}_0) of (?) satisfies the conditions {{\bm \phi}}_0 = {{\bm A}} \nabla u_0 and -\operatorname{div}({{\bm \phi}}_0) + B u_0 = f a.e. in \Omega and exploiting the upper bound in (Proof), we have

Finally, by combining this estimate with the lower bound in (Proof), and exploiting the fact that {{\bm q}}_0 satisfies (?) a.e. in \Omega, we reach the estimate

for every {{\bm q}}_m \in \mathcal{I}_m and m > m_0. Since \varepsilon is arbitrary small, this concludes the proof.

Since we are using the same parameter R in the definitions of L and L_{\lambda,N}, if |{{\bm \Theta}}| > R we have L({{\bm \Theta}}) = L_{\lambda,N}({{\bm \Theta}}) = +\infty and there is nothing to be proven. We therefore assume |{{\bm \Theta}}| \le R. Recalling V_N(\lambda) = \cup_{i \le N} \{X_i(\lambda)\} with \lambda \in \Lambda and the definition of G_1 and G_2 (3.1), an application of the strong law of large numbers yields

and

for all (u,{{\bm \phi}}) \in \mathcal{A}_m. It follows immdiately that L_{\lambda,N}({{\bm \Theta}}) \to L({{\bm \Theta}}) P-almost surely as N \to \infty.

We first observe that the lim-sup inequality is a trivial corollary of Lemma 3.3. Indeed, it suffices to consider the recovery sequence \{{{\bm \Theta}}_N\}_{N \in \mathbb{N}} \subset \mathbb{R}^m, {{\bm \Theta}}_N \equiv {{\bm \Theta}}, and by Lemma 3.3 we have L_{\lambda,N}({{\bm \Theta}}_N) \to L({{\bm \Theta}}) with N \to \infty P-almost surely.

We next prove the lim-inf inequality. Given {{\bm \Theta}} \in \mathbb{R}^m, let \{{{\bm \Theta}}_N\}_{N \in \mathbb{N}} \subset \mathbb{R}^m be a sequence of parameters such that {{\bm \Theta}}_{N} \to {{\bm \Theta}}. We aim to prove that

We observe that, if |{{\bm \Theta}}|>R then there exists N_0 = N_0(\lambda) such that L({{\bm \Theta}})=L_{\lambda,N}({{\bm \Theta}}_N) = +\infty for all N>N_0, and (?) trivially holds. Therefore, without loss of generality we assume \{{{\bm \Theta}}_N\}_{N \in \mathbb{N}} \subset \overline{B(0,R)}. In that case, we extract a subsequence in such a way that L_{\lambda,N}({{\bm \Theta}}_N) \to \liminf_{N \to \infty} L_{\lambda,N}({{\bm \Theta}}_N) and, for the sake of simplicity, we omit the relabeling. By Lemma 3.1, the map {{\bm \Theta}} \mapsto (u_{{{\bm \Theta}}},{{\bm \phi}}_{ {{\bm \Theta}}}) \in (\mathcal{A}_m,\| \cdot \|_{H^1(\Omega) \times H(\operatorname{div}; \Omega)}) is continuous and therefore (u_{{{\bm \Theta}}_N}, {{\bm \phi}}_{{{\bm \Theta}}_N}) \to (u_{{{\bm \Theta}}}, {{\bm \phi}}_{{{\bm \Theta}}}) in the H^1(\Omega) \times H(\operatorname{div}; \Omega) norm. Because \Omega is bounded, this implies

Then, defining G_1 and G_2 as in (3.1), we extract another subsequence in such a way that G_1({{\bm \Theta}}_N,x) + G_2({{\bm \Theta}}_N,x) \to G_1({{\bm \Theta}},x) + G_2({{\bm \Theta}},x) almost everywhere in \Omega, and, as before, we omit the relabeling.

In order to prove (?), we are going to show that the latter subsequence satisfies L_{\lambda,N}({{\bm \Theta}}_N) \to L({{\bm \Theta}}) with N \to \infty P-almost surely. Let \varepsilon>0 be an arbitrary number, using the triangle inequality, we split

From Lemma 3.3, it follows that |L_{\lambda,N}({{\bm \Theta}}) - L({{\bm \Theta}})| \to 0 P-almost surely. Thus, there exists N_0=N_0(\lambda) such that |L_{\lambda,N}({{\bm \Theta}}) - L({{\bm \Theta}})|\le \varepsilon/4 for all N>N_0.

In order to bound the first term in the right hand side in (?), we first observe that Lemma 3.1 shows that G_1 is uniformly bounded and G_2 is bounded above by some integrable function. Thus, there exists s \in L^1(\Omega), depending on R, such that

for all ({{\bm \Theta}},x) \in B(0,R) \times \Omega. Now we apply Egorov’s Theorem to construct a set \mathcal{K} \subset \Omega such that \int_{\mathcal{K}} s(x) dx < \varepsilon/8 and G_1({{\bm \Theta}}_N, \cdot) + G_2({{\bm \Theta}}_N,\cdot) \to G_1({{\bm \Theta}},\cdot) + G_2({{\bm \Theta}},\cdot) uniformly in \Omega \setminus \mathcal{K}. We bound

where

Using the uniform convergence in \Omega \setminus \mathcal{K}, P-almost surely there exists N_1 = N_1(\lambda) such that, if N > N_1, then \big| G_1({{\bm \Theta}}_N,x) + G_2({{\bm \Theta}}_N,x) - G_1({{\bm \Theta}},x) - G_2({{\bm \Theta}},x) \big| < \frac{\varepsilon}{4|\Omega|} for all x \in \Omega \setminus \mathcal{K}. Then, it follows that A_1 < \varepsilon/4 if N>N_1.

On the other hand, we use (?) to derive

By the strong law of large numbers, we have

Therefore, P-almost surely there exists N_2 = N_2(\lambda) such that, if N>N_2 then

which implies that \frac{|\Omega|}{N}\sum_{ x \in V_N(\lambda) } \chi_\mathcal{K}(x) s(x) < \frac{\varepsilon}{4}. Consequently, we have A_2 < \frac{\varepsilon}{4}.

Collecting the estimates above, it follows that P-almost surely we can choose N' = N'(\lambda) = \max \{N_0,N_1,N_2\} such that

for all N > N'. This shows that (?) holds, and concludes the proof.

Let \varepsilon > 0 and consider the set of neural network quasi-minimizers introduced in Lemma 3.2, \mathcal{I}_m = \{ {{\bm q}} \in \mathcal{A}_m : \mathcal{L}({{\bm q}}) \leq \mathcal{L}({{\bm q}}^*) + 1/m \ \forall {{\bm q}}^* \in \mathcal{A}_m \}. By that lemma, there exists m_0 > 0 such that

for all {{\bm q}}_{m_0} \in \mathcal{I}_{m_0}. Next, we fix R_0>0 large enough so that there exists {{\bm \Theta}} \in B(0,R_0) with {{\bm q}}_{{\bm \Theta}}^* = (u_{{{\bm \Theta}}}^*,{{\bm \phi}}_{ {{\bm \Theta}}}^*) \in \mathcal{I}_{m_0}. For the functional L defined in (?), this implies that q_{{{\bm \Theta}}} \in \mathcal{I}_m for all {{\bm \Theta}} \in \arg \min_{{{\bm \Theta}} \in B(0,R_0)} L({{\bm \Theta}}).

For this choice of m_0 and R_0, from Theorem 3.2 we have L_{\lambda,N} \xrightarrow[]{\Gamma} L P-almost surely. From the definition of L_{\lambda,N} (?), it follows immediately that \{L_{\lambda , N}\}_{N \in \mathbb{N}} is an equi-coercive sequence, according to Definition 3.2. Therefore, we deduce that P-almost surely there exists N_0 > 0 such that

for all N>N_0 for some {{\bm q}}_{m_0} \in \mathcal{I}_{m_0}. This bound follows by Theorem 3.1 because every cluster point of \{ {{\bm \Theta}}_N\} is a minimum point for L, and because of the continuity of the map {{\bm \Theta}} \mapsto (u_{{{\bm \Theta}}},{{\bm \phi}}_{ {{\bm \Theta}}}).

The proof concludes upon combining (Proof) and (?).

The arguments used in the proof of Theorem 3.2 can be easily adapted to this case.

Indeed, the lim-sup inequality follows trivially by taking the recovery sequence \{{{\bm \Theta}}_N\}_{N \in \mathbb{N}}, {{\bm \Theta}}_N \equiv {{\bm \Theta}} and using a strong law of large numbers.

To prove the lim-inf inequality, we start from a bounded sequence of parameters \{ {{\bm \Theta}}_N\}_{N \in \mathbb{N}} and use (H1) to extract a converging subsequence \{ {{\bm q}}_{{{\bm \Theta}}_N}\}_{N \in \mathbb{N}} in the \mathcal{A}-norm. Then, by (H2) we can extract another subsequence such that G_i({{\bm \Theta}}_{n_j},x) \to G_i({{\bm \Theta}},x) \mu_i-almost everywhere for all 1 \le i \le K and by (H3) we know that every function G_i has an upper bound in L_{\mu_i}^1( \Omega). The conclusion then follows by applying Egorov’s Theorem on every subset \omega_1, \ldots \omega_K.

We first remark that hypothesis (H4) is needed to guarantee the existence of a well-defined minimizer {{\bm q}}_0 \in \mathcal{A}, and therefore (H5) is meaningful. Given \varepsilon > 0, we use hypothesis (H5) to find m_0 \in \mathbb{N} such that, if {{\bm q}}_{m_0} \in \mathcal{I}_{m_0} then \| {{\bm q}}_0 - {{\bm q}}_{m_0}\|_{\mathcal{A}} < \varepsilon/2.

Next, we fix R_0>0 large enough so that there exists {{\bm \Theta}} \in B(0,R_0) with {{\bm q}}_{ {{\bm \Theta}}} \in \mathcal{I}_{m_0}, and use this R_0 in Theorem 4.1 to deduce that L_{\lambda,N} \xrightarrow[]{\Gamma} L with N \to \infty P-almost surely. The result then follows by the equi-coercivity of the sequence \{L_{\lambda , N}\}_{N \in \mathbb{N}} by applying the fundamental theorem of \Gamma-convergence (Theorem 3.1).