arXiv reader

Optimally-Weighted Herding is Bayesian Quadrature

Ferenc Huszár and David Duvenaud

arXiv
1204.1664v3 · PDF · 15 July 2016
Categories
stat.ML, cs.NA, math.NA
Licence
CC BY 4.0 The site shows the text.

Abstract

Herding and kernel herding are deterministic methods of choosing samples which summarise a probability distribution. A related task is choosing samples for estimating integrals using Bayesian quadrature. We show that the criterion minimised when selecting samples in kernel herding is equivalent to the posterior variance in Bayesian quadrature. We then show that sequential Bayesian quadrature can be viewed as a weighted version of kernel herding which achieves performance superior to any other weighted herding method. We demonstrate empirically a rate of convergence faster than O(1/N). Our results also imply an upper bound on the empirical error of the Bayesian quadrature estimate.

Items 4

Proposition

Proposition 1

8RSC

The expected variance in the Bayesian quadrature \epsilon^{2}_{{\textsc{bq}}{}} is the maximum mean discrepancy between the target distribution p and q_{{\textsc{bq}}{}}(x) = \sum_{n=1}^{N}w^{(n)}_{{\textsc{bq}}{}}\delta_{x_n}(x)

Proof of Proposition 1

The proof involves invoking the representer theorem, using bilinearity of scalar products and the fact that if f is a standard Gaussian process then \forall g\in\mathcal{H}: \left\langle f,g\right\rangle \sim \mathcal{N}(0,\left\|g\right\|_{\mathcal{H}}^2):

\begin{aligned} &\mathbb{V}_{} \left[ {Z_{f,p}\vert f_{x_1}, \dots, f_{x_N}} \right]=\\ &= \mathbb{E}_{f\sim GP} \left( \int f(x) p(x) dx - \sum_{n=1}^{N}w^{(n)}_{{\textsc{bq}}{}} f(x_n)\right)^2\\ &= \mathbb{E}_{f\sim GP} \left( \int \left\langle f, \phi (x)\right\rangle p(x) dx - \sum_{n=1}^{N}w^{(n)}_{{\textsc{bq}}{}} \left\langle f, \phi (x_n)\right\rangle\right)^2\\ &= \mathbb{E}_{f\sim GP} \left\langle f , \int\phi(x) p(x) dx - \sum_{n=1}^{N}w^{(n)}_{{\textsc{bq}}{}}\phi(x_n)\right\rangle^2\\ &= \left\|\mu_p - \mu_{q_{{\textsc{bq}}{}}}\right\|_{\mathcal{H}}^2\\ &= {\rm MMD}^2(p,q_{{\textsc{bq}}{}}) \end{aligned}
Proposition

Proposition 2

MJ5M

\epsilon^{2}_{{\textsc{bq}}{}}(\emptyset)-\epsilon^{2}_{{\textsc{bq}}{}}(\cdot) is a weakly submodular set function with constant \epsilon<4r, where r is the incoherency

r = \max_{x,x'\in\mathcal{P}\subseteq\mathcal{X}} \frac{k(x,x')}{\sqrt{k(x,x)k(x',x')}}
Proof of Proposition 2

By the definition of {\rm MMD} we can see that -\epsilon^{2}_{{\textsc{bq}}{}} = -\inf_{w\in\mathbb{R}^N}\left\|\mu_p - \sum_{n=1}^N w^{(n)}_{{\textsc{bq}}{}}k(\cdot,{x}_n)\right\|_{\mathcal{H}}^2 is the negative squared distance between the mean element \mu_p and its projection onto the subspace spanned by the elements k(\cdot,{x}_n). Substituting k=1 into Theorem 1 of [KrauseCevher10] concludes the proof.

Source and licence

The text on this page is from arXiv:1204.1664v3 by Ferenc Huszár and David Duvenaud, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.

The record of each item is in analysis/objects/1204/1204.1664.jsonl.