Computational Information Geometry Wonderland

It is well known that the Kullback-Leibler divergence of two densities belonging to the same exponential family has a closed-form equation, since it is equivalent to the Bregman divergence on swapped natural parameters (see here).
Since the Kullback-Leibler divergence is the relative entropy, it is equal to the cross-entropy minus the entropy (note that this decomposition does not apply to other deformed logarithms such as Renyi/Tsallis relative entropies). So by decomposing the Bregman divergence into two terms (up to a constant, we find the cross-entropy and entropy of exponential families). What about the constant? Well it is the expectation of the carrier measure. (For Gaussians, it is zero, and therefore the entropy has a simple closed form). Even if the expectation integral does not admit a closed-form, we can always compare exactly entropy of members of the same family. This should be advantageous in a number of image processing algorithms.

• The ICIP 2010 paper
• The ICIP 2010 poster

All these are easily implemented in the jMEF

Frank.

I recently read articles on paleomagnetism. It is common to make the assumption of antipodal symmetry for the distribution of the dispersion of the directions. Two such spherical distributions (directional statistics) with such an antipodal symmetry are the Bingham and the Kent distributions. It will be worth writting the jMEF files for those ones.

Frank.

They are only 6 exponential family distributions that admit variance as a quadratic function (QVF=quadratic variance function) of the parameter. For the multivariate case, it is a bit more complex but well defined and studied:

The $2d+4$ simple quadratic natural exponential families on $R^d$

Frank.

The jMEF library is easily interfaced with Matlab. You can compute Gaussian Mixture Models (GMMs) and manipulate them easily now in Matlab.

jMEF in Matlab

Source GMM
Sample points from GMM

GMM obtained using Bregman soft clustering (expectation maximization)

We have developed a library for manipulating exponential families in statistics:

jMEF

We can learn mixture of exponential families (such as Gaussian mixture models), mixture of Poisson, mixture of Laplacians, etc.

It is well-known that any smooth density can be well approximated using a mixture of Gaussians. Gaussian distributions belong to the family of exponential families in statistics. There is even a more powerful property of exponential families with so-called rich sufficient statistics.

They take advantage of RKHS (Reproducing Kernel Hilbert Space). So forget the mixture and just consider one exponential family for modeling non-parametric distribution.

Details are in: Exponential families for conditional random fields (2004)

To get a random Gaussian N(mu,Sigma), sample the mu uniformly from [0,1] (or N(0,1)) and sample the symmetric positive definite matrix from a Wishart distribution: Sample each entry of a matrix A from a Normal distribution N(0,1), and set SIGMA=lambda AA^T where lambda is a uniform random number that controls the size of the covariance matrice.

More than a year ago, I printed 3D Bregman balls using a lithography process. I took may hours to print these shapes...

Generalized Kullback-Leibler (positive measures):

Itakura-Saito (Burg entropy):

Logistic loss:

To create the STL files, you need to discretize the surface of theses balls by walking on their geodesics passing through the centers. A bisection search does this to stop a more or less the radius.

Since most pdfs can be equivalently expressed as a Gaussian mixture model, it is tempting to simplify them. The measure taken between two GMMs (one with many components, and the other with a reduced number) is the Kullback-Leibler relative entropy (=cross entropy-entropy). The problem is to decide whether we should minimize KL(GMMsource||GMM simplified) or the converse KL(GMM simplified||GMMsource) or the symmetrized KL (Jensen-Shannon, in green). Here are a few experiments carried out when simplidying a GMM to a single normal:


I let you guess whether the blue/red is the left/right or the converse... -:)

I invite you to read this paper:

Empirical development of an exponential probabilistic model for text retrieval: using textual analysis to build a better model

The authors concentrate on determining what is a good model for explaining a text corpus. They show that the multinomial model falls short, as it is widely known, and go on learning the 1-order exponential family that better represent corpora for naive Bayesian retrieval.

Overall, although we cannot explain all data sets, and risk over-fitting, it is crucial to design better generative models. One big success these years was the LDA (Latent Dirichlet Allocation) scheme. But, this has to be learnt and not hardcoded anymore. There is plenty room for this line of research...