Computational Information Geometry Wonderland

@FrnkNlsn

Today is a short post on a NIPS'98 paper:

Batch and on-line parameter estimation of Gaussian mixtures based on the joint entropy

The authors compute the relative entropy (KL divergence) among two Gaussian mixture models (GMMs) and derive a simple update rule (in sec 5). They call their method joint entropy (JE) update and claim that it requires half the steps of the EM.

The important remark is to notice that the relative entropy between two GMMs is non-convex. So they proceed by studying an upper-bound called the joint entropy distance