Here is the reference work:
Output-sensitive peeling of convex and maximal layers

Doing some survey, I found that since Fermat (who allegedly first ask the question for 3 points), it has been studied and rediscovered in many communities. The current labeling of this point should be Fermat-Fagnano-Weber-Torricelli-Steiner point, and I forgot many names...

History is full of insights!
Frank.

The $2d+4$ simple quadratic natural exponential families on $R^d$

Frank.

1946: On some analogues of the amount of information and their use in statistical estimation: I
1947: On some analogues of the amount of information and their use in statistical estimation: II
1948: On some analogues of the amount of information and their use in statistical estimation: III

The celebrated Cramer-Rao inequality can be seen as a particular case of Bhattacharyya inequality. (provide a lower bound on the variance of ANY unbiased estimator of a parametric function).

The Bhattacharyya bounds obtained from Bhattacharyya matrices become sharper viz. the order of the matrix.

And information geometry cherishes lower bounds too. A great result not so popular somehow. It deserves prime time nowadays.

Frank.

See Attosecond imaging of molecular electronic wavepackets

I noticed that using twitter slow down blogging. What is the trade-off? Ponder!

Twitter: FrnkNlsn (yes, the one with vowels was already taken...)

FrnkNlsn

Everyone is welcome to subscribe -:)

Here are some accompanying materials:

• paper pdf
• poster pdf
• slides pdf

It will be on display as follows:

SPTM-P8: Signal and System Modeling and Estimation I
Location:   Poster Area B
Time:   Friday, March 19, 10:00 - 12:00

Frank.

On an absolute criterion for fitting frequency curves

Worth a glance!

Frank.

Quantification method of classification processes. Concept of structural a-entropy
from a 1967 paper of Jan Havrda and Frantisek Charvat.

Frank.

More in the book of Jeremy Gray
Ideas of space: Euclidean, non-Euclidean, and relativistic.

Frank.

In bioinformatics, beta mixtures have been proven useful for analyzing gene expressions. Either the same gene observed under different modalities (say, different maker microarrays or radioactivity labeled DNAc) or for extracting pathways (co-expressed genes). The basic feature is the correlation number of pairwise expressions. Modeling those correlation coefficient distribution allows one to fit beta mixtures with two components: The similar and dissimilar without using a priori threshold (often set arbitrarily to 0.5).
A standard EM algorithm using numerical optimization let us fit beta mixture. What is the best method and best tool for doing that? If there are biologists surfing here around, let me know please.
The reference of the paper is:

Bioinformatics. 2005 May 1;21(9):2118-22. Epub 2005 Feb 15.
Applications of beta-mixture models in bioinformatics.
Ji Y, Wu C, Liu P, Wang J, Coombes KR.

See link
for retrieving the paper.

See also our open source library for handling mixture models: jMEF

Frank.

If instead, we assume f strictly positive concave (and self-dual f(x)=x f(1/x)) then the formula of f-divergences yields metrics.

Frank.

• Statistical manifolds (parametric distributions),
• Neural manifolds (Boltzmann machines with fixed topology, i.e., number of nodes),
• ARMA(p,q) time-series manifolds (e-flat=-1-flat)

Strictly speaking, geometrizing information-theoretic problems does not provide a more powerful framework in theory. This is because synthetical and analytical geometries are equivalent. Informally, that means that we can do geometry by algebraic equations.

However, geometrizing problems help grab intuition on the problem at hand. Geometry also yields novel notions to mathematical theories. For example, let us cite the two curvature notions in statistics: exponential and mixture curvatures emanating from conjugate connections. So although synthetical geometry provides the same power as analytical geometry, the third-order asymptotic theory of statistics has been obtained so far only from synthetical information geometry.

Dual differential geometries are also useful to tackle information-theoretic problems such as

• Multiterminal problems met in information theory,
• Linear programming problems (e.g., continuous Karmarkar inner method walking along the m-geodesic),
• Clustering (negative entropy and dual Legendre log-normalizer conjugate for soft/hard clustering).

Frank.

• Beta=1 -> Kullback-Leibler
• Beta=0 -> Itakura-Saito
• Beta=2 -> squared Euclidean distance

In the paper, we derive formula for the beta left and right-sided centroids. The program RepresentationalBetaBregman.java shows this equivalence (up to numerical errors).

Shows that beta divergences can be obtained from representational Bregman divergences
beta-div=0.0060607537896566754  equals Bregman rep. div=0.006060753789656717
beta-div=0.03596836005227946    equals Bregman rep. div=0.03596836005227946
beta-div=0.03786639961675385    equals Bregman rep. div=0.03786639961675385
beta-div=0.015356733711556145   equals Bregman rep. div=0.015356733711556173
beta-div=0.16665973512136045    equals Bregman rep. div=0.16665973512136045
beta-div=0.006143185064308276   equals Bregman rep. div=0.006143185064308207
beta-div=0.012777128199946086   equals Bregman rep. div=0.012777128199946072
beta-div=2.42453303134018E-4    equals Bregman rep. div=2.4245330313402494E-4
beta-div=0.07962156613977964    equals Bregman rep. div=0.07962156613977961
beta-div=2.6549301732092453E-4  equals Bregman rep. div=2.6549301732092974E-4
Press any key to continue...

Frank.

The geometry is spherical on renormalized density functions.

Bhattacharyya with Mahalanobis distances were precursors of the Fisher-Rao Riemannian distances.

Frank.

• Shun-ichi Amari, Integration of Stochastic Models by Minimizing \alpha-Divergence, Neural Computation (NECO), (19)10:2780-2796, October 2007.

• F. Nielsen and R. Nock, The dual Voronoi diagrams with respect to representational Bregman divergences, International Symposium on Voronoi Diagrams (ISVD), June 2009.

• F. Nielsen and R. Nock, Sided and Symmetrized Bregman Centroids, IEEE transactions on information theory (2009), vol. 55, no. 6, pp. 2882-2904

jMEF in Matlab

Source GMM
Sample points from GMM

GMM obtained using Bregman soft clustering (expectation maximization)

The dual Voronoi diagrams with respect to representational Bregman divergences

slides

I show that by using a representational function, we can obtain alpha-divergences as a representational Bregman divergences. Therefore, it is easy to extend algorithms tailored to Bregman divergences to alpha divergences.

Here is a code snippet in Java: RepresentationalAlphaBregman.java

Running it, you get something like

Shows that alpha divergences can be obtained from representational Bregman divergences
alpha-div=0.9008838766115398    equals Bregman rep. div=0.9008838766115399
alpha-div=0.14730849849416264   equals Bregman rep. div=0.14730849849416267
alpha-div=0.05455969651963932   equals Bregman rep. div=0.054559696519639225
alpha-div=1.2439567444853374    equals Bregman rep. div=1.2439567444853372
alpha-div=0.15345391125768915   equals Bregman rep. div=0.15345391125768917
alpha-div=0.12118392973570616   equals Bregman rep. div=0.12118392973570632
alpha-div=1.038494366079179 equals Bregman rep. div=1.0384943660791794
alpha-div=0.08541142546071197   equals Bregman rep. div=0.08541142546071195
alpha-div=0.06842729068201092   equals Bregman rep. div=0.06842729068201084
alpha-div=0.6941500904965174    equals Bregman rep. div=0.6941500904965173

In the ISVD 2009 paper, we give closed-form solutions for centroids of representational Bregman divergences, including alpha-means et beta-means.

Frank.

Check the information monotonicity of Kullback-Leibler divergence:
by merging bins into a coarser histogram, the Kullback-Leibler divergence is less than the higher resolution:
0.08522624581487719 0.1320022228947157 0.13591019441965485 0.06674528382980667 0.05029196864402132 0.19946790829780184 0.20516964900877682 0.1251865270903457
0.02648151081895093 0.17500830227728026 0.2779122146863664 0.06203415984687348 0.2535969883830692 0.04424529112439828 0.1320533102395284 0.028668222623533027
                KL(p,q)=0.46400208957858724     KL(q,p)=0.4558758820302893
4 bins  KL(p,q)=0.10555546240269079     KL(q,p)=0.09733318810652078
2 bins  KL(p,q)=0.029647768170771346    KL(q,p)=0.029836929625659273
Information monotonicity holds only for Csiszar's f-divergences

Rao's distance is the Riemannian geodesic distance induced by the Fisher information matrix as the tensor. Computing Rao's distance for given parametric distributions (let d be the number of parameters), thus involved to compute explicitly the geodesic. The distance is the length of that geodesic, the sum of its infinitesimal elements along the shortest path curve. It is thus quite complicated to compute in practice the exact Rao's distance as we need to solve the differential equation of the geodesic stated by the Euler-Lagrange equations. In this paper, three different approaches are proposed;

• Classic Euler-Lagrange equations (d second order differential equations)
• Hamilton's equations (2d first order differential equations)
• Hamilton-Jacobi equations (nonlinear partial differential equation)

Frank.

The Gamma distribution belongs to the exponential families. Therefore, the Fisher information metric is $I(\theta)=\nabla^2 F(\theta)$. However, integrating the square root of the information matrix is difficult (no closed form solution). The author proceeds by characterizing the Riemannian geodesic using the differential equation relying on Christoffel symbols. Geodesics on the Gamma manifold are unique since the manifold is simply connected, complete with all sectional curvatures nonpositive. The authors come up with a Newton-like numerical optimization algorithm that depends on a good initialization. First, they show that the metric is bounded by Poincare metrics for which closed form equations of the geodesics are known. This yields a good starting tangent vector.
It is quite impressive to look at the formula of the closed-form equation of the Poincare geodesics. Those formula are surprisingly quite complicated.
The authors implemented their algorithm in FORTRAN and show that the algorithm always convergence on the domain examples, with high numerical precisions.

jMEF

We can learn mixture of exponential families (such as Gaussian mixture models), mixture of Poisson, mixture of Laplacians, etc.

The Voronoi diagram on hyperbolic geometry has thus been studied in conformal Poincare upper-plane representation. However, geodesics are visualized by arcs of circles and it makes computation more difficult if not tricky. Another problem is that it requires more numerical precision to carry out predicate evaluations.

Now consider the Klein ball realization. In this non-conformal representation, geodesics are straight line segment (but the mid Euclidean point is not the mid hyperbolic point). Bisectors are also hyperplanes and the diagram is therefore affine, and can be computed from an equivalent power diagram. So hyperbolic Voronoi diagrams are handy and do not require more specific implementation than a weighted Voronoi diagram.

The details are explained in the following report, illustrated with an application on image browsing
PDF

To compute the PCA of high-dim datasets, we just need to compute the SVD of the covariance matrix of zero-mean normalized data sets. So I looked for a good source of explanations of SVD and I came across the lecture of Gilles Strang:
SVD lecture

Here, the 4 subspaces (image and nullspace) of column/row matrices are reviewed and it is shown how to compute the SVD by simply solving left/right eigenproblems.

Definitively worth watching! (you'll see on one example a problem with the sign in a SVD decomposition to solve!!!)

class VarargsSum
{
    public static double cumul(double... elements)
    {
    double cumul=0.0d;
    for (double el:elements)
        cumul+=el;
    return cumul;   
    }
    
    public static void write(String... records)
    {
    for (String record: records)
      System.out.println(record);
    }
    public static void main(String [] args)
    {
    Double sum=cumul(5.0,4.0,23.0); 
    write("Computational","Information","Geometry",sum.toString());// explicit cast needed
    }   
}

Compiling and running this above code, you will get

Computational
Information
Geometry
32.0

Let me see how to best use this in the library now... Frank.

applet
Frank.

For the small story, here is the inscription plate:

Here as he walked by on the 16th of October 1843 Sir William Rowan Hamilton in a flash of genius discovered the fundamental formula for quaternion multiplication i2 = j2 = k2 = ijk = -1 & cut it on a stone of this bridge

How much progress has been done in a century! (eg., Lie groups and algebras)
Frank.

All technical details are described in the paper:
S.Wu and S. Amari, Conformal Transformation of Kernel Functions: A Data-Dependent Way to Improve Support Vector Machine Classifiers, Neural Processing Letters, 15, pp. 59-67, 2002.

Frank.