Computational Information Geometry Wonderland

It is well known that for finite discrete distributions [that is, multinomials], Shannon entropy is continuous inside the open probability simplex. However when the space becomes countably infinite, surprisingly, Shannon entropy is discontinuous (everywhere) as well as other quantities like the mutual information. All details can be found in the paper [pdf]

S.W. Ho and R. W. Yeung,
On the Discontinuity of the Shannon Information Measures, IEEE Transactions on Information Theory, 2009

Frank.