Information Geometry

  • Home
  • About Us
  • Privacy Policy
  • Contact Us

Daily Archives: March 17, 2010

Additive versus non-additive property of entropy

Frank March 17, 2010 News Comments are off

Shannon entropy is said additive in the sense that the entropy of the joint distribution

H(X*Y)

is the sum of the entropies:

H(X*Y) = H(X)+H(Y).

This property is not true for the quadratic entropy (sum of squares). The Java program below checks experimentally those entropy functional properties. Frank.

Here is the Java code

/*
 (C) March 2010, Frank Nielsen
 Demonstrates that Shannon entropy is additive
 and that the quadratic entropy is not
*/
public class ShannonAdditiveEntropy
{
 //quadratic entropy
public static double Q(double [] p)
{
double res=0.0d;
for(int i=0;i < p.length;i++)
    res+= p[i]*p[i];
    
return res;     
}   
 //Shannon entropy
public static double H(double [] p)
{
double res=0.0d;
for(int i=0;i < p.length;i++)
    res+= -p[i]*Math.log(p[i]);
    
return res;     
}   
    
public static void main (String [] args)
{
int n=10,i,j;
double sump=0.0, sumq=0.0;
double [] p=new double[n];
double [] q=new double[n];
double [] pq=new double[n*n];

for(i=0;i < n;i++)
{
p[i]=Math.random(); q[i]=Math.random(); 
sump+=p[i]; sumq+=q[i];
}
// normalize to densities
for(i=0;i < n;i++)
{
    p[i]/=sump;  q[i]/=sumq;
}

for(i=0;i < n;i++)
    for(j=0;j < n;j++)
     pq[i*n+j]=p[i]*q[j];
     
System.out.println("Show that Shannon entropy is an additive entropy:");
System.out.println("H(p)="+H(p)+" H(q)="+H(q)+" H(pq)="+H(pq) );
double delta=H(pq)-H(p)-H(q);
System.out.println("Additive entropy:"+delta);
System.out.println("Show that the quadratic entropy is NOT an additive entropy:");
System.out.println("Q(p)="+Q(p)+" Q(q)="+Q(q)+" H(pq)="+Q(pq) );
double deltaq=Q(pq)-Q(p)-Q(q);
System.out.println("Not an ddditive entropy:"+deltaq);
     
}
    
}

Frank.

Recent Posts

  • How to Choose a General Contractor for a Custom Home
  • How Do Security Systems Work at Home?
  • Common CCTV Problems and Their Fixes
  • How to Choose Upholstery Fabric?
  • Why Educational Signage Is Important?
  • Pros and Cons of Foam Filled Tires
  • What Does an Accountant Do?
  • Types of Vacuum Cleaners
  • Tips For Hiring a Ducted Air Conditioning Installer
  • Common Roofing Problems And How To Fix Them

Archives

  • December 2022
  • November 2022
  • October 2022
  • September 2022
  • August 2022
  • July 2022
  • June 2022
  • May 2022
  • April 2022
  • February 2022
  • January 2022
  • December 2021
  • November 2021
  • October 2021
  • September 2021
  • August 2021
  • June 2010
  • May 2010
  • April 2010
  • March 2010
March 2010
M T W T F S S
1234567
891011121314
15161718192021
22232425262728
293031  
    Apr »
Theme by ThemesPie | Proudly Powered by WordPress