Information Geometry

  • Home
  • About Us
  • Privacy Policy
  • Contact Us
Home  /  News  /  Additive versus non-additive property of entropy

Additive versus non-additive property of entropy

Frank March 17, 2010 News Comments are off

Shannon entropy is said additive in the sense that the entropy of the joint distribution

H(X*Y)

is the sum of the entropies:

H(X*Y) = H(X)+H(Y).

This property is not true for the quadratic entropy (sum of squares). The Java program below checks experimentally those entropy functional properties. Frank.

Here is the Java code

/*
 (C) March 2010, Frank Nielsen
 Demonstrates that Shannon entropy is additive
 and that the quadratic entropy is not
*/
public class ShannonAdditiveEntropy
{
 //quadratic entropy
public static double Q(double [] p)
{
double res=0.0d;
for(int i=0;i < p.length;i++)
    res+= p[i]*p[i];
    
return res;     
}   
 //Shannon entropy
public static double H(double [] p)
{
double res=0.0d;
for(int i=0;i < p.length;i++)
    res+= -p[i]*Math.log(p[i]);
    
return res;     
}   
    
public static void main (String [] args)
{
int n=10,i,j;
double sump=0.0, sumq=0.0;
double [] p=new double[n];
double [] q=new double[n];
double [] pq=new double[n*n];

for(i=0;i < n;i++)
{
p[i]=Math.random(); q[i]=Math.random(); 
sump+=p[i]; sumq+=q[i];
}
// normalize to densities
for(i=0;i < n;i++)
{
    p[i]/=sump;  q[i]/=sumq;
}

for(i=0;i < n;i++)
    for(j=0;j < n;j++)
     pq[i*n+j]=p[i]*q[j];
     
System.out.println("Show that Shannon entropy is an additive entropy:");
System.out.println("H(p)="+H(p)+" H(q)="+H(q)+" H(pq)="+H(pq) );
double delta=H(pq)-H(p)-H(q);
System.out.println("Additive entropy:"+delta);
System.out.println("Show that the quadratic entropy is NOT an additive entropy:");
System.out.println("Q(p)="+Q(p)+" Q(q)="+Q(q)+" H(pq)="+Q(pq) );
double deltaq=Q(pq)-Q(p)-Q(q);
System.out.println("Not an ddditive entropy:"+deltaq);
     
}
    
}

Frank.

Share this...
Share on Facebook
Facebook
Pin on Pinterest
Pinterest
Tweet about this on Twitter
Twitter
Share on LinkedIn
Linkedin
Previous Article
Next Article

About Author

Frank

Related Posts

  • What are the Types of Rehabilitation?

    What are the Types of Rehabilitation?

    May 19, 2022
  • Article About Craft Brewery Supplies

    Article About Craft Brewery Supplies

    May 9, 2022
  • Preparing Your Garden

    Preparing Your Garden

    April 25, 2022

Recent Posts

  • What are the Types of Rehabilitation?
  • Article About Craft Brewery Supplies
  • Preparing Your Garden
  • The Importance of Pool Cleaning
  • Executive Coaching For Business Owners
  • Home Alarm Systems – Why You Should Buy One
  • When Do You Need to Call a Locksmith?
  • What Can a Mortgage Broker Do For You?
  • Cattery Services in Melbourne
  • The Importance of Electrical Installation

Archives

  • May 2022
  • April 2022
  • February 2022
  • January 2022
  • December 2021
  • November 2021
  • October 2021
  • September 2021
  • August 2021
  • June 2010
  • May 2010
  • April 2010
  • March 2010
May 2022
M T W T F S S
 1
2345678
9101112131415
16171819202122
23242526272829
3031  
« Apr    
Theme by ThemesPie | Proudly Powered by WordPress