{"id":51,"date":"2010-03-17T11:57:26","date_gmt":"2010-03-17T11:57:26","guid":{"rendered":"https:\/\/blog.informationgeometry.org\/?p=51"},"modified":"2021-07-31T11:57:52","modified_gmt":"2021-07-31T11:57:52","slug":"additive-versus-non-additive-property-of-entropy","status":"publish","type":"post","link":"https:\/\/blog.informationgeometry.org\/additive-versus-non-additive-property-of-entropy\/","title":{"rendered":"Additive versus non-additive property of entropy"},"content":{"rendered":"<p>Shannon entropy is said additive in the sense that the entropy of the joint distribution<\/p>\n<pre>H(X*Y)<\/pre>\n<p>is the sum of the entropies:<\/p>\n<pre>H(X*Y) = H(X)+H(Y).<\/pre>\n<p>This property is not true for the quadratic entropy (sum of squares). The Java program below checks experimentally those entropy functional properties. Frank.<\/p>\n<p>Here is the Java code<\/p>\n<pre><code>\/*\r\n (C) March 2010, Frank Nielsen\r\n Demonstrates that Shannon entropy is additive\r\n and that the quadratic entropy is not\r\n*\/\r\npublic class ShannonAdditiveEntropy\r\n{\r\n \/\/quadratic entropy\r\npublic static double Q(double [] p)\r\n{\r\ndouble res=0.0d;\r\nfor(int i=0;i &lt; p.length;i++)\r\n    res+= p[i]*p[i];\r\n    \r\nreturn res;     \r\n}   \r\n \/\/Shannon entropy\r\npublic static double H(double [] p)\r\n{\r\ndouble res=0.0d;\r\nfor(int i=0;i &lt; p.length;i++)\r\n    res+= -p[i]*Math.log(p[i]);\r\n    \r\nreturn res;     \r\n}   \r\n    \r\npublic static void main (String [] args)\r\n{\r\nint n=10,i,j;\r\ndouble sump=0.0, sumq=0.0;\r\ndouble [] p=new double[n];\r\ndouble [] q=new double[n];\r\ndouble [] pq=new double[n*n];\r\n\r\nfor(i=0;i &lt; n;i++)\r\n{\r\np[i]=Math.random(); q[i]=Math.random(); \r\nsump+=p[i]; sumq+=q[i];\r\n}\r\n\/\/ normalize to densities\r\nfor(i=0;i &lt; n;i++)\r\n{\r\n    p[i]\/=sump;  q[i]\/=sumq;\r\n}\r\n\r\nfor(i=0;i &lt; n;i++)\r\n    for(j=0;j &lt; n;j++)\r\n     pq[i*n+j]=p[i]*q[j];\r\n     \r\nSystem.out.println(\"Show that Shannon entropy is an additive entropy:\");\r\nSystem.out.println(\"H(p)=\"+H(p)+\" H(q)=\"+H(q)+\" H(pq)=\"+H(pq) );\r\ndouble delta=H(pq)-H(p)-H(q);\r\nSystem.out.println(\"Additive entropy:\"+delta);\r\nSystem.out.println(\"Show that the quadratic entropy is NOT an additive entropy:\");\r\nSystem.out.println(\"Q(p)=\"+Q(p)+\" Q(q)=\"+Q(q)+\" H(pq)=\"+Q(pq) );\r\ndouble deltaq=Q(pq)-Q(p)-Q(q);\r\nSystem.out.println(\"Not an ddditive entropy:\"+deltaq);\r\n     \r\n}\r\n    \r\n}\r\n\r\n<\/code><\/pre>\n<p>Frank.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Shannon entropy is said additive in the sense that the entropy of the joint distribution H(X*Y) is the sum of the entropies: H(X*Y) = H(X)+H(Y). This property is not true for the quadratic entropy (sum of squares). The Java program<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/blog.informationgeometry.org\/wp-json\/wp\/v2\/posts\/51"}],"collection":[{"href":"https:\/\/blog.informationgeometry.org\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.informationgeometry.org\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.informationgeometry.org\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.informationgeometry.org\/wp-json\/wp\/v2\/comments?post=51"}],"version-history":[{"count":3,"href":"https:\/\/blog.informationgeometry.org\/wp-json\/wp\/v2\/posts\/51\/revisions"}],"predecessor-version":[{"id":54,"href":"https:\/\/blog.informationgeometry.org\/wp-json\/wp\/v2\/posts\/51\/revisions\/54"}],"wp:attachment":[{"href":"https:\/\/blog.informationgeometry.org\/wp-json\/wp\/v2\/media?parent=51"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.informationgeometry.org\/wp-json\/wp\/v2\/categories?post=51"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.informationgeometry.org\/wp-json\/wp\/v2\/tags?post=51"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}