{"id":209,"date":"2006-07-08T13:06:09","date_gmt":"2006-07-08T19:06:09","guid":{"rendered":"http:\/\/hunch.net\/?p=209"},"modified":"2006-07-08T18:34:11","modified_gmt":"2006-07-09T00:34:11","slug":"209","status":"publish","type":"post","link":"https:\/\/hunch.net\/?p=209","title":{"rendered":"MaxEnt contradicts Bayes Rule?"},"content":{"rendered":"<p>A few weeks ago I read <a href=\"http:\/\/emotion.inrialpes.fr\/~dangauthier\/blog\/2006\/04\/26\/maximum-entropy-and-bayesian-updating\/\">this<\/a>. David Blei and I spent some time thinking hard about this a few years back (thanks to Kary Myers for pointing us to it):<\/p>\n<p><em>In short I was thinking that \u00c3\u00a2\u00e2\u201a\u00ac\u00c5\u201cbayesian belief updating\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u009d and \u00c3\u00a2\u00e2\u201a\u00ac\u00c5\u201cmaximum entropy\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u009d were two othogonal principles. But it appear that they are not, and that they can even be in conflict !<br \/>\nExample (from Kass 1996); consider a Die (6 sides), consider prior knowledge E[X]=3.5.<br \/>\nMaximum entropy leads to P(X)= (1\/6, 1\/6, 1\/6, 1\/6, 1\/6, 1\/6).<br \/>\nNow consider a new piece of evidence A=\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u009dX is an odd number\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u009d<br \/>\nBayesian posterior P(X|A)= P(A|X) P(X) = (1\/3, 0, 1\/3, 0, 1\/3, 0).<br \/>\nBut MaxEnt with the constraints E[X]=3.5 and E[Indicator function of A]=1 leads to (.22, 0, .32, 0, .47, 0) !! (note that E[Indicator function of A]=P(A))<br \/>\nIndeed, for MaxEnt, because there is no more \u00c3\u00a2\u00e2\u201a\u00ac\u00cb\u01536\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u00b2, big numbers must be more probable to ensure an average of 3.5. For bayesian updating, P(X|A) doesn\u00c3\u00a2\u00e2\u201a\u00ac\u00e2\u201e\u00a2t have to have a 3.5 expectation. P(X) and P(X|a) are different distributions.<br \/>\nConclusion ? MaxEnt and bayesian updating are two different principle leading to different belief distributions. Am I right ?<br \/>\n<\/em><\/p>\n<p>I don&#8217;t believe there is any paradox at all between MaxEnt (perhaps more generally, MinRelEnt) and Bayesian updates. Here, straight MaxEnt make no sense. The implication of the problem is that the ensemble average 3.5 is <em>no longer an active constraint<\/em>. That is, we no longer believe the contraint E[X]=3.5 once we have the additional data that X is an odd number. The sequential update using minimum relative entropy is identical to Bayes rule and produces the correct answer. These two answers are simply (correct) answers to different questions.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A few weeks ago I read this. David Blei and I spent some time thinking hard about this a few years back (thanks to Kary Myers for pointing us to it): In short I was thinking that \u00c3\u00a2\u00e2\u201a\u00ac\u00c5\u201cbayesian belief updating\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u009d and \u00c3\u00a2\u00e2\u201a\u00ac\u00c5\u201cmaximum entropy\u00c3\u00a2\u00e2\u201a\u00ac\u00c2\u009d were two othogonal principles. But it appear that they are not, and &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/hunch.net\/?p=209\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;MaxEnt contradicts Bayes Rule?&#8221;<\/span><\/a><\/p>\n","protected":false},"author":9,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[29],"tags":[],"class_list":["post-209","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"_links":{"self":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts\/209","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=209"}],"version-history":[{"count":0,"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts\/209\/revisions"}],"wp:attachment":[{"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=209"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=209"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=209"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}