{"id":65,"date":"2005-04-23T09:29:51","date_gmt":"2005-04-23T15:29:51","guid":{"rendered":"\/?p=65"},"modified":"2005-04-23T10:07:30","modified_gmt":"2005-04-23T16:07:30","slug":"advantages-and-disadvantages-of-bayesian-learning","status":"publish","type":"post","link":"https:\/\/hunch.net\/?p=65","title":{"rendered":"Advantages and Disadvantages of Bayesian Learning"},"content":{"rendered":"

I don’t consider myself a “Bayesian”, but I do try hard to understand why Bayesian learning works. For the purposes of this post, Bayesian learning is a simple process of:<\/p>\n

    \n
  1. Specify a prior over world models.<\/li>\n
  2. Integrate using Bayes law with respect to all observed information to compute a posterior over world models.<\/li>\n
  3. Predict according to the posterior.<\/li>\n<\/ol>\n

    Bayesian learning has many advantages over other learning programs:<\/p>\n

      \n
    1. Interpolation<\/strong> Bayesian learning methods interpolate all the way to pure engineering. When faced with any learning problem, there is a choice of how much time and effort a human vs. a computer puts in. (For example, the mars rover pathfinding algorithms are almost entirely engineered.) When creating an engineered system, you build a model of the world and then find a good controller in that model. Bayesian methods interpolate to this extreme because the Bayesian prior can be a delta function on one model of the world. What this means is that a recipe of “think harder” (about specifying a prior over world models) and “compute harder” (to calculate a posterior) will eventually succeed. Many other machine learning approaches don’t have this guarantee. <\/li>\n
    2. Language<\/strong> Bayesian and near-Bayesian methods have an associated language for specifying priors and posteriors. This is significantly helpful when working on the “think harder” part of a solution.<\/li>\n
    3. Intuitions<\/strong> Bayesian learning involves specifying a prior and integration, two activities which seem to be universally useful. (see intuitions<\/a>).<\/li>\n<\/ol>\n

      With all of these advantages, Bayesian learning is a strong program. However, there are also some very significant disadvantages.<\/p>\n

        \n
      1. Information theoretically infeasible<\/strong> It turns out that specifying a prior is extremely difficult. Roughly speaking, we must specify a real number for every setting of the world model parameters. Many people well-versed in Bayesian learning don’t notice this difficulty for two reasons:\n
          \n
        1. They know languages allowing more compact specification of priors. Acquiring this knowledge takes some signficant effort.<\/li>\n
        2. They lie. They don’t specify their actual prior, but rather one which is convenient. (This shouldn’t be taken too badly, because it often works.)<\/li>\n<\/ol>\n<\/li>\n
        3. Computationally infeasible<\/strong> Let’s suppose I could accurately specify a prior over every air molecule in a room. Even then, computing a posterior may be extremely difficult. This difficulty implies that computational approximation is required.<\/li>\n
        4. Unautomatic<\/strong> The “think harder” part of the Bayesian research program is (in some sense) a “Bayesian employment” act. It guarantees that as long as new learning problems exist, there will be a need for Bayesian engineers to solve them. (Zoubin<\/a> likes to counter that a superprior over all priors can be employed for automation, but this seems to add to the other disadvantages.)<\/li>\n<\/ol>\n

          Overall, if a learning problem must<\/em> be solved a Bayesian should probably be working on it and has a good chance of solving it.
          \nI wish I knew whether or not the drawbacks can be convincingly addressed. My impression so far is “not always”.<\/p>\n","protected":false},"excerpt":{"rendered":"

          I don’t consider myself a “Bayesian”, but I do try hard to understand why Bayesian learning works. For the purposes of this post, Bayesian learning is a simple process of: Specify a prior over world models. Integrate using Bayes law with respect to all observed information to compute a posterior over world models. Predict according … <\/p>\n

          Continue reading “Advantages and Disadvantages of Bayesian Learning”<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[],"class_list":["post-65","post","type-post","status-publish","format-standard","hentry","category-bayesian"],"_links":{"self":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts\/65","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=65"}],"version-history":[{"count":0,"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts\/65\/revisions"}],"wp:attachment":[{"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=65"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=65"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=65"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}