{"id":223,"date":"2006-08-28T18:21:29","date_gmt":"2006-08-29T00:21:29","guid":{"rendered":"http:\/\/hunch.net\/?p=223"},"modified":"2006-08-28T18:21:30","modified_gmt":"2006-08-29T00:21:30","slug":"learning-theory-standards-for-nips-2006","status":"publish","type":"post","link":"https:\/\/hunch.net\/?p=223","title":{"rendered":"Learning Theory standards for NIPS 2006"},"content":{"rendered":"

Bob Williamson<\/a> and I are the learning theory PC members at NIPS<\/a> this year. This is some attempt to state the standards and tests I applied to the papers. I think it is a good idea to talk about this for two reasons:<\/p>\n

    \n
  1. Making community standards a matter of public record seems healthy. It give us a chance to debate what is and is not the right standard. It might even give us a bit more consistency across the years.<\/li>\n
  2. It may save us all time. There are a number of papers submitted which just aren’t there yet. Avoiding submitting is the right decision in this case.<\/li>\n<\/ol>\n

    There are several criteria for judging a paper. All of these were active this year. Some criteria are uncontroversial while others may be so.<\/p>\n

      \n
    1. The paper must have a theorem establishing something new for which it is possible to derive high confidence in the correctness of the results. A surprising number of papers fail this test. This criteria seems essential to the definition of “theory”.\n
        \n
      1. Missing theorem statement<\/strong><\/li>\n
      2. Missing proof<\/strong> This isn’t an automatic fail, because sometimes reviewers can be expected to fill in the proof from discussion. (Not all theorems are hard.) Similarly, sometimes a proof sketch is adequate. Providing the right amount of detail to give confidence in the results is tricky, but general advice is: err on the side of being explicit.<\/li>\n
      3. Imprecise theorem statement<\/strong> A number of theorems are simply too imprecise to verify or imagine verifying. Typically they are written in english or mixed math\/english and have words like “small”, “very small”, or “itsy bitsy”.<\/li>\n
      4. Typos and thinkos<\/strong> Often a theorem statement or proof is “right” when expressed correctly, but it isn’t expressed correctly: typos and thinkos (little correctable bugs in how you were thinking) confuse the reader.<\/li>\n
      5. Not new<\/strong> This may be controversial, because the test of ‘new’ is stronger than some people might expect. A theorem of the form “algorithm A can do B” is not new when we already know “algorithm C can do B”.<\/li>\n<\/ol>\n

        Some of these problems are sometimes fixed by smart reviewers. Where that happens, it’s fine. Sometimes a paper has a reasonable chance of passing evaluation as an algorithms paper (which has experimental requirements). Where that happens, it’s fine.\n<\/li>\n

      6. The paper should plausibly lead to algorithmic implications. This test is applied in a varying strength. For an older mathematical model of learning, we tried to apply at the level of “I see how an algorithm might be developed from this insight”. For a new model of learning, this test was applied only weakly.\n<\/li>\n
      7. We did not<\/em> require that the paper be about machine learning. For non-learning papers, we decided to defer to the judgement of referees on whether or not the results were relevant to NIPS. It seemed more natural that authors\/reviewers be setting the agenda here.<\/li>\n
      8. I had a preference for papers presenting new mathematical models. I liked Neil Lawrence<\/a>‘s comment: “If we started rejecting learning theory papers for having the wrong model, where would we stop?” There is a natural tendency to forget the drawbacks of the accepted models in machine learning when evaluating new models, so it seems appropriate to provide some encouragement towards exploration.\n<\/li>\n
      9. Papers were not penalized for having experiments. Sometimes experiments helped (especially when the theory was weak), and sometimes they had no effect.<\/li>\n<\/ol>\n

        Reviewing is a difficult process—it’s very difficult to get 82 (the number this time) important decisions right. It’s my hope that more such decisions can be made right in the future, so I’d like to invite comments on what the right criteria are and why. This year’s decisions are made now (and will be released soon), so any suggestions will just influence the future.<\/p>\n","protected":false},"excerpt":{"rendered":"

        Bob Williamson and I are the learning theory PC members at NIPS this year. This is some attempt to state the standards and tests I applied to the papers. I think it is a good idea to talk about this for two reasons: Making community standards a matter of public record seems healthy. It give … <\/p>\n

        Continue reading “Learning Theory standards for NIPS 2006”<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[29],"tags":[],"class_list":["post-223","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"_links":{"self":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts\/223","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=223"}],"version-history":[{"count":0,"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts\/223\/revisions"}],"wp:attachment":[{"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=223"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=223"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=223"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}