{"id":178,"date":"2006-03-27T17:12:34","date_gmt":"2006-03-27T23:12:34","guid":{"rendered":"http:\/\/hunch.net\/?p=178"},"modified":"2006-03-27T17:12:49","modified_gmt":"2006-03-27T23:12:49","slug":"gradients-everywhere","status":"publish","type":"post","link":"https:\/\/hunch.net\/?p=178","title":{"rendered":"Gradients everywhere"},"content":{"rendered":"

One of the basic observations from the atomic learning workshop<\/a> is that gradient-based optimization is pervasive. For example, at least 7 (of 12) speakers used the word ‘gradient’ in their talk and several others may be approximating a gradient. The essential useful quality of a gradient is that it decouples local updates from global optimization. Restated: Given a gradient, we can determine how to change individual parameters of the system so as to improve overall performance.<\/p>\n

It’s easy to feel depressed about this and think “nothing has happened”, but that appears untrue. Many of the talks were about clever techniques for computing gradients where your calculus textbook breaks down.<\/p>\n

    \n
  1. Sometimes there are clever approximations of the gradient. (Simon Osindero<\/a>)<\/li>\n
  2. Sometimes we can compute constrained gradients via iterated gradient\/project steps. (Ben Taskar<\/a>)<\/li>\n
  3. Sometimes we can compute gradients anyways over mildly nondifferentiable functions. (Drew Bagnell<\/a>)<\/li>\n
  4. Even given a gradient, the choice of update is unclear, and might be cleverly chosen (Nic Schraudolph<\/a>)<\/li>\n<\/ol>\n

    Perhaps a more extreme example of this is Adaboost which repeatedly reuses a classifier learner to implicitly optimize a gradient. Viewed as a gradient optimization algorithm, Adaboost is a sublinear<\/em> algorithm (in the number of implicit parameters) when applied to decision trees.<\/p>\n","protected":false},"excerpt":{"rendered":"

    One of the basic observations from the atomic learning workshop is that gradient-based optimization is pervasive. For example, at least 7 (of 12) speakers used the word ‘gradient’ in their talk and several others may be approximating a gradient. The essential useful quality of a gradient is that it decouples local updates from global optimization. … <\/p>\n

    Continue reading “Gradients everywhere”<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[29,26,25],"tags":[],"class_list":["post-178","post","type-post","status-publish","format-standard","hentry","category-machine-learning","category-structured","category-supervised"],"_links":{"self":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts\/178","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=178"}],"version-history":[{"count":0,"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts\/178\/revisions"}],"wp:attachment":[{"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=178"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=178"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=178"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}