{"id":269,"date":"2007-05-12T09:00:17","date_gmt":"2007-05-12T15:00:17","guid":{"rendered":"http:\/\/hunch.net\/?p=269"},"modified":"2007-05-12T09:00:17","modified_gmt":"2007-05-12T15:00:17","slug":"loss-function-semantics","status":"publish","type":"post","link":"https:\/\/hunch.net\/?p=269","title":{"rendered":"Loss Function Semantics"},"content":{"rendered":"

Some loss functions have a meaning, which can be understood in a manner independent of the loss function itself. <\/p>\n

    \n
  1. Optimizing squared loss lsq<\/sub>(y,y’)=(y-y’)2<\/sup><\/em> means predicting the (conditional) mean of y<\/em>.<\/li>\n
  2. Optimizing absolute value loss lav<\/sub>(y,y’)=|y-y’|<\/em> means predicting the (conditional) median of y<\/em>. Variants can handle other quantiles<\/a>. 0\/1 loss for classification is a special case.<\/li>\n
  3. Optimizing log loss llog<\/sub>(y,y’)=log (1\/Prz~y’<\/sub>(z=y))<\/em> means minimizing the description length of y<\/em>.<\/li>\n<\/ol>\n

    The semantics (= meaning) of the loss are made explicit by a theorem in each case. For squared loss, we can prove a theorem of the form:
    \nFor all distributions D<\/em> over Y<\/em>, if

    y’ = arg miny’<\/sub> Ey ~ D<\/sub> lsq<\/sub> (y,y’)<\/em><\/center> then
    y’ = Ey~D<\/sub> y<\/em><\/center><\/p>\n

    Similar theorems hold for the other examples above, and they can all be extended to predictors of y’<\/em> for distributions D<\/em> over a context X<\/em> and a value Y<\/em>.<\/p>\n

    There are 3 points to this post.<\/p>\n

      \n
    1. Everyone doing general machine learning should be aware of the laundry list above. They form a handy toolkit which can match many of the problems naturally encountered.<\/li>\n
    2. People also try to optimize a variety of other loss functions. Some of these are (effectively) a special case of the above. For example, “hinge loss” is absolute value loss when the hinge point is at the upper range. Some of the other losses do not have any known semantics. In this case, discovering a semantics could be quite valuable.<\/li>\n
    3. The natural direction when thinking about how to solve a problem is to start with the semantics you want and then derive a loss. I don’t know of any general way to do this other than simply applying the laundry list above. As one example, what is a loss function for estimating the mean of a random variable y<\/em> over the 5th to 95th quantile? (How do we do squared error regression which is insensitive to outliers?) Which semantics are satisfiable with a loss?<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"

      Some loss functions have a meaning, which can be understood in a manner independent of the loss function itself. Optimizing squared loss lsq(y,y’)=(y-y’)2 means predicting the (conditional) mean of y. Optimizing absolute value loss lav(y,y’)=|y-y’| means predicting the (conditional) median of y. Variants can handle other quantiles. 0\/1 loss for classification is a special case. … <\/p>\n

      Continue reading “Loss Function Semantics”<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[29,16],"tags":[],"class_list":["post-269","post","type-post","status-publish","format-standard","hentry","category-machine-learning","category-problems"],"_links":{"self":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts\/269","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=269"}],"version-history":[{"count":0,"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts\/269\/revisions"}],"wp:attachment":[{"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=269"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=269"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=269"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}