{"id":627,"date":"2009-03-18T04:47:23","date_gmt":"2009-03-18T10:47:23","guid":{"rendered":"http:\/\/hunch.net\/?p=627"},"modified":"2009-03-18T04:47:23","modified_gmt":"2009-03-18T10:47:23","slug":"parallel-ml-primitives","status":"publish","type":"post","link":"https:\/\/hunch.net\/?p=627","title":{"rendered":"Parallel ML primitives"},"content":{"rendered":"

Previously, we discussed parallel machine learning<\/a> a bit. As parallel ML is rather difficult, I’d like to describe my thinking at the moment, and ask for advice from the rest of the world. This is particularly relevant right now, as I’m attending a workshop tomorrow on parallel ML.<\/p>\n

Parallelizing slow algorithms seems uncompelling. Parallelizing many algorithms also seems uncompelling, because the effort required to parallelize is substantial. This leaves the question: Which one fast algorithm is the best to parallelize? What is a substantially different second?<\/p>\n

One compellingly fast simple algorithm is online gradient descent on a linear representation. This is the core of Leon’s sgd<\/a> code and Vowpal Wabbit<\/a>. Antoine Bordes<\/a> showed a variant was competitive in the large scale learning challenge<\/a>. It’s also a decades old primitive which has been reused in many algorithms, and continues to be reused. It also applies to online learning<\/i> rather than just online optimization<\/i>, implying the algorithm can be used in a host of situations where batch algorithms are awkward or unviable.<\/p>\n

If we start with a fast learning algorithm as a baseline, there seem to be two directions we can go with parallel ML:<\/p>\n

    \n
  1. (easier) Try to do more in the same amount of time. For example, Paul and Neil<\/a> suggest using parallelism to support ensemble methods.<\/li>\n
  2. (harder) Try to use parallelism to reduce the amount of time required to effectively learn on large amounts of data. For this approach, bandwidth and latency become uncomfortably critical implementation details. Due to these issues, it appears important to at least loosen the goal to competing with learning on large amounts of data. Alternatively, we could consider this as evidence some other learning primitive is desirable, although I’m not sure which one.<\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"

    Previously, we discussed parallel machine learning a bit. As parallel ML is rather difficult, I’d like to describe my thinking at the moment, and ask for advice from the rest of the world. This is particularly relevant right now, as I’m attending a workshop tomorrow on parallel ML. Parallelizing slow algorithms seems uncompelling. Parallelizing many … <\/p>\n

    Continue reading “Parallel ML primitives”<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[37,29],"tags":[],"class_list":["post-627","post","type-post","status-publish","format-standard","hentry","category-computation","category-machine-learning"],"_links":{"self":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts\/627","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=627"}],"version-history":[{"count":0,"href":"https:\/\/hunch.net\/index.php?rest_route=\/wp\/v2\/posts\/627\/revisions"}],"wp:attachment":[{"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=627"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=627"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/hunch.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=627"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}