This post is about a confusion of mine with respect to many commonly used machine learning algorithms. <\/p>\n
A simple example where this comes up is Bayes net prediction. A Bayes net where a directed acyclic graph over a set of nodes where each node is associated with a variable and the edges indicate dependence. The joint probability distribution over the variables is given by a set of conditional probabilities. For example, a very simple Bayes net might express: What I don’t understand is the mechanism commonly used to estimate P(A | B, C)<\/em>. If we let N(A,B,C)<\/em> be the number of instances of A,B,C<\/em> then people sometimes form an estimate according to: To compensate, people often try to pick some prior (such as Dirichlet prior with one “virtual count” per joint parameter setting) to provide a reasonable default value for the count. Naturally, in the “big learning” situations where this applies, the precise choice of prior can greatly effect the system performance leading to finicky tuning of various sorts. It’s also fairly common to fit some parametric model (such as a Gaussian) in an attempt to predict A<\/em> given B<\/em> and C<\/em>.<\/p>\n Stepping back a bit, we can think of the estimation of P(A | B, C)<\/em> as a simple self-contained prediction (sub)problem. Why don’t we use existing technology for doing this prediction? Viewed as a learning algorithm “counting with a Dirichlet prior” is exactly memorizing the training set and then predicting according to either (precisely) matching training set elements or using a default. It’s hard to imagine a more primitive learning algorithm. <\/p>\n There seems to be little downside to trying this approach. In low count situations, a general purpose prediction algorithm has a reasonable hope of performing well. In a high count situation, any reasonable general purpose algorithm converges to the same estimate as above. In either case something reasonable happens.<\/p>\n Using a general purpose probabilistic prediction algorithm isn’t a new idea, (for example, see page 57<\/a>), but it appears greatly underutilized. This is a very small modification of existing systems with a real hope of dealing with low counts in {speech recognition, machine translation, vision}. It seems that using a general purpose probabilistic prediction algorithm should be the default rather than the exception.<\/p>\n","protected":false},"excerpt":{"rendered":" This post is about a confusion of mine with respect to many commonly used machine learning algorithms. A simple example where this comes up is Bayes net prediction. A Bayes net where a directed acyclic graph over a set of nodes where each node is associated with a variable and the edges indicate dependence. The … <\/p>\n
\nP(A,B,C) = P(A | B,C)P(B)P(C)<\/em><\/p>\n
\n
\n… in other words, people just estimate P'(A | B,C)<\/em> according to observed relative frequencies. This is a reasonable technique when you have a large number of samples compared to the size space A x B x C<\/em>, but it (naturally) falls apart when this is not the case as typically happens with “big” learning problems such as machine translation, vision, etc…<\/p>\n