A new direction of research seems to be arising in machine learning: Interactive Machine Learning. This isn’t a familiar term, although it does include some familiar subjects.
What is Interactive Machine Learning? The fundamental requirement is (a) learning algorithms which interact with the world and (b) learn.
For our purposes, let’s define learning as efficiently competing with a large set of possible predictors. Examples include:
- Online learning against an adversary (Avrim’s Notes). The interaction is almost trivial: the learning algorithm makes a prediction and then receives feedback. The learning is choosing based upon the advice of many experts.
- Active Learning. In active learning, the interaction is choosing which examples to label, and the learning is choosing from amongst a large set of hypotheses.
- Contextual Bandits. The interaction is choosing one of several actions and learning only the value of the chosen action (weaker than active learning feedback).
More forms of interaction will doubtless be noted and tackled as time progresses. I created a webpage for my own research on interactive learning which helps define the above subjects a bit more.
What isn’t Interactive Machine Learning?
There are several learning settings which fail either the interaction or the learning test.
- Supervised Learning doesn’t fit. The basic paradigm in supervised learning is that you ask experts to label examples, and then you learn a predictor based upon the predictions of these experts. This approach has essentially no interaction.
- Semisupervised Learning doesn’t fit. Semisupervised learning is almost the same as supervised learning, except that you also throw in many unlabeled examples.
- Bandit algorithms don’t fit. They have the interaction, but not much learning happens because the sample complexity results only allow you to choose from amongst a small set of strategies. (One exception is EXP4 (page 66), which can operate in the contextual bandit setting.)
- MDP learning doesn’t fit. The interaction is there, but the set of policies learned over is still too limited—essentially the policies just memorize what to do in each state.
- Reinforcement learning may or may not fit, depending on whether you think of it as MDP learning or in a much broader sense.
All of these not-quite-interactive-learning topics are of course very useful background information for interactive machine learning.
Why now? Because it’s time, of course.
- We know from other fields and various examples that interaction is very powerful.
- From online learning against an adversary, we know that independence of samples is unnecessary in an interactive setting—in fact you can even function against an adversary.
- From active learning, we know that interaction sometimes allows us to use exponentially fewer labeled samples than in supervised learning.
- From context bandits, we gain the ability to learn in settings where traditional supervised learning just doesn’t apply.
- From complexity theory we have “IP=PSPACE” roughly: interactive proofs are as powerful as polynomial space algorithms, which is a strong statement about the power of interaction.
- We know that this analysis is often tractable. For example, since Sanjoy’s post on Active Learning, much progress has been made. Several other variations of interactive settings have been proposed and analyzed. The older online learning against an adversary work is essentially completely worked out for the simpler cases (except for computational issues).
- Real world problems are driving it. One of my favorite problems at the moment is the ad display problem—How do you learn which ad is most likely to be of interest? The contextual bandit problem is a big piece of this problem.
- It’s more fun. Interactive learning is essentially a wide-open area of research. There are plenty of kinds of natural interaction which haven’t been formalized or analyzed. This is great for beginnners, because it means the problems are simple, and their solution does not require huge prerequisites.
- It’s a step closer to AI. Many people doing machine learning want to reach AI, and it seems clear that any AI must engage in interactive learning. Mastering this problem is a next step.
- For natural interaction form [insert yours here], how do you learn? Some of the techniques for other methods of interactive learning may be helpful.
- How do we blend interactive and noninteractive learning? In many applications, there is already a pool of supervised examples around.
- Are there general methods for reducing interactive learning problems to supervised learning problems (which we know better)?