The prevailing wisdom in machine learning seems to be that motivating a paper is the responsibility of the author. I think this is a harmful view—instead, it’s healthier for the community to regard this as the responsibility of the reviewer.
There are lots of reasons to prefer a reviewer-responsibility approach.
- Authors are the most biased possible source of information about the motivation of the paper. Systems which rely upon very biased sources of information are inherently unreliable.
- Authors are highly variable in their ability and desire to express motivation for their work. This adds greatly to variance on acceptance of an idea, and it can systematically discriminate or accentuate careers. It’s great if you have a career accentuated by awesome wording choice, but wise decision making by reviewers is important for the field.
- The motivation section in a paper doesn’t do anything in some sense—it’s there to get the paper in. Reading the motivation of a paper is of little use in helping the reader solve new problems.
- Many motivation sections are a waste of time. The 30th paper on a subject should not require a motivation as if it’s the first paper on a subject, and requiring or expecting this of authors is an exercise in busy work by the research community.
Some caveats to make sure I’m understood:
- I’m not advocating the complete removal of a motivation section (motivectomy?), which would be absurd (and frankly harmful to your career). A paragraph describing common examples where the problem addressed comes up is desirable for readers who are not specialists. This paragraph should not be in the abstract, where it seems to often sneak in.
- I’m also not arguing against discussion of motivations. I regard discussion of motivations as quite important, and totally unsuited to the paper format. It’s hard to imagine any worse method for discussion than one with a year-size latency where quasi-anonymous people are quasi-randomly paired and each attempts to accomplish several different tasks one of which happens to be a one-sided discussion of motivation. A blog can work much better for this sort of thing, and I definitely invite discussion on motivational questions.
So, how do we change the prevailing wisdom? The answer is always “gradually”, but there are a number of steps we can take.
- As an author, one clever technique is to pass serious discussion of motivation by reference. “For a general discussion and motivation of this problem see [].” This would save space in the large number of papers which attempt to address an old problem better than previous approaches.
- Participate in public discussion of motivations. We need to encourage a real mechanism for discussion. Until these alternative (and far better) formats for discussion are developed the problem of “who motivates” will always exist.
- Have private discussions about motivation where you can. Random conversations at conferences are great for this, and the process often sharpens your appreciation.
- Learn to take responsibility for motivation as a reviewer. This might sound hard, but it’s actually somewhat easier than careful evaluation of technical content in my experience.
- The first step is to disbelieve all the motivational parts of a paper by default. As mentioned above, the authors are not a reliable source anyways. Skip it and move on.
- Make sure you understand the problem being addressed.
- Make sure you understand how well the problem is addressed, relative to previous work.
- Think about how important that increment is. This is not equivalent to asking “how many people will appreciate the increment?” which is a popularity question. Frankly, all of Machine Learning fails the popularity test in a wider sense, even though many people appreciate the fruits of machine learning on a daily basis. First, think about the problem.
- How many people might a solution to the problem help? 0 is fairly common amongst submitted papers.
- How much would it help them? If it’s “alot”, then that should add a bit to the importance of the paper.
- How familiar are you with the problem? If not very, then it’s appropriate to give the benefit of the doubt to the authors.
Think about the solution.
- This solution might be useful to some other researchers who come up with something useful. This is a a warning sign.
- This solution might be useful to me in coming up with a useful algorithm for solving problems.
- This paper improves an algorithm. This is also fairly common. It should be improving an algorithm with a reasonable claim at being the best method for solving some problem.
- This paper can provide improvements to many algorithms. Theory papers often fall here, but they can also fall under (1) or (2) easily.
Now, take these considerations into account in forming your own opinion about how motivated the paper is.
- Go multimodel. If you only know one model of what machine learning is, you don’t really know machine learning. Learn multiple ideas of what machine learning are, and actively consider their merits and downsides.
As a reviewer and a reader, I tend to agree with you. I often completely skip the intro and motivation sections, and start reading from the section that defines the model (of course this also depends on whose paper I’m reading. Some authors have a reputation for overhyping their results, while a few write insightful motivations that help you get the big picture).
But as an author, I think still the best strategy is to put a lot of effort in motivating the results. In fact, at least for theory papers, I think motivation plays a more important role in the fate of the paper than the technical contents. I had a paper resubmitted to FOCS after being rejected from STOC. The resubmitted version, which was basically the same as the original, with some change to the intro to motivate the problem better, was accepted. It was quite amusing to compare the reviews I got from the two conferences; the STOC reviews were mostly saying that the results are interesting but the problem is not important, while the FOCS reviews were all very enthusiastic about the problem, and were only complaining that the result is not strong enough. So, at least in the theory community, I can’t trust the reviewer to motivate the paper.
I personally find the review process RANDOM, at all conferences, ML or Theory ones.
(I thought I posted already here, but I don’t see the post).
I very strongly disagree with this post. Motivation makes the difference between knowing whether there is any real application to the paper, or if it’s purely abstract. It’s to everyone’s benefit that authors make things as easy as possible for the reader (within space constraints), and this includes understanding why the paper was written. I wouldn’t ask the reviewer guess what motivates the paper just as I wouldn’t guess them to ask, e.g., what the different symbols stand for or what the experimental setup is.
I get the impression from your post that you’re really referring to a broad class of papers that all attack the same, well-established problems. Certainly, there’s no point in having every paper being “Machine learning has become a fundamental tool in many problems…” But it is important to discuss the particular motivation of the work. This often happens with methods motivated by a particular application. To pick one random example, Roweis and Globerson’s “Nightmare at Test Time” paper (in which the learning must be robust to an adversary that gets to delete features) might seem to be a purely intellectual curiousity if you didn’t know that it’s motivated by spammers deliberately designing emails to foil learned spam filters.
Motivation is most important for the more innovative and wacky papers, ones that don’t fit in the standard ways of doing things, or that target new applications. Removing motivation encourages conservativism, and reviewing processes are conservative enough as it is. It is a very bad idea to leave anything to the imagination of your reviewers.
Motivation also provides a standard for evaluation for your paper. If your method is designed to solve classification problem X, and it does, then it will be less likely to be shot down by some reviewer complaining that it doesn’t solve classification problem Y (they might complain that problem X isn’t interesting, but at least then the discussion is more focused).
When I review a paper, I don’t take the motivation given on face value, and, if it is lacking, I do try to imagine what the motivation might be. But I generally find that poorly-motivated papers often lack a good application (the applications I can think of would often be solved better by some existing technique); authors who haven’t thought carefully about what their work is good for are often solving non-problems.
The motivation section in a paper doesn’t do anything in some senseâ€â€it’s there to get the paper in. Reading the motivation of a paper is of little use in helping the reader solve new problems.
There are many papers in which the Introduction is the most interesting and stimulating part of the paper; where the paper proposes a new way of looking at problems or seeing the problem domain. This is the part of the paper where the authors can contribute a larger view of the field and where research should be going beyond just the particular experiments they did or the model that they set up.
The post has several points, so I’m not sure which you disagree with (or maybe it’s all of them?). Do you think authors should spend much more than a paragraph on motivation? Do you think reviewers should not be responsible for thinking about motivations? Do you think it’s healthy to have a system where the expectation is that motivation is entirely abdicated to authors?
The Globerson and Roweis paper seems to put motivation in the abstract (which I don’t like), but it otherwise spends about a paragraph (or maybe a bit more) on motivation inline with the post’s suggestion.
Maybe it would be help to be clear by what is meant by “motivation.” I read that word as meaning all explanation as to why the work is being done, as opposed to the mathematical/algorithmic formulation that represents the meat of the paper. I believe the motivation is essential to paper, whether it takes one sentence or two pages to do justice to.
> Do you think authors should spend much more than a paragraph on motivation?
I think this depends on the paper. If you’re writing a paper on a very well-established problem—say, classification—you don’t need to say why classification is important. However, you should motivate your twist on it, e.g., if your paper improves on some property of classification methods, you should say why that property is important and why it hasn’t been adequately addressed.
For papers that move in a new direction, tackle or introduce new applications or ways of thinking about the problem, a motivation of more than one paragraph may be appropriate—whatever it takes to make it clear to the reader why the work is interesting.
> Do you think reviewers should not be responsible for thinking about motivations? Do you think it’s healthy to have a system where the expectation is that motivation is entirely abdicated to authors?
Authors have a responsibility to write every part of the paper clearly and correctly (motivation, body, equations, experiments, etc.), and reviewers have a responsibility to evaluate all parts of the paper. This applies to the motivation: authors should supply one, and reviewers should evaluate it. A generous reviewer may attempt to fill the gap if no satisfactory motivation is provided, but it’s in everyone’s best interest that the authors supply a compelling and convincing motivation.
I should say that I hadn’t read the Globerson and Roweis paper recently before my comment, and I see that it’s more than just spam filtering. My point remains that real-world motivation is absolutely essential (whether it be one sentence or several paragraphs); otherwise, their model seems completely contrived.
Here are some good papers with slightly longer (and definitely worthwhile) motivations (I don’t know the ML literature well enough to find examples quickly):
http://grail.cs.washington.edu/pub/papers/Jacobs2003.pdf
http://grail.cs.washington.edu/pub/papers/phototour.pdf
I think we do simply disagree to some extent. I regard a reviewer’s role as more active than just evaluation, as detailed in the post.
In my experience, having an objective sense of importance for research work rather than one strongly controlled by authors/speakers is pretty valuable.
I do agree that streams of papers aren’t the best medium for tracking motivation, and what the state of the art is. Perhaps instead of depending on the author or the reviewer, they can be offloaded (http://aclweb.org/aclwiki/index.php?title=State_Of_The_Art), and that many papers could get away with just citing what they believe is the current state of the art (as of x date in the ACL state-of-the-art wiki). It might also then be easier to evaluate new problems, and maybe even make them easier to discover (hey, there is no problem with tags w, v, and y;Hmmm…). Cultivating an objective sense of the research might be much easier with such a system.
We can all agree with Aaron Hertzmann that we don’t need boilerplate motivation. It’s worse than useless — it’s mind numbing and offputting. But you do need to do enough of a sales job in the title and abstract that I’ll read your paper. As I used to tell my CMU students, academic papers aren’t short stories; they need to draw in the audience. If you can’t hook me with the abstract, I’m not likely to read the paper.
Science just isn’t as objective as scientists like to believe. Specifically, John Langford might as well search for the Holy Grail as an “objective sense of importance for research work”. The subjectivity is not just limited to comparing incremental improvements to paradigm shifting big ideas. Does anyone else remember the raging debates about whether “all this counting stuff” would ever amount to anything in the 1980s? Ken Church’s seminal paper on statistical tagging didn’t even get the Bayes’ Law right, but it helped change the direction of the field by showing what was possible.
So who decides what’s important? It’s not a cabal of editors who have it in for neural networks (or whatever you perceive biases to be). It’s the result of the whole research community having discussions on blogs, in elevators, in classes, and yes, even in papers. That’s why I also used to tell my students to go out to dinner with speakers and meet with them whenever they could; there’s no better way of finding out what’s “hot”.
For a dose of practical advice, I can recommend Richard Hamming’s You and Your Research. Pay attention to the part on selling your research and also the networking/schmoozing aspects. I’m thinking he’d have loved blogs like this one.
And to the anonymous poster who thought editing was random, it’s not totally random. It just has a very high bias and variance. So much so that the decision boundary is fairly arbitrary when subjected to a bootstrap variance estimate on reviewer ratings. The best thing to do to mitigate the variance is provide a good motivation putting the paper in context. Nothing you can do about reviewer bias other than try to make sure it goes to the right reviewer with a clear abstract.
I am not the first Anonymous, but I don’t think it will be very assuring for him to hear “It just has a very high bias and variance.”. I am not sure if this has been discussed here before, but I think it merits a discussion.
There are a lot of bias/variance reduction techniques in conference reviewing, such as anonymized reviewing, rebuttals, having multiple committee members look at each paper, having all papers discussed in committee, having a discussion period, having the process overseen by chairs for fairness, etc.
In all of the PCs I’ve been on, I felt like people were trying very hard to be fair and to make good decisions. (I’ve only been on committees for graphics conferences). Objectivity is impossible, but most people (with perhaps a few exceptions) really seem to try hard and take the process seriously. There is still a lot of randomness, but, in my opinion, 99% of this happens in cases that are very hard to judge or have significant elements of subjectivity anyway.
I actually disagree with this post very much, although perhaps I’m interpreting it too strongly. Choosing a problem with high impact is as central to a researcher’s job as proving theorems or running experiments. Therefore, I would expect them to convince me that their contribution is important, just as I expect them to convince me that that their theorems are correct.
Caveats:
* I am in no way advocating boilerplate paragraphs like “Classification is an important problem because…” I mean tell me why YOUR new classifier is important. In general, when I say “motivate your problem” I mean “tell me why your particular delta to the literature is important”.
* If a quality paper is poorly motivated in the text, but a reviewer can still see that it will have high impact, then the paper should be accepted. (Just as a correct paper might be accepted even if some other aspect of the paper is poorly explained.)
* Clearly the 30th paper on a topic should not have the same motivation as the first. In particular, the 30th paper had d@#n well better explain why the previous 29 papers weren’t good enough.