Machine Learning (Theory)

5/2/2005

Reviewing techniques for conferences

Tags: Reviewing jl@ 9:28 pm

The many reviews following the many paper deadlines are just about over. AAAI and ICML in particular were experimenting with several reviewing techniques.

  1. Double Blind: AAAI and ICML were both double blind this year. It seemed (overall) beneficial, but two problems arose.
    1. For theoretical papers, with a lot to say, authors often leave out the proofs. This is very hard to cope with under a double blind review because (1) you can not trust the authors got the proof right but (2) a blanket “reject” hits many probably-good papers. Perhaps authors should more strongly favor proof-complete papers sent to double blind conferences.
    2. On the author side, double blind reviewing is actually somewhat disruptive to research. In particular, it discourages the author from talking about the subject, which is one of the mechanisms of research. This is not a great drawback, but it is one not previously appreciated.
  2. Author feedback: AAAI and ICML did author feedback this year. It seemed helpful for several papers. The ICML-style author feedback (more space, no requirement of attacking the review to respond), appeared somewhat more helpful and natural. It seems ok to pass a compliment from author to reviewer.
  3. Discussion Periods: AAAI seemed more natural than ICML with respect to discussion periods. For ICML, there were “dead times” when reviews were submitted but discussions amongst reviewers were not encouraged. This has the drawback of letting people forget their review before discussing it.
3 Comments to “Reviewing techniques for conferences”
  1. I take the opportunity to start a debate about theoretical papers in conferences. In the recent years, the level of technicality of theoretical machine learning papers has increased significantly. Nowadays, most results involve proofs and technical details that do not fit in the standard conference format.
    So, authors typically put the proofs aside (e.g. in a tech.rep.) but this is not satisfactory because reviewers either have to trust the proofs or to read this extra material.
    One possible approach would be to have 2 different status for papers:
    1) Completely checked: for self-contained papers that can be read from beginning to end by the reviewers
    2) Partially checked: for papers that do not contain the proofs

    This status should be clear in the proceedings. One way would be to simply skip from the proceedings the partially checked papers, or include them as ‘abstracts’ only.

    Actually, this is what happens in mathematics: most conferences do not have proceedings edited before the conference. Authors simply submit abstracts (announcing the results without proofs) and present their results at the conference, but then they have to write a full paper (of the same format as a journal paper) and the papers are reviewed for inclusion in a collection.
    So, presenting a result at a conference is not considered as a publication. And I think this is perfectly fair for technical papers.

    I have tried to push those ideas at COLT, but there were little echo.

    What is the opinion of people reading this blog ???

  2. Part of the difficulty with this proposal is that it is anti-traditional. Traditionally, in computer science conference papers are fully reviewed. I’ve seen some evidence that this tradition is weaker than the tradition for journal papers, but it is still significant. (I certainly try.) When this tradition is removed, the conference typically changes it’s title to “workshop”. Workshops are great for doing research with other people but they aren’t “publishing”.

    My understanding about why conference publication is preferred over journal publication is:
    1) It provides an excuse to talk to other people and announce your work broadly.
    2) It’s way faster. There are hard deadlines involved, so everyone (reviewers, authors, program committee) accepts that decisions need to be made now. My experience is that even when journals try to be fast, the lack of hard deadlines leads to delays.

    I have a counterproposal. My understanding is that currently even complete technical conference papers are always reviewed for interest and sometimes reviewed for correctness, depending on the reviewer (with typically 3-4 reviewers). This is a bit haphazard. A different system with the same overall work would be to name one of the reviewers “primary” and the rest “secondary”. The primary reviewer would be responsible for determining correctness while the secondary reviewers would be responsible for discussing relevance, interest, and significance.

    Thus, instead of getting 10 papers to review, you the reviewer would get 3 to review in detail and 7 to review at the introduction level. The big question is: Does this reduce or increase the variance of the review process? I suspect “reduce”, but I have no experience.

  3. Anonymous says:

    The reason its ok in Math is that conference publications are meaningless—their “conferences” much like our workshops, are simply excuses to meet people and papers are really published only in journals. That of course has its advantages and disadvantages.

    CS is, in my opinion, a more collaborative and a faster moving area and one of the reasons for that is the importance we ascribe to conferences. At the same time, we accept that one price to pay is that once in a while, there will be a paper that is hard to verify and we might end up accepting a wrong paper. In most cases, technically difficult and/or important papers do end up being submitted to a journal where they are more thoroughly reviewed. I find this tradeoff acceptable and would rather not move to the math style journal-dominant system.

    At the same time, we should do what we can to alleviate the problems with the refereeing process. John’s proposal seems like a good idea, as do author feedback and discussion periods. Making it double blind requires considerable effort and may not be worth the payoff.

    I think the page limit is also often a hindrance since authors end up wasting hours trying to fit the material into the limit; moving stuff to the appendix, cutting out proofs, decreasing the margins, etc. This often also makes the paper harder to read for the reviewer. No easy fixes though.

Leave a Reply


Powered by WordPress