A modest proposal – Machine Learning (Theory)

(This is a proposal for a new NIPS review process from Zoubin, lightly edited for html formatting.)

Here is a plan for redesigning reviewing and the relationship between journals and conferences for the field. No, I am not going to propose that we eat our own papers to reduce our reviewing load [with apologies to Jonathan Swift].

Imagine a paired Journal and Conference, with four deadlines per year at 3 month intervals for the Journal, a fairly large, say initially about 300, set of Members of an Editorial College acting as reviewers, and a smaller set of Senior Members overlooking the process. The Conference would occur annually, based primarily on papers accepted to the Journal (plus perhaps a lightly reviewed poster session based on submitted abstracts, like in other fields).

SUBMISSION and REVIEWING

Submitted papers can only be reviewed by Members or other reviewers nominated by Members.

Submitted papers can be any reasonable length (e.g. up to 30 pages) in a standard format.

Members are not allowed to review more than 2 papers by any one author in any given year (to prevent undue positive or negative influence), except for follow-up reviews of resubmissions or revisions.

At submission time, authors have to give keywords and suggest 3 Members as reviewers (this will help with matching but need not be enforced in the assignments).

After each of the 4 annual deadlines, Members can bid for papers still on the to-be-reviewed List. Papers are removed from the List once they have 4 confirmed reviewers. There is incentive for Members to bid, to get good papers and to earn Points so as to remain members (see below).

For a paper to be accepted it needs to get at least 2 accept recommendations from Members.

Importantly, not all papers will garner enough reviewers in this bidding process. These papers remain on the List either until they get enough reviews or they are withdrawn by the authors. The idea is to use supply-and-demand to control reviewing. Members supply reviews, submissions generate demand, submissions that are less interesting or worthwhile will simply not be reviewed.

One month after submission, authors are notified of how many confirmed reviewers their paper has. If their paper did not get any reviewers they can withdraw it and send it elsewhere.

A record of all submissions is kept for future reference. Resubmissions and revisions would either be marked by the authors or automatically flagged based on text and author list. Previously published papers and all previous reviews, responses, etc would be available to reviewers to allow them to judge incrementality, originality, etc.

A more draconian measure would be for any author with a 0% success rate on 5 or more papers to be flagged as a ‘low grade’ submitter for a year. This could be marked on further submissions in that year as useful information for Members.

To further cap the number of submissions, we can place some limits on the number of co-authored papers anyone can submit in a given year, for example: 3 for Non-Members, 5 for Members, and 7 for Senior Members. This is slightly dangerous as place limits can distort the scientific criteria for authorship — authorship should be based on contributions.

MEMBERSHIP in the Editorial College

To maintain your Membership, you have to earn m reviewing Points each year. Points are earned based on the number of papers you review and their length (e.g. [4 + number of pages] points). Reviewing revisions and resubmissions would earn 1/2 as many points since they are presumably easier. Late reviews would give 1/2 the points. The variable ‘m’ could be adapted to ensure that the pool of reviews keeps up roughly with the submissions. Excess Points can be rolled over to subsequent years.

To calibrate reviewers and help ensure quality, other Members reviewing a paper can anonymously flag low-quality reviews by their co-reviewers. This information can go to the Senior Members of the Editorial College, and perhaps be used to automatically discount points from Members, or ‘retire’ poorly performing Members.

Membership would be transparent with clear demographic (affiliation, gender, country, etc) and keyword information publicly available. This could be scrutinised for perceived bias and to ensure appropriate diversity. New Members and retirements are kept up to date or even publicly announced at regular intervals.

Models could be developed for suggesting assignments, reviewer calibration, figuring out new areas in need of reviewers, etc, based on well-collated information on submissions and members over several years.

ADVANTAGES of Membership

Being a Member of course gives some level of influence on the field, is good for the CV, etc.

Members would be able to write short published comments on the final version of any paper (e.g. papers they reviewed)–for example pointing out relevant literature, flaws, or giving praise.

We could offer some Conference registration discount to Members since they have obviously contributed.

Best paper awards at the Conference could be given with the benefit of hindsight to papers with a substantial number of praising comments from Members.

After three years as a Member in good standing, you become eligible for election to Senior Membership.

ELECTION to Membership in the Editorial College

To be eligible for election to the College, you have to satisfy one of:

have been asked to review by n different Members in the last year (which suggests your area is in demand).
have been nominated by n’ different Members (assume we limit 2 nominations per Member)
have co-authored n” papers in the Journal or Conference in the last 3 years.
have been cited by n”’ papers in the J or C in the last 3 years.

A list of eligible people would be circulated to Members with these numbers, i.e. Jane Doe (3 reviews, 2 nominations, 4 citations… etc), and Members would vote for new members based on this information.

SENIOR MEMBERSHIP

Senior Members have mostly similar rights and responsibilities as Members, except:

they require fewer reviewing Points to maintain Senior Membership.
they may have to serve on various panels, e.g. Ethics panel, Best Paper award panels, etc.
they can contribute one short un-reviewed paper per year to the Journal, say an opinion piece etc.

The CONFERENCE

Once a paper is accepted to the Journal, if there is enough space, authors are invited to give a talk/poster at the annual Conference, and perhaps include a shorter, say 4 page, conference version of the paper in the Conference proceedings.

In addition, the Conference could include short abstracts presented as posters, but not appearing as publications.

SUMMARY

There are a lot of distinct ideas in these proposals, but the basic thought in my mind is that we need to create a market based approach for reviewing and for increasing quality of submissions. I’m proposing a market based not on money but on Membership of the Editorial College, where the actions of this College will determine which papers end up getting enough reviewers to be accepted. The goal is to create an open, transparent and democratic system that can adapt to the demands of a growing field.

Zoubin Ghahramani
Professor of Information Engineering
University of Cambridge
zoubin@eng.cam.ac.uk