ICML Board and Reviewer profiles

The outcome of the election for the IMLS (which runs ICML) adds Emma Brunskill and Hugo Larochelle to the board. The current members of the board (and the reason for board membership) are:

• Andreas Krause (Elected & 2018 Program Chair)
• Andrew McCallum (Past president)
• Bernhard Schoelkopf (Elected)
• Corinna Cortes (Elected)
• David Blei (2020 General chair)
• Doina Precup (2017 Program Chair)
• Emma Brunskill (Elected)
• Eric Xing (Elected & 2019 General Chair)
• Francis Bach (Elected & 2018 General Chair)
• Hanna Wallach (Elected)
• Hugo Larochelle (Elected)
• Jennifer Dy (2018 Program Chair & Secretary)
• Joelle Pineau (Elected & President)
• Kamalika Chaudhuri (2019 Program Chair)
• John Langford (President Elect & 2016 General Chair)
• Kilian Weinberger (Elected & 2016 Program Chair)
• Nina Balcan (Elected & 2021 General Chair & 2016 Program Chair)
• Ruslan Salakhutdinov (Elected & 2019 Program Chair)
• Thorsten Joachims (Elected)
• Tony Jebara (2017 General Chair)
• Yee-Whye Teh (2017 Program Chair)

President Elect is a 2-year position with little responsibility, but I decided to look into two things. One is the website which seems relatively difficult to navigate. Ideas for how to improve are welcome.

The other is creating a longitudinal reviewer profile. I keenly remember the day after reviews were due when I was program chair (in 2012) which left a panic-inducing number of unfinished reviews. To help with this, I’m planning to create a profile of reviewers which program chairs can refer to in making decisions about who to ask to review. There are a number of ways to do this wrong which I’m avoiding with the following procedure:

1. After reviews are assigned, capture the reviewer/paper assignment. Call this set A.
2. After reviews are due, capture the completed & incomplete reviews for papers. Call these sets B & C respectively.
3. Strip the paper ids from B (completed reviews) turning it into a multiset D of reviewers completed reviews.
4. Compute C-A (as a set difference) then turn it into a multiset E of reviewers incomplete reviews.
5. Store D & E for long term reference.

This approach:

• Is objectively defined. Approaches based on subjective measurements seem both fraught with judgment issues and inconsistent. Consider for example the impressive variation we all see in review quality.
• Does not record a review as late for reviewers who are assigned a paper late in the process via step (1) and (4). We want to encourage reviewers to take on the unusual but important late tasks that arrive.
• Does not record a review as late for reviewers who discover they are inappropriate after assignment and ask for reassignment. We want to encourage reviewers to look at their papers early and, if necessary, ask for a paper to be reassigned early.
• Preserves anonymity of paper/reviewer assignments for authors who later become program chairs. The conversion into a multiset removes the paper id entirely.

Overall, my hope is that several years of this will provide a good and useful tool enabling program chairs and good (or at least not-bad) reviewers to recognize each other.