uai2008@helsinki.fi
ICML
COLT

INSTRUCTIONS FOR THE UAI 2008 SENIOR PROGRAMME COMMITTEE

Important dates

Monday, March 3:Bidding opens
Monday, March 10:Bids due
Friday, March 14:Allocation to reviewers, reviewing starts
Friday, March 28:Initial reviews due, discussion period starts
Monday, April 7:Final reviews due
Monday, April 14:SPC recommendations due
Monday, April 21:Author notifications

Frequently asked questions

NOTE: Please examine the PC FAQ list first. Most questions are addressed in the general review FAQ list. This list is for SPC-specific questions.

  1. How many papers should I bid on?
  2. How do I assign papers to reviewers?
  3. Are SPC members also expected to write reviews for the papers they are assigned?
  4. How do I enter a review if I want/need to do so?
  5. What should be the first thing I do once the discussion period starts?
  6. How do I initiate discussion?
  7. What are the goal of the discussion period?
  8. What if a review is missing?
  9. How do I conclude a discussion?
  10. What happens after the discussion period?
  11. What are the criteria for deciding on a poster vs. plenary recommendation?
  12. What do I do with the "ballot" section on the right side of the ConfMaster window?

Answers

  1. How many papers should I bid on?

    SPC members should give positive bids to at least 30 papers. Bidding on more papers increases the chances that you will get a paper you want.

  2. How do I assign papers to reviewers?

    3. You don't. For UAI this is the job of the program chairs.

  3. Are SPC members also expected to write reviews for the papers they are assigned?

    This is not required in general, but there are a few cases where it may be necessary:

    • If you are unsatisfied with the quality of the reviews provided the your PC members and have been unable to coax them into doing a better job, you might want to add your own review as a courtesy to the authors.
    • If the reviews disagree or are inconclusive, you may want to add a brief summary review that emphasizes the most important points from the reviews and discussion.

    In general, the guiding principle for your actions should be to ensure that the authors get a clear and fair response to their submission.

  4. How do I enter a review if I want/need to do so?

    Click on "view assigned papers" then click on the circled R on the right to enter your review.

  5. What should be the first thing I do once the discussion period starts?

    Read the reviews. (Click on Ballot and Discussion on the left, and the click on the circled D on in the column on the right.) Look out for the following:

    • Disagreement among the reviewers.
    • Apathy among the reviewers (see below).
    • Evidence of bias, rudeness or unprofessional content in reviews.
    • Anything that reveals the identity of the reviewers in the reviews.
    • Technical or other errors in the reviews.
    • An overly narrow interpretation of UAI's scope. (See What are the review criteria?)
    • Unhelpful reviews.

    All of the above should be addressed through discussion and, if necessary, personal emails to the reviewers. If you are unable to resolve these issues in a satisfactory manner, contact the program chairs.

    Regarding apathy: We want to accept papers because they are good, not just because they fail to be bad. Watch out for short reviews, middle-of-the-road scores, or reviews that have low confidence. Ideally a paper should have no such reviews and if it has more than one, there could be a serious problem. Push the reviewers to take a position. If there is an issue with the qualifications of the reviewers to review the paper contact the program chairs.

  6. How do I initiate discussion?

    • Click on Ballot and Discussion in the column on the left side of your window. This will list all of the papers you have been assigned.
    • Click on the circled D on the right. This will list all of the reviews for the paper.
    • Type comments in the text box on the bottom and click the submit button.

  7. What are the goal of the discussion period?

    You should use the discussion period to resolve any questions raised by the reviewers and push the reviewers toward a consensus. If this is not possible, you should make sure that the main pros and cons are elicited from the reviewers and they you have enough information to make a recommendation to the program chairs. (If this doesn't happen, you will need to to set aside time for your own careful reading of the paper, so you should be highly motivated to manage the discussion well.)

    If the reviewers are already in agreement and no further action is required, you should still send out a single "discussion" message indicating this fact and thanking the reviewers for their effort.

  8. What if a review is missing?

    You may want to hold off starting discussion until all three reviewers have completed their reviews (but don't wait until the second week of discussion period to start discussion if you are still missing a review). The most important thing for you to do now is to prod your tardy reviewer(s). The email addresses of all reviewers appear on the top of the page showing the reviews. Cut and paste this into your mailer and start pestering. You are welcome to CC us if you want. Note: Don't be confused by the "Reviewer Management" link on the left of the screen. This isn't any help to SPC members since the reviewer assignments were done by the chairs.

  9. How do I conclude a discussion?

    When the discussion has reached a conclusion (or reached an impasse with a clear statement of the pros and cons) then inform the reviewers that you are concluding the discussion and, as a courtesy, inform them of your recommendation to the program chairs. "Borderline" is a possible recommendation, but we hope that you will use the week after the discussion period to refine such recommendations.

  10. What happens after the discussion period?

    After the conclusion of the discussion period, you will have a week to prepare your final report for the program chairs. Your final report should contain the following:

    • A sorted list of papers from strongest to weakest.
    • Your recommendation on acceptance or rejection for each paper.
    • Your recommendation on poster vs. plenary presentation for papers that might be accepted.
    • A few sentence summary of each paper that includes the paper's main strengths and weaknesses.

    Please email your reports directly to pc-chairs@uai2008.confmaster.net.

    If you have managed the discussion period well, this should be a relatively easy task. However, it may be necessary for you to read a few papers carefully yourself if the reviewers were not able to come to a consensus. It also may be necessary for you to write a summary review or meta-review and enter it into the ConfMaster system if you feel the authors will not get a clear message from the three reviews already in the system.

  11. What are the criteria for deciding on a poster vs. plenary recommendation?

    A plenary presentation should be a high quality paper that would be of interest to a significant portion of the UAI community. A paper would be better suited for a poster if the results are more narrow in scope or appeal. Poster papers shouldn't necessary be viewed as lower quality papers - just papers with a narrower audience.

  12. What do I do with the "ballot" section on the right side of the ConfMaster window?

    You don't need to do anything with it. Your report to the program chairs will be used instead of ConfMaster's ballot system.