INSTRUCTIONS FOR THE UAI 2008 PROGRAMME COMMITTEE
Important dates
| Monday, March 3: | Bidding opens |
| Monday, March 10: | Bids due |
| Friday, March 14: | Allocation to reviewers, reviewing starts |
| Friday, March 28: | Initial reviews due, discussion period starts |
| Monday, April 7: | Final reviews due |
| Monday, April 14: | SPC recommendations due |
| Monday, April 21: | Author notifications |
Frequently asked questions
- What's the difference between being a reviewer and being on the program committee?
- How do I bid on papers?
- When does this bidding period end?
- When are the reviews due?
- How many papers should I bid on?
- Is there a way to view more than one abstract at a time?
- Should I bid on papers that look interesting to me even if I'm not an expert in the area?
- What constitutes a conflict of interest?
- What if I feel unqualified to review a paper that I have been assigned?
- Why did I get this strange paper that I did not bid on?
- What if I suspect plagiarism or a duplicate submission?
- What are the review criteria?
- Are we grading on a curve? (Should the quality of the other papers in my batch influence my review?)
- May I delegate a review to one of my students or colleagues?
- What is the appropriate tone for a review?
- What is the role of the discussion period?
- Are there different standards for poster papers and plenary papers?
- What is confidential?
- How do I participate in discussion?
- Should I initiate a discussion, or should I wait for the SPC member to start one?
- Is it Ok to email the other reviewers or the SPC member directly?
- Can authors see the discussion comments?
- Why aren't the other reviewers cc'd on the discussion emails that ConfMaster sends out?
- How do I interpret the 1-10 scale of the scores?
- What does the "Confidential (visible to PCC, PCM, SV only)" box mean?
- How is the average score of a paper computed?
- Does UAI have a "short paper" category?
- Does UAI conditionally accept papers?
Answers
- What's the difference between being a reviewer and being on the program committee?
- ConfMaster Reviewers = UAI PC
- ConfMaster PC = UAI Senior PC
- How do I bid on papers?
- Log in
- Click on Apply for papers
- Use the keyword boxes to control the set of papers displayed, or stick with your default choices based upon the keywords you provided to ConfMaster when you created your account.
- Scroll through the list of paper titles, clicking on the magnifying class on the right hand side of the page to view abstracts
- If you want to view more than 20 items per page, click on 50 or all on the top right.
- Click on the bottom right to see additional pages
- IMPORTANT: Your bids will not be saved unless you click submit. If you change pages before you hit submit, your bid information may be lost.
- Enter your preferences by choosing one of
- ++ = very interested
- + = interested
- 0 = neutral (default)
- - = dislike (do not want to review)
- squiggly arrow = conflict
- When does this bidding period end?
- When are the reviews due?
- Make sure you can download and print all of your papers.
- Make sure that you are qualified to review all of your papers.
- Make sure that you have no conflicts of interest with your assigned papers.
- How many papers should I bid on?
- Is there a way to view more than one abstract at a time?
- Should I bid on papers that look interesting to me even if I'm not an expert in the area?
- What constitutes a conflict of interest?
- What if I feel unqualified to review a paper that I have been assigned?
- Why did I get this strange paper that I did not bid on?
- What if I suspect plagiarism or a duplicate submission?
- What are the review criteria?
- Evaluation of work and contribution
- Significance, Originality and novelty
- Relevance to the call of papers
- Readability and organization
- Technical quality
- Are we grading on a curve? (Should the quality of the other papers in my batch influence my review?)
- May I delegate a review to one of my students or colleagues?
- What is the appropriate tone for a review?
Please try to keep your reviews polite and constructive at all times. If you think the paper should be accepted, give the authors suggestions on how to make it an even better paper. If you think the paper should be rejected, politely make your reasons clear to the authors. Give them advice on how to improve things.
- What is the role of the discussion period?
The discussion period is an opportunity for the reviewers to iron out any confusions our misunderstandings about a paper and, hopefully, reach a consensus on the status of the paper. The discussion will be moderated by a senior program committee member. It is not required for the reviewers to reach a consensus. Sometimes there will be irreconcilable differences of opinion. However, any factual questions or disagreements should be clarified in the discussion.
Discussion should take place through the ConfMaster system. To avoid bias, you will not be able to participate in discussion or see other reviews until you have entered your own review.
- Are there different standards for poster papers and plenary papers?
Both poster and plenary papers appear as full length papers in the UAI proceedings. Your review should determine whether the paper lives up to the standards of a UAI paper. You are also welcome to include recommendations about whether the paper would be most appropriate for a poster presentation or plenary presentation. The program chairs will assign papers to presentation types after the final accept/reject decisions have been made.
- What is confidential?
UAI reviewing is not double blind. Author identities should be visible to PC members. Of course, reviewer identities will not be visible to authors and should be kept confidential at all times. You should use care to avoid revealing your identity directly or indirectly through the content of your review.
You should treat the details of submissions as confidential information. Do not discuss ideas you have encountered in papers you have reviewed until these idea are revealed in a public manner by the authors.
- How do I participate in discussion?
- Click on View Assigned Papers in the column on the left side of your window. This will list all of the papers you have been assigned.
- Click on the circled D on the right. This will list all of the reviews for the paper.
- Type comments in the text box on the bottom and click the submit button.
- Should I initiate a discussion, or should I wait for the SPC member to start one?
You might want to give your SPC member a chance to set the tone of the discussion, but if you are very enthusiastic, it is technically possible (and not a faux pas) for PC members to start discussion themselves.
Of course, you cannot start or participate in a discussion until you have entered your own review.
- Is it Ok to email the other reviewers or the SPC member directly?
We can't prevent this, but we encourage you to use the ConfMaster system so that there is a record of the discussion. The discussion details can help the SPC and chairs make their final decisions.
- Can authors see the discussion comments?
No. However, you are encouraged to revise your reviews to reflect insights you have gained from the discussion.
- Why aren't the other reviewers cc'd on the discussion emails that ConfMaster sends out?
ConfMaster sends out separate emails to each of the reviewers and the SPC member whenever a new comment is added. This gives the option of keeping reviewer identities secret during discussions, an option that UAI does not exploit.
- How do I interpret the 1-10 scale of the scores?
Your written comments are much more important than your numerical scores, but we have kept the numerical scoring system to help people organize their thoughts and keep certain key issues in mind. If you would like to calibrate your numerical scores, we suggest the following: A score of 10 should correspond to a strong accept and a score of 1 should correspond to a strong reject. The scale is linear, so 5 and 6 should correspond to borderline cases.
- What does the "Confidential (visible to PCC, PCM, SV only)" box mean?
If you check this box, your comments will be visible only the SPC and chairs, not the other people writing reviews.
- How is the average score of a paper computed?
ConfMaster previously used a fairly nonsensical method to compute the average scores. (We'll spare you the details.) On 4/18, we changed the formula. For each review, we compute the following:
5.5 + (overall_score - 5.5) * (confidence - 1)/9
So, what does this mean? 5.5 is the middle of the scale. (overall_score - 5.5) indicates how far the reviewer wants to move from the middle of the scale. This shift is then discounted by the confidence. Some examples: (a) A score of 10 with a confidence of 10 evaluates to 10. (b) A score of 10 with a confidence of 5 evalutes to 7.5. (c) A score of 10 with a confidence of 1 evalutes to 5.5. (d) A score of 1 with a confidence of 5 evalutes to 3.5.
Why does everything contract towards 5.5? 5.5 is middle of the scale. The further the score is away from 5.5 (in either direction), the clearer the decision.
The average score for the paper is then the simple average of this criterion across the reviews.
- Does UAI have a "short paper" category?
There has been some confusion about this because ConfMaster has the option to accept something as "short" paper. UAI does not have short papers. All accepted papers, poster or plenary, appear as full length papers in the proceedings. If you see anything in ConfMaster about short papers, please ignore this feature.
- Does UAI conditionally accept papers?
No. Accepted papers should acceptable with minor revisions that the authors can be trusted to do in a few weeks. Yes - you will need to use some judgement here.
We're sorry about the confusion. There is a discrepancy between the terminology used by ConfMaster and the terminology traditionally used by UAI.
This discrepancy only affects the way things appear on the ConfMaster web page. In the UAI web page and proceedings, we will continue to use the titles, Program Committee and Senior Program Committee.
The bidding period runs through Monday, March 10, midnight GMT. (EST = GMT - 5).
Papers will be assigned to reviewers by March 14. Your initial reviews are due on March 28, followed by a discussion period, after which your final reviews will be due on April 7. Please adhere strictly to these deadlines! Late or missing reviews compromise the discussion period and can prevent authors from getting complete and fair reviews. You would not want your own paper to receive inadequate review and discussion, so please treat other submissions with the same care you would like your own submission(s) to receive.
Even though the review deadline may (initially) seem far off, you shouldn't delay getting started on your papers. As soon as you receive your reviewing assignment, please do the following:
PC members should give at least 15 papers positive bids. Bidding on more papers increases the chances that you will get a paper you want.
Yes. In the apply for papers area, click on the printer icon on the top right.
It's OK to bid on a few papers that are at the edge of your interests if you think there's a good chance that you can provide a fair review, but please try to keep the majority of your positive bids on papers at the core of your interests.
In general, you should avoid reviewing papers in cases where you may have a personal stake in the outcome or where a personal interest may cloud your professional judgment. Avoid papers from authors who are recent collaborators, close personal friends, enemies, or people in your company/university. Avoid papers from anybody with whom you currently have or previously had an adviser/student relationship. Note that recent and close are not defined rigorously. You will need to use some judgment here. Contact the program chairs if you need guidance.
First, ask yourself if you are really unqualified. Remember that UAI papers should be understandable to more than just the two or three top experts in the world on the topic of the paper. The authors have a burden to define their terms clearly enough that a well informed expert in a related area can still follow the paper. If you think this through and you still feel that you are unqualified to review the paper, then please contact the program chairs.
The people who bid on time got higher priority and we assigned what was left to people who did not bid based upon our guesses about their preferences. Please also keep in mind that there will always be a handful of papers that nobody wants. Somebody has to handle these papers and even if you entered your bids on time, you may have gotten one or two of these unwanted papers. Finally, it's possible that we made a mistake. Please contact the program chairs if you want to double check.
If the paper is unoriginal, plagiarized, or otherwise fails to describe its contribution in an accurate and honest manner, then you should write a negative review and explain the reason in the review. Contact the program chairs if you have concerns about this that cannot be adequately addressed through your review.
ConfMaster is configured for scores in the following areas:
You should keep all of these in mind when writing your review, but also keep in mind that your final recommendation, and your written review are far more important than the numerical scores you provide. The categories and numerical scores should help organize your thoughts. Your written review should evaluate whether the paper makes a significant, novel contribution to UAI.
Please remember that UAI is not just about Bayesian networks. An overly narrow view of UAI will discourage participation and negatively impact the future of the conference.
No. Your review should be based upon your assessment of whether the paper is suitable for UAI. Do not promote bad papers just because your other papers are worse. Do not deprecate good papers just because you happened to get a batch of mostly excellent papers. Remember that your selection of papers is small and is not a random sample.
We put a lot of thought into choosing an excellent program committee, so we hope that the people we chose will do the job. We discourage delegating reviews, but we understand that it is necessary and even desirable in some cases. If one of your colleagues is truly more expert in the area of the paper than you are, and you trust this person to write a high quality review, then feel free to pass the paper on to him or her. If you are a supervisor and would like one of your more senior and trusted students to get experience writing a review, then please supervise the process carefully to ensure that it is both educational for the student and fair for the authors.