Guidelines for Reviewers
This page includes review guidelines for meta-reviewers and reviewers for this year's International Conference on Social Robotics. These guidelines are developed by the program co-chairs. Please contact us with questions regarding the guidelines or with general questions about reviewing for ICSR. Creating a strong technical program depends largely on the hard work of volunteer reviewers and we very much appreciate your contribution to this year's conference.
Christoph Bartneck & Bilge Mutlu
ICSR 2011 Program Co-Chairs
Jun 1 Submission deadline
Jun 5 Meta-reviewer assignments
Jun 13 Reviewer assignments
Jul 15 Paper reviews due
Jul 22 Meta-reviews due
Jul 29 Program committee discussion ends
Aug 1 Author notification
Meta-reviewers are program committee members who are top experts on various areas of social robotics. They are faculty members or senior researchers from leading institutions, laboratories, or research groups in social robotics and represent a diversity of perspectives (e.g., academic vs. corporate research) and geographic locations (i.e., Europe, Asia, Americas, and other locations). The sections below list the responsibilities for meta-reviewers and key considerations in identifying and assigning reviewers and provide an outline of the meta-review.
- Identifying qualified reviewers who are committed to reviewing papers by the reviewer assignment deadline,
- Providing reviewers with information on the review process, milestones, and deadlines,
- Ensuring the timely completion of all reviews,
- Creating a meta-review for each paper, highlighting key strengths and weaknesses of the paper,
- Participating in the Program Committee discussion on paper decisions.
Key Considerations for Reviewer Assignments:
- Each paper should be reviewed by two reviewers in addition to the meta-review,
- Each paper should be reviewed by at least one faculty member or senior researcher,
- Other meta-reviewers should not serve as reviewers,
- Reviewers should not be assigned more than three papers.
- Summary – The meta-review should start with a brief summary of the research presented in the paper, outline the potential contributions of the work, and the significance and relevance of the research for the ICSR community.
- Strengths and Weaknesses – The meta-review should outline the key strengths and weaknesses of the research and the presentation of the work in the paper. This outline could be in brief paragraphs or in a bulleted list form. Three key strengths and three key weaknesses tend to capture the important points for most papers. These descriptions should cite the individual reviewers' comments (e.g., "R1 highlighted that...") and involve the meta-reviewer's evaluation.
- Recommendation – The meta-review should end with a brief conclusion with the meta-reviewer's recommendation for the authors (e.g., recommending a different venue for publication) as well as for the program chairs (e.g., "I strongly recommend that this paper is included/not included in the conference program," etc.).
- Numeric Evaluation – The numeric evaluation involves a scale from "Strong Accept" to "Strong Reject" with the addition of "Reject without Review." The last option is reserved for papers that are clearly not relevant to social robotics, have no implications (technical, design, or knowledge) for robots that might function in the human environment, or are not original research work (e.g., published work, under review at other venues, etc.).
Reviewers are researchers and students who are actively involved in research in the field of social robotics. They might be authors of papers submitted to this year's conference or past contributors to the conference or others whose research fall within the topics covered in the conference call. The sections below outline reviewer responsibilities and provide an outline of the paper review.
- The main responsibility of reviewers is to evaluate the intellectual merit of the submission and the significance of its contributions to social robotics.
- Reviewers have the responsibility to keep the submissions confidential.
- The reviews should be as constructive as possible, providing the authors with actionable comments and suggestions. For instance, if the presented work or the presentation of the work could be improved in any way, a breakdown of what improvements could be made would be most helpful to the authors. If the reviewer thinks that key related work is missing in the paper, the review should include pointers to this work (e.g., links or citation information). If the presentation of the work lacks clarity, the review should posit specific questions that the authors should seek to answer in the paper.
Numeric Evaluation – The numeric evaluation involves a scale from "Strong Accept" to "Strong Reject" with the addition of "Reject without Review." The last option is reserved for papers that are clearly not relevant to social robotics, have no implications (technical, design, or knowledge) for robots that might function in the human environment, or are not original research work (e.g., published work, under review at other venues, etc.).
- Summary – The review should start with a brief summary of the work presented in the paper and outline its main findings and potential contributions to social robotics.
- Strengths and Weaknesses – The review should list the strengths and weaknesses of the paper in a bulleted list form. Most reviews list three key strengths and three key weaknesses.
- Detailed Comments – This section is the main part of the review. It will expand on the strengths and weaknesses that that are listed in the previous section. The reviewers should use this section to discuss the intellectual contributions of the paper, make detailed recommendations on design, implementation, and evaluation, provide pointers to relevant work that the paper does not mention, and point to future directions that might benefit the authors' research program. We invite reviewers to be a constructive as possible in their comments and provide authors with actionable suggestions.
- Suggestions for Improvement – The review should provide authors with a forward-looking summary of the changes they can make in their future work or future revision of their paper including changes that should be made to the camera-ready version of the paper. This section could be made up of a bulleted list or short paragraphs that list the changes the reviewers thinks are important to improve the work.
- Recommendation – The review should conclude with a brief qualitative statement of whether or not the reviewer recommends including the paper in this year's technical program. This recommendation is extremely important interpreting the numeric evaluation particularly for borderline cases. Recommendations such as "I believe that this paper makes a significant contribution to social robotics and should be included in this year's technical program," "While the paper has some key weaknesses, it could be included in the technical program, as it makes the following contributions: ...," "While the work presented in the paper is promising, it does not seem to be ready for publication due to key weaknesses in ...," or "The paper does not seem to be appropriate for the topics and goals of the conference and the authors should be encouraged to submit their work elsewhere such as ...." are appropriate.