Paper Reviewing Guidelines

This document guides those who participate in the ISMAR 2024 reviewing process and is directed towards those who perform full reviews of papers (i.e., secondary review coordinator 2AC, committee members, and reviewers) and meta-reviews (primary review coordinator, or 1AC) for both the journal track and the conference track submissions.

Please note that for 2024 you may be assigned papers from both or either the Journal Paper or Conference Paper track.  This guideline covers recommendations and best practice for both tracks. Therefore please read carefully the guidelines below for each track. Finally, to ensure the integrity of the review process, we ask that reviewers refrain from uploading papers to any AI system (e.g., Claude AI, Wordtune etc.) for analysis, support or guidance during the review period.

This page talks about –

Journal Papers

For papers submitted into the Journal Paper track please use the following guidelines:

  • We consider a paper of sufficient journal quality if it provides obvious and strong contributions to the field, either in theory, methods, engineering, design, framework/taxonomy, and/or evaluation approaches. Journal papers should reflect an in-depth overview of related work (as compared to a conference paper).
  • Consider the value of the contribution and merits of the paper to the ISMAR community. The paper may be an edge case. But, if it discusses a topic that is currently unknown to the community or is extremely relevant at the time of submission , then it should be considered and reviewed accordingly.
  • Journal papers are up to 9 pages long (+2 pages of references). Longer papers with a smaller contribution or that may require a major revision to be of journal quality, will be rejected. Please note  ISMAR 2024 does not grant major revisions for journal papers. Authors who submit survey papers can when requested have an extra page of references.
  • The ISMAR 2024 review form will ask you to provide a single score (6-point ranking, see below) that represents the quality of the submission. Your written appraisal must support this score.

Journal to Conference Papers

  • In cases where reviewers determine that a journal submission does not meet the publication standards for the journal but still has the potential to make a valuable contribution to the conference, they may suggest re-submitting the paper to the conference track. Please note this option does not guarantee the paper will be accepted into the conference and should only be used for papers where there could be a clear contribution made from the paper within the timeframe given to the authors (1 week)
  • The final decision to submit the paper to the conference track rests with the authors. They will have the opportunity to consider the reviewers’ feedback and decide whether to revise their paper and submit it to the conference, or pursue other publication venues.
  • We ask that reviewers of these journal submissions be open to re-reviewing them as conference track submissions in order to maintain continuity of the review process for the authors.

Conference Papers 

For papers submitted to the Conference Paper track please use the following guidelines:

  • The primary aspect for accepting/rejecting a paper is its contribution: We consider a paper of sufficient quality if it presents a strong, tangible contribution to an area of research of relevance to ISMAR. 
  • Consider the value of the contribution and merits of the paper to the ISMAR community. The paper may be an edge case. However, if it discusses a topic, e.g., currently unknown to the community or extremely relevant at the time, then it should not be rejected outright. 
  • For conference papers, the contribution needs to be proportional to the paper length. A small contribution is acceptable if the paper is short (minimum length is 4 full pages NOT including references). Longer papers (e.g., 8 pages NOT including references) with a small contribution that may require a major revision will need to be rejected given that ISMAR 2024 does not grant major revisions.
  • Papers are acceptable if the authors can easily correct the weaknesses (e.g., missing references, minor spell-checking, fuzzy statements, lack of implementation details, and others).
  • Remember that the ISMAR 2024 review form will ask you to provide a single score (6-point ranking, see below) judging the quality of the work. Your written appraisal must support this score.

In general, as you perform your review for either journal or conference submissions, we ask that you reflect on the overall merit of the work and read papers with care and sympathy. Avoid seeking hidden flaws. Many hours of work — in some cases, years of work — have gone into the research and writing of this paper. Try to avoid last-minute reviews.

Review Scores: 6-point Ranking

The section explains the ISMAR 6-point ranking and explains when we think one should select a particular score. We are aware that the decision can be subjective in many cases and that selecting between two is often a judgment call. We hope that this explanation removes some gray area for decision making and helps to align reviewers on how to make borderline decisions for ratings.

  • Definitely accept (rating of 6): I would argue strongly for accepting this submission.
    Select this option if the paper is acceptable as-is (except for minor edits), with a strong contribution and merits for the ISMAR community.
  • Probably accept (rating of 5): I would argue for accepting this submission.
    Select this option if the paper has a valid contribution and merits for the ISMAR community. Some additional explanations or minor corrections are required.
  • Weak accept (rating of 4): The paper has weaknesses but the contributions outweigh the weaknesses. Select this option if the research is relevant, the topic is of value for the ISMAR community, and the attitude towards this contribution is overall positive despite the identified weaknesses.
  • Weak reject (rating of 3): The paper has contributions but the weaknesses outweigh the contributions. Select this option if the research is relevant, the topic is of value for the ISMAR community, but the attitude is overall negative because of the identified weaknesses.
  • Probably reject (rating of 2): I would argue for rejecting this submission.
    Select this option if the research is relevant and the topic is of value for the ISMAR community, but the research has several severe weaknesses.
  • Definitely reject (rating of 1): I would argue strongly for rejecting this submission.
    Select this option if the contribution is not understandable and the paper has no recognizable merits, or it is entirely unclear what information the ISMAR community gains from this submission.

Writing a Review (for both journal and conference papers)

The following guidelines outline the content and key points of a high-quality review for ISMAR 2024. We believe that both journal and conference paper reviews should be written with the same guidelines in mind. Please adhere to the guidelines and contact the program chairs or review coordinators for any questions.

  • A high-quality review should have about 1-page of well-considered commentary (at least 500 words), or more, if warranted. Short and/or content-free reviews are insufficient and frustrate the authors.
  • While we expect high-quality reviews for both journal and conference papers, high-quality reviews are of particular importance for journal paper submissions, as TVCG will only accept high quality papers that also receive high-quality reviews. The reviews are checked by the TVCG administration to ensure quality before acceptance is confirmed.
  • The program committee and primary reviewers will be asked to evaluate the reviews for journal paper submissions based on the criteria set for TVCG journal associate editors. Those criteria are given here:
    • “When assessing the reviews, ask yourself, “Does the reviewer provide specific feedback in their ‘Public Comments’ (which are comments for the author) for a recommendation to be made?” We recommend reading the review reports to make sure they meet appropriate standards of quality, thoroughness, and professionalism. If a review lacks sufficient detail or is otherwise incomplete, please ask the reviewer for more information or invite a new reviewer, so that the recommendation and decision are substantiated by these comments. Further, It is unacceptable for a reviewer to require additional citations for the sole purpose of influencing bibliometric measures of either an individual or a periodical, as mentioned in the IEEE PSPB Operations Manual (PDF) Section 8.2.2.A.4.”
  • Thus, for both types of reviews (journal and conference) reviewers must state specifically the reason(s) for the decision selected for the paper. Clearly describe on what grounds the paper should be accepted (or rejected).
  • Reviewers should describe the contributions in the paper and why they are noteworthy or important to the community.
  • They should explicitly and clearly discuss the weaknesses and limitations in a positive and constructive manner. Specifically, be positive and not insulting.
  • Reviewer comments should assess the work the authors did and whether their methods are appropriate to support their claims. They should avoid judging and explaining what the authors should have done.

Mind that your decisions affect the public appearance of ISMAR 2024. Therefore, the program chairs are very serious about ensuring the highest possible reviewing standards for ISMAR 2024. The coordinators and/or program chairs will ask you to improve your review if they think the reasons for your judgment are unclear.

We strongly recommend that you read the entire (short) article by Ken Hinckley, which we found to give a lot of constructive advice:

Hinckley, K. (2016). So You’re a Program Committee Member Now: On Excellence in Reviews and Meta-Reviews and Championing Submitted Work That Has Merit. https://www.microsoft.com/en-us/research/wp-content/uploads/2016/10/Excellence-in-Reviews-MobileHCI-2015-Web-Site.pdf

Here are some of the major elements from Hinckley’s paper, in some cases modified by adding insights from further advice by Steve Mann and Mark Bernstein, for quick reference:

  • A high quality review should have a number of paragraphs (or even several pages of well-considered commentary if warranted). Note, this is required for journal paper submissions in order to meet the TVCG requirements.
  • Short and/or content-free reviews are insufficient and will be caught by our reviewer’s review process and the TVCG administration if it is a journal paper submission.
  • Read papers with care and sympathy. Many hours of work — in some cases, years of work — have gone into research and writing this paper. Try to avoid last-minute reviews.
  • State specifically why the paper is “great”, “mediocre” or “bad”.
  • Clearly describe on what grounds the paper should be accepted (or rejected).
  • Describe the contributions in the paper and why they are noteworthy, or important.
  • Reflect on the contributions or possible contributions of the work.
  • Explicitly and clearly discuss the weaknesses and limitations in a positive and constructive manner. Specifically, don’t be insulting – be positive.
  • Clearly and explicitly call out the strengths and utility of the work.
  • Your review should not be about what should have been done; rather it should be a critique of what the authors actually did.
  • Consider how the author’s arguments, results, and demonstrations fit into closely related work as well as the field as a whole.
  • Do not reject a paper because of a few missed citations.
  • In fact, do not reject a paper because of anything that can be easily fixed/addressed. Clearly state the requirements as all conditionally accepted papers will be shepherded towards an improved version that fulfills those requirements.
  • A paper’s failure to justify or fully motivate certain decisions likely represent a correctable oversight, not an unequivocal sign of poorly conceived research.
  • Avoid the fallacy of novelty. Specifically, do not simply reject papers because they replicated experiments.
  • Reviewing scores: Around 2/3 of your rankings should be 1 or 6, around 1/3 should be 2 or 5, and you should rarely, rarely, rarely give a rating of 3 or 4.

From ISMAR’s perspective, we would like to add that in your role to maintain the high-quality of papers, we ask that you not categorically find hidden flaws and assassinate papers wherever possible but instead review papers for their merits.

When reviewing, keep in mind that almost every paper we review “could have done x, or y, or z”. Don’t fall into this trap! We cannot reject papers because authors did not perform their research the way we would have done it, or even how it is typically done. Instead, we must judge each paper on its own merit, and whether or not the body of work presented can stand on its own, as presented. Sure, every paper “could have done more”, but is the work that has been done of sufficient quality and impact?

Furthermore, we should not reject papers because “this experiment has been done before”.  This fallacy of novelty, ignores the long-standing tradition in science of replication.  New work that performs similar research and finds consistent (or even conflicting) results are of value to the community and to science in general, and should be considered on their own merits.

When evaluating papers that run human-subjects studies, it is important that the participant sample population is representative of the population for which the technology is being designed. For example, if the technology is being designed for a general population, then the participant population should include equal gender representation and a wide range of ages. All papers with human-subject studies must report at a minimum demographic information including age, gender, and every possible hint of social and diversity representation. If this is not the case, reviewers should not automatically reject the paper, but instead provide appropriate constructive critique and advice regarding general claims that do not use representative sample populations.

And finally, to quote Ken Hinckley:

“When in doubt, trust the literature to sort it out.”

Writing a Meta-Review

The following guidelines outline the content of a good meta-review for both journal and conference submissions. The section is for review coordinators (1AC) and explains the content the chairs believe best supports a decision.

  • Describe the primary contribution of the paper.
  • Summarize the most significant pros and cons of the paper. The most critical are often those the majority of reviewers highlight in their reviews. Abstain from reiterating every single aspect (we have the reviews for that).
  • Explain the decision and the pros and cons that support this decision.
  • In case of conditional acceptance, describe the conditions the authors have to meet before the paper can be accepted.
  • In case the paper is rejected, add suggestions for improvement.
  • Avoid adding discussion details or the score into the meta-review.

Mind that the authors will see the meta-review with the final decision. Be constructive and explain more rather than less, especially for cases when the authors receive an unfavorable decision. Very often, the research or paper was not ready at the time of submission. Invite the authors to re-submit next year if feasible.

Frequently Asked Questions (FAQs)

  • Should we reject papers that have been published on ArXiv?

    In a nutshell: No, you should not reject papers that have been published on ArXiv or a similar service, as authors may have done it as a way to get a timestamp for their work. However, if the ArXiv submission explicitly states that the submission is under review at ISMAR, or if the authors listed this prepublication on their individual or institutional webpages or generated publicity for it through other forms of media, then yes, it may constitute a violation of ISMAR policies. Please raise any related concerns in your review and/or contact the Program Chairs by email at: program_chairs@ieeeismar.org
    Please read the full “Double-Blind Process and Anonymity Policy” in the Author Guidelines.
  • Should we reject papers that have been presented before in a different format (e.g., poster or demo)?

    In some situations, a submission may build upon prior work. As part of the Author Guidelines, paper authors were instructed to be proactive about clarifying such cases by uploading additional documents to the submission system that are anonymous and can only be seen by the primary coordinator assigned to this submission. If you suspect any issues related to this point, please first contact the primary coordinator of this paper. If no clarification has been uploaded by the authors, please carefully assess how far the publications overlap. Note that ISMAR does not consider a prior non-archival 2-page poster/demo extended abstract a reason for rejection of a paper submitted on the same topic.

Document History:

This document was updated and extended by the ISMAR 2024 Program Chairs:  Ulrich Eck, Misha Sra, Jeanine Stefanucci, Maki Sugimoto, Markus Tatzgern, Ian Williams after being extended by the ISMAR 2024 Conference Paper Chairs: Jens Grubert, Andrew Cunningham, Evan Peng, Gerd Bruder, Anne-Hélène Olivier, and Ian Williams. These guidelines were updated for the ISMAR 2022 conference papers review process: Henry Duh, Jens Grubert, Jianmin Zheng, Ian Williams, and Adam Jones. These guidelines are based on the work of the ISMAR 2019 PC Chairs: Shimin Hu, Denis Kalkofen, Joseph L. Gabbard, Jonathan Ventura, Jens Grubert, and Stefanie Zollmann.

Thank you for your support and work to ensure the highest-quality ISMAR reviews.
Do not hesitate to contact us for any further information: program_chairs@ieeeismar.org
ISMAR 2024 Programme Chairs
Ulrich Eck, Misha Sra, Jeanine Stefanucci, Maki Sugimoto, Markus Tatzgern, Ian Williams
ISMAR 2024 Science and Technology Program Chairs