Intructions for Reviewers
Thank you for agreeing to help in the reviewing process for ACL2020! This edition of the ACL received over 3,400 submissions (the largest number of submissions ever for an NLP conference)! In order to ensure that all valid submissions receive a fair and thorough review, we would like to bring your attention to a few guidelines for reviewing papers. Please read these instructions before you start reviewing.
All reviewers were assigned no more than 6 reviews each. Since we had a joint long and short paper deadline, please be mindful that you may have a combination of long and short papers to review.
Please note that the content of any submission to ACL2020, and the participants in and content of discussion on submissions, are confidential.
The Review Form
We adapted the ordinary review form and Theme review form and instructions from EMNLP-IJCNLP 2019 and ACL 2019. The review form consists of three main sections.
1. In-Depth Review. This section is for you to give your overall assessment of the paper and to provide evidence to support your opinions. There are 6 subsections:
- The core review: This is the most important part. It should include your view of the main contributions that the paper intended to make and how well it succeeds at making these contributions. From your point of view, what are the significant strong and weak parts of the paper and the work it describes? This could be a 2 paragraph (or longer) essay and/or a bullet list. Remember to describe how the work advances the state of knowledge in computational linguistics and/or highlights why it fails to make a sufficient contribution.
- Reasons to accept: please briefly summarize from your core review the main reasons why this paper should be accepted for the conference, and how the ACL community would benefit from it. You may refer back to your review to provide more context and details.
- Reasons to reject: please briefly summarize the possible risks or harm that might come from having this paper published and presented in something close to its current form. What are the parts that would need to be improved in order to advance the state of knowledge?
- Overall recommendation: Here you are asked to synthesize the above and come up with your own recommendation for the paper.
- Like EMNLP 2019, we have used a 5 point scale with a half point increments. The detailed explanation for each point level is provided in the review form. These numbers are just a concise way of expressing your overall opinion and relative importance of the factors mentioned above.
- A new thing this year is that we have removed the rating 3 (ambivalent) as we would like reviewers to take a stand on whether the paper is a bit above the borderline or below the borderline.
- Decisions will be made not just on the scores and certainly not on average scores, but will also take into account the whole review above, reviewer discussion and Area Chair meta-reviews and recommendations. However it is important to align your recommendation with the reasoning given above, so that authors will be able to understand the motivation for the recommendations and how decisions were arrived at.
- Reviewer confidence: This section should be used to inform the committee and authors how confident you are about your recommendation, taking into account your own expertise and familiarity with this area and the paper’s contents.
- Author response: ACL 2020 will have an author response period. It is important for you to check whether author responses have cleared up your questions or misunderstandings. This may influence your overall recommendation and the core review. If that’s the case, please update your recommendation and review accordingly (and state in your review any new decisions you made so the Area Chairs are aware).
2. Questions and Additional Feedback for the authors: Since we will have an author response process, for questions you would like the author(s) to respond to during the response period, please include them here. This is also the place for you to give suggestions to the authors to help them improve the paper for the final version (or a future submission).
3. Confidential information: Your answers to questions in this section will not be shared with the authors. Here we ask you about the recommended presentation type (oral vs. poster), recommendation for awards, any ethical concerns, and confidential comments to the area chairs and/or PC chairs.
General Guidelines for ACL Reviewing
Please take a balanced approach when reviewing the papers. On the one hand, we would like to have a solid technical program with high-quality papers describing a complete piece of work; on the other hand, we also want a broad and interesting program, so please keep an open mind when evaluating and recommending the papers assigned to you.
Please note that a short paper is not a shortened long paper. Instead short papers should have a point that can be made in a few pages and present a focused contribution.
Following ACL 2019, we suggest that you consult some of the excellent advice from experienced reviewers and conference organizers on the web. In particular:
- Read the excellent blog post by NLP veterans Mirella Lapata, Marco Baroni, Yoav Artzi, Emily Bender, Joel Tetreault, Ani Nenkova, and Tim Baldwin, who compiled their reviewing recommendations for ACL 2017: https://acl2017.wordpress.com/2017/02/23/last-minute-reviewing-advice/
- Nikas Elmqvist from the University of Maryland has great advice on mistakes to avoid when reviewing: https://sites.umiacs.umd.edu/elm/2016/02/01/mistakes-reviewers-make/
- Sample reviews by the NAACL 2018 chairs (note they have a different form):https://naacl2018.wordpress.com/2018/01/20/a-review-form-faq/
- And on the lighter side: https://naacl2018.wordpress.com/2017/12/19/some-holiday-reviewing-advice/
Supplementary materials are allowed as a stand-alone document uploaded as an additional file. Supplementary materials are, as the name suggests, supplementary, and you have no obligation to read them. You should treat them like other citations in submissions that may be helpful in understanding background or details beyond the scope of the paper itself.
As in most previous NLP conferences, you are allowed to solicit help from others. However, when it comes to writing the final review and giving the final scores, we expect you to take the secondary reviewer’s review and rewrite it using your own words and adjust the scores when you see fit. Essentially, the final review should reflect your own opinions about the paper, and you need to be able to justify the opinions you present in the final review.
Format of Submissions
The program chairs and area chairs have already identified submissions that violated our formatting guidelines and have desk-rejected those submissions. Therefore, you do not need to worry about formatting issues with the submissions assigned to you.
Your reviews are due Friday, February 7, 2020 (midnight anywhere on Earth). Please note that there is a reviewer discussion period from February 18 to 24 after the author response. Your duties are marked in bold. Don’t leave reviewing to the last minute!
- Jan 17 – Feb 07: Review Period
- Feb 12 – Feb 17: Author Response
- Feb 18 – Feb 24: Reviewer Discussion Period (ACs lead discussion)