A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications (N18-1)
Copied to clipboard
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, Roy Schwartz
| Challenge: | a dataset of 14.7K paper drafts and accept/reject decisions in top-tier venues including ACL, NIPS and ICLR is presented to study peer reviews. |
| Approach: | They propose to use the dataset to collect peer reviews from top-tier venues including ACL, NIPS and ICLR and to use it to create a dataset of peer reviews for research purposes. |
| Outcome: | The proposed dataset includes 14.7K paper drafts and accept/reject decisions in top-tier venues including ACL, NIPS and ICLR. |
Similar Papers
NLPeer: A Unified Resource for the Computational Study of Peer Review (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies of peer review for scholarly publications lack datasets and multi-domain corpora to support this complex process. |
| Approach: | They propose to use NLPeer to build a multi-domain corpus of more than 5k papers and 11k review reports from five different venues to support reviewers. |
| Outcome: | The proposed datasets and analysis of three review assistance tasks include a guided skimming task. |
PeerQA: A Scientific Question Answering Dataset from Peer Reviews (2025.naacl-long)
Copied to clipboard
| Challenge: | a dataset of 579 QA pairs from 208 scientific articles contains answers that reviewers raised while thoroughly examining the scientific article. |
| Approach: | They propose a dataset that contains questions that reviewers raised while thoroughly examining the scientific article. |
| Outcome: | The proposed dataset contains 579 QA pairs from 208 academic articles . the results show that decontextualization approaches improve retrieval performance . |
Autonomous Machine Learning-Based Peer Reviewer Selection System (2025.coling-demos)
Copied to clipboard
| Challenge: | Existing systems that match papers with experts are inefficient and often require long turnaround times. |
| Approach: | They propose an autonomous peer reviewer selection system that employs the natural language processing model to match submitted papers with expert reviewers independently of traditional journals and conferences. |
| Outcome: | The proposed system performs faster and smaller than current models while being more scalable. |
Program Chairs’ Report on Peer Review at ACL 2023 (2023.acl-long)
Copied to clipboard
| Challenge: | ACL'23 makes its peer review report public and an official part of the conference proceedings. |
| Approach: | They present an analysis of the factors affecting peer review and identify the most problematic issues that the authors complained about. |
| Outcome: | The authors identified the most problematic issues and provided suggestions for the future chairs. |
Identifying Aspects in Peer Reviews (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to peer review are limited in how they identify aspects . a growing volume of peer review submissions is straining the process . |
| Approach: | They propose a data-driven schema for deriving aspects from peer reviews . they propose augmented peer reviews and show how it can be used for community-level review analysis. |
| Outcome: | The proposed approach can be used to support peer review, but lacks formal definition of aspect . it also shows that the choice of aspects can impact downstream applications . |
Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026.acl-long)
Copied to clipboard
Sihong Wu, Owen Jiang, Yilun Zhao, Tiansheng Hu, Yiling Ma, Kaiyan Zhang, Manasi Patwardhan, Arman Cohan
| Challenge: | Recent advances in large language models (LLMs) motivated methods that assist or automate different stages of peer review pipeline. |
| Approach: | They synthesize techniques to enhance peer review generation and after-review tasks aligned to reviews. |
| Outcome: | The proposed methods improve the peer review process by fine-tuning strategies, agent-based systems, and emerging paradigms. |
What Can We Do to Improve Peer Review in NLP? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Traditionally, peer review is expected to act as a filter for high-quality, impactful work, but this does not hold in practice. |
| Approach: | They argue that peer review is becoming increasingly spurious and that it is a problem for NLP . they propose a reproducibility checklist at EMNLP 2020 that could be used to ensure that papers are reproducible. |
| Outcome: | The reproducibility checklist at EMNLP 2020 is the first step in that direction. |
What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review (2022.naacl-main)
Copied to clipboard
| Challenge: | a survey of the NLP community shows that paper-reviewer matching is a problem . authors lose valuable time and opportunities by writing reviews that are arbitrarily low . |
| Approach: | They propose to use paper-reviewer matching to improve peer review . they identify common issues and perspectives on what factors should be considered . |
| Outcome: | The proposed method improves the quality of peer review and improves interpretable peer review assignments. |
Beyond Metadata: What Paper Authors Say About Corpora They Use (2021.findings-acl)
Copied to clipboard
| Challenge: | Currently, dataset retrieval relies almost exclusively on metadata provided by the publishers. |
| Approach: | They propose to use metadata to extract review statements from scientific publications . they argue that a crucial piece of information is missing to inform the examination of search results . |
| Outcome: | The proposed analysis is the first of its kind in the field of Natural Language Processing. |
Does My Rebuttal Matter? Insights from a Major NLP Conference (N19-1)
Copied to clipboard
| Challenge: | Peer review is a core element of the scientific process, but few studies have evaluated its properties empirically. |
| Approach: | They propose to use peer review to assess the effectiveness of rebuttal phase in NLP conferences. |
| Outcome: | The proposed task predicts after-rebuttal scores from initial reviews and author responses. |