NLPeer: A Unified Resource for the Computational Study of Peer Review (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies of peer review for scholarly publications lack datasets and multi-domain corpora to support this complex process. |
| Approach: | They propose to use NLPeer to build a multi-domain corpus of more than 5k papers and 11k review reports from five different venues to support reviewers. |
| Outcome: | The proposed datasets and analysis of three review assistance tasks include a guided skimming task. |
Similar Papers
A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications (N18-1)
Copied to clipboard
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, Roy Schwartz
| Challenge: | a dataset of 14.7K paper drafts and accept/reject decisions in top-tier venues including ACL, NIPS and ICLR is presented to study peer reviews. |
| Approach: | They propose to use the dataset to collect peer reviews from top-tier venues including ACL, NIPS and ICLR and to use it to create a dataset of peer reviews for research purposes. |
| Outcome: | The proposed dataset includes 14.7K paper drafts and accept/reject decisions in top-tier venues including ACL, NIPS and ICLR. |
What Can We Do to Improve Peer Review in NLP? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Traditionally, peer review is expected to act as a filter for high-quality, impactful work, but this does not hold in practice. |
| Approach: | They argue that peer review is becoming increasingly spurious and that it is a problem for NLP . they propose a reproducibility checklist at EMNLP 2020 that could be used to ensure that papers are reproducible. |
| Outcome: | The reproducibility checklist at EMNLP 2020 is the first step in that direction. |
Autonomous Machine Learning-Based Peer Reviewer Selection System (2025.coling-demos)
Copied to clipboard
| Challenge: | Existing systems that match papers with experts are inefficient and often require long turnaround times. |
| Approach: | They propose an autonomous peer reviewer selection system that employs the natural language processing model to match submitted papers with expert reviewers independently of traditional journals and conferences. |
| Outcome: | The proposed system performs faster and smaller than current models while being more scalable. |
Program Chairs’ Report on Peer Review at ACL 2023 (2023.acl-long)
Copied to clipboard
| Challenge: | ACL'23 makes its peer review report public and an official part of the conference proceedings. |
| Approach: | They present an analysis of the factors affecting peer review and identify the most problematic issues that the authors complained about. |
| Outcome: | The authors identified the most problematic issues and provided suggestions for the future chairs. |
Reviewing Natural Language Processing Research (2021.eacl-tutorials)
Copied to clipboard
| Challenge: | a tutorial on reviewing is a useful tool for researchers who are new to the field of NLP. |
| Approach: | this tutorial provides an opportunity to learn the basics of reviewing . more experienced researchers might find this tutorial interesting to revise their reviewing procedure. |
| Outcome: | This tutorial teaches researchers how to revise their reviewing procedure . |
What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review (2022.naacl-main)
Copied to clipboard
| Challenge: | a survey of the NLP community shows that paper-reviewer matching is a problem . authors lose valuable time and opportunities by writing reviews that are arbitrarily low . |
| Approach: | They propose to use paper-reviewer matching to improve peer review . they identify common issues and perspectives on what factors should be considered . |
| Outcome: | The proposed method improves the quality of peer review and improves interpretable peer review assignments. |
A Survey of Data Augmentation Approaches for NLP (2021.findings-acl)
Copied to clipboard
Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, Eduard Hovy
| Challenge: | Data augmentation is a field of research that has been underexplored due to the discrete nature of language data. |
| Approach: | They present a comprehensive survey of data augmentation for NLP by summarizing the literature in a structured manner. |
| Outcome: | The proposed methods are used for popular NLP applications and tasks and highlight current challenges and directions for future research. |
Position Paper: How Should We Responsibly Adopt LLMs in the Peer Review Process? (2026.findings-eacl)
Copied to clipboard
| Challenge: | a recent paper criticizes the current use of Large Language Models (LLMs) for simple review text generation. |
| Approach: | They propose to use Large Language Models to support key aspects of the review process . they argue that this approach overlooks more meaningful applications of LLMs . authors argue that the increased reviewing burden per reviewer is a factor . |
| Outcome: | The proposed approach would support reproducibility, correctness and relevance of citations and ethics review flagging. |
Understanding Ethics in NLP Authoring and Reviewing (2023.eacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial will equip participants with basic guidelines for thinking deeply about ethical issues . |
| Approach: | This tutorial will equip participants with basic guidelines for thinking deeply about ethical issues . the methodology is interactive and participatory, including case studies and working in groups . |
| Outcome: | This tutorial will equip participants with basic guidelines for thinking deeply about ethical issues . the methodology is interactive and participatory, including case studies and working in groups. |
Navigating Ethical Challenges in NLP: Hands-on strategies for students and researchers (2025.acl-tutorials)
Copied to clipboard
Luciana Benotti, Fanny Ducel, Karën Fort, Guido Ivetta, Zhijing Jin, Min-Yen Kan, Seunghun J. Lee, Minzhi Li, Margot Mieskes, Adriana Pagano
| Challenge: | This tutorial will equip participants with basic guidelines for thinking deeply about ethical issues . participants will gain practical experience on when to flag a paper for ethics review . |
| Approach: | This tutorial will equip participants with basic guidelines for thinking deeply about ethical issues . participants will gain practical experience on when to flag a paper for ethics review . |
| Outcome: | This tutorial will equip participants with basic guidelines for thinking deeply about ethical issues . participants will gain practical experience on when to flag a paper for ethics review . |