| Challenge: | a double-blind review system enforces author anonymity during the review period . authors can be inferred with accuracy as high as 87% on ACL and 78% on EMNLP . |
| Approach: | They examine how well deep learning techniques can infer authors of a paper . authors found authors can be inferred with accuracy as high as 87% on ACL and 78% on EMNLP . |
| Outcome: | The proposed method can infer authors with 87% accuracy on ACL and 78% on EMNLP for the top 100 most prolific authors. |
Similar Papers
The Double Bind: Revisiting Preprinting and Peer Review Two Years After the Removal of the ACL Anonymity Period (2026.findings-acl)
Copied to clipboard
| Challenge: | ACL removed the anonymity period for conference submissions in February 2024 . |
| Approach: | They track preprinting trends for 47k publications and analyze 1.9k peer reviews . they suggest improving visibility and investing in diversity initiatives . |
| Outcome: | The proposed anonymity period was removed in 2024, but it was ineffective for underrepresented researchers . the authors suggest addressing D&I issues rather than implementing anonymity policies. |
A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications (N18-1)
Copied to clipboard
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, Roy Schwartz
| Challenge: | a dataset of 14.7K paper drafts and accept/reject decisions in top-tier venues including ACL, NIPS and ICLR is presented to study peer reviews. |
| Approach: | They propose to use the dataset to collect peer reviews from top-tier venues including ACL, NIPS and ICLR and to use it to create a dataset of peer reviews for research purposes. |
| Outcome: | The proposed dataset includes 14.7K paper drafts and accept/reject decisions in top-tier venues including ACL, NIPS and ICLR. |
What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review (2022.naacl-main)
Copied to clipboard
| Challenge: | a survey of the NLP community shows that paper-reviewer matching is a problem . authors lose valuable time and opportunities by writing reviews that are arbitrarily low . |
| Approach: | They propose to use paper-reviewer matching to improve peer review . they identify common issues and perspectives on what factors should be considered . |
| Outcome: | The proposed method improves the quality of peer review and improves interpretable peer review assignments. |
Towards Reliable Paper Contributions Annotation in the ACL Rolling Review (2026.findings-acl)
Copied to clipboard
| Challenge: | Identifying the types of contributions an article makes can help readers grasp its significance. |
| Approach: | They propose to use a typology to categorize articles by their contributions to improve review quality and fairness. |
| Outcome: | The ACL Rolling Review (ARR) introduced a typology requiring authors to specify their contributions to improve review quality and fairness. |
NLPeer: A Unified Resource for the Computational Study of Peer Review (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies of peer review for scholarly publications lack datasets and multi-domain corpora to support this complex process. |
| Approach: | They propose to use NLPeer to build a multi-domain corpus of more than 5k papers and 11k review reports from five different venues to support reviewers. |
| Outcome: | The proposed datasets and analysis of three review assistance tasks include a guided skimming task. |
Open-World Authorship Attribution (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models do not evaluate their performance in academic research . authors aim to identify authors from anonymous text without additional information . |
| Approach: | They propose a benchmark to quantitatively assess LLMs' ability to infer author from text . they propose 'open-world' authorship attribute' to be a two-stage framework . |
| Outcome: | The proposed approach achieves 60.7% accuracy and 44.3% accuracy in two stages. |
Beyond Metadata: What Paper Authors Say About Corpora They Use (2021.findings-acl)
Copied to clipboard
| Challenge: | Currently, dataset retrieval relies almost exclusively on metadata provided by the publishers. |
| Approach: | They propose to use metadata to extract review statements from scientific publications . they argue that a crucial piece of information is missing to inform the examination of search results . |
| Outcome: | The proposed analysis is the first of its kind in the field of Natural Language Processing. |
A Systematic Review of Reproducibility Research in Natural Language Processing (2021.eacl-main)
Copied to clipboard
| Challenge: | Despite the recent progress in reproducibility, the field is far from reaching a consensus on how reproducibility should be defined, measured and addressed. |
| Approach: | They propose to provide a wide-angle snapshot of current work on reproducibility in NLP. |
| Outcome: | The proposed work will provide a wide-angle snapshot of current work on reproducibility in NLP. |
Justice in Judgment: Unveiling (Hidden) Bias in LLM-assisted Peer Reviews (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that large language models carry implicit biases across race, gender, and religion . prior studies documented such biase based on text generation and classification tasks . |
| Approach: | They investigate bias in large language models by controlling metadata on author metadata . authors found affiliation bias favoring authors from highly ranked institutions . |
| Outcome: | The proposed model favors authors from highly ranked institutions, the authors show . the model also favors author affiliations from highly-ranked institutions . |
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (2026.acl-short)
Copied to clipboard
| Challenge: | ACL is 30+ times larger than two decades ago, and we face issues such as overwhelming participants, outdated papers, and low quality review. |
| Approach: | aaron carroll: ACL has become 30+ times larger than two decades ago . he says increasing research in LLM, AI accelerating research can help . carroll will share some of his recent work on AI review automation, paper recommendation, and AI arXiv . |
| Outcome: | aaron e. muller: ACL has become 30+ times larger than two decades ago . he says recent work on AI review automation, paper recommendation, and arXiv is promising . |