The Good, the Bad and the Constructive: Automatically Measuring Peer Review’s Utility for Authors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Providing constructive feedback to authors is a core component of peer review . authors lack guidance on how to improve their review, a problem that is often overlooked . |
| Approach: | They use a RevUtil dataset to benchmark fine-tuned models for assessing review comments . they find that machine-generated reviews generally underperform human reviews on these aspects . |
| Outcome: | The proposed model outperforms closed models on four aspects of review comments . the proposed model achieves agreement levels comparable to and exceeding those of human models . |
Similar Papers
Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis (2024.findings-emnlp)
Copied to clipboard
Jianxiang Yu, Zichen Ding, Jiaqi Tan, Kangyang Luo, Zhenmin Weng, Chenghua Gong, Long Zeng, RenJing Cui, Chengcheng Han, Qiushi Sun, Zhiyong Wu, Yunshi Lan, Xiang Li
| Challenge: | Existing approaches to review scientific papers are limited by their content or quality . SEA is a framework for automated scientific review, but its contents are generic or partial. |
| Approach: | They propose a framework for automated scientific review using large language models . they propose to use a standardized review dataset to fine-tune an LLM to generate high-quality reviews. |
| Outcome: | The proposed framework can generate high-quality reviews from standardized datasets and improves on the existing feedback mechanisms. |
The lack of theory is painful: Modeling Harshness in Peer Review Comments (2022.aacl-main)
Copied to clipboard
| Challenge: | a new study shows that peer-review has a power imbalance, making it fraught for authors . authors argue that a little more effort to remain critical but be constructive would help foster a positive outcome . |
| Approach: | They propose to use a dataset to show peer-review comments' harshness scores . they argue that this moderation could help authors to be more constructive . |
| Outcome: | The proposed dataset shows that it can be used to make peer reviews less hurtful and more welcoming. |
ReviewEval: An Evaluation Framework for AI-Generated Reviews (2025.findings-emnlp)
Copied to clipboard
| Challenge: | escalating volume of academic research necessitates innovative approaches to peer review . authors propose reviewEval, ReviewAgent and ReviewEval to improve on existing reviews . |
| Approach: | They propose a framework for AI-generated reviews that measures alignment with human assessments . they propose 'reviewAgent' that iteratively optimizes its intermediate outputs and external improvement loops . |
| Outcome: | The proposed framework improves actionable insights and analytical depth by 6.78% and 47.62% over baselines and expert reviews. |
Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026.acl-long)
Copied to clipboard
Sihong Wu, Owen Jiang, Yilun Zhao, Tiansheng Hu, Yiling Ma, Kaiyan Zhang, Manasi Patwardhan, Arman Cohan
| Challenge: | Recent advances in large language models (LLMs) motivated methods that assist or automate different stages of peer review pipeline. |
| Approach: | They synthesize techniques to enhance peer review generation and after-review tasks aligned to reviews. |
| Outcome: | The proposed methods improve the peer review process by fine-tuning strategies, agent-based systems, and emerging paradigms. |
Preference Optimization for Review Question Generation Improves Writing Quality (2026.findings-acl)
Copied to clipboard
| Challenge: | Peer reviewers are overloaded and face tight deadlines, leading some to rely on large language models (LLMs) to draft questions and comments. |
| Approach: | They use open-review review datasets to train a human preference model based on human reviewer questions . human evaluations show IntelliAsk generates more grounded, substantive and effortful questions than strong baselines . |
| Outcome: | The proposed model predicts reviewer-question quality better than API-based SFT baselines and provides scalable evaluation. |
PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality (2026.findings-acl)
Copied to clipboard
| Challenge: | Increasing use of large language models (LLMs) in academic review has raised concerns about quality and fairness. |
| Approach: | They propose a framework to improve the quality of LLM-generated reviews by using retrieval-augmented generation. |
| Outcome: | The proposed framework improves the human-level quality of LLM-generated reviews by adopting prompt engineering and retrieval-augmented generation. |
Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework (2026.tacl-1)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used as fully automatic review generators (ARGs). |
| Approach: | They propose a fully automated counterfactual evaluation framework that isolates and tests a core review skill that underpins high-quality peer review: detecting faulty research logic. |
| Outcome: | The proposed framework isolates and tests a range of ARG approaches and shows that flaws in research logic have no significant effect on their output reviews. |
Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews (2025.emnlp-main)
Copied to clipboard
Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee, Juho Kim
| Challenge: | Large Language Models (LLMs) can automatically draft reviews, but determining whether they are trustworthy requires systematic evaluation. |
| Approach: | They propose an automatic focus-level evaluation pipeline based on two sets of facets . authors evaluated LLM reviews at surface-level or content-level . |
| Outcome: | The proposed framework enables automatic evaluation of paper reviews based on two sets of facets . the framework compared open review paper reviews with human experts on validity, clarity, novelty . |
Evaluating the Impact of Reviewer Guideline Design on LLM-Based Automated Peer Review (2026.findings-acl)
Copied to clipboard
| Challenge: | a growing workload has made peer review automation an urgent necessity, says a new study . official conference guidelines and reviewer-imitating guidelines degraded review performance . current human-based peer review system faces serious challenges, authors say . |
| Approach: | They analyze how reviewer guidelines influence automated peer review . official conference guidelines produce review results consistent with human judgments . |
| Outcome: | The proposed reviewer guidelines produce results consistent with human judgments . the proposed reviewers' imitations degraded performance, the authors note . |
CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells (2025.naacl-long)
Copied to clipboard
| Challenge: | Current review comment evaluation metrics rely on comparisons with a human-written reference for a given code change (also called a diff). |
| Approach: | They propose to use a reference-free metric to measure review quality like conciseness, comprehensiveness, and relevance to compare the quality of code changes with human-written references. |
| Outcome: | The proposed metric can produce fine-grained scores that have the greatest alignment with human judgment and are more sensitive than reference-based metrics. |