Generative Reviewer Agents: Scalable Simulacra of Peer Review (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing peer review mechanisms are limited by the small fraction of researchers with established networks. |
| Approach: | They propose a system that extends a large language model and equips agents with reviewer personas derived from historical data to enable generative reviewers. |
| Outcome: | The proposed architecture performs comparable to human reviewers in providing detailed feedback and predicting paper outcomes. |
Similar Papers
Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026.acl-long)
Copied to clipboard
Sihong Wu, Owen Jiang, Yilun Zhao, Tiansheng Hu, Yiling Ma, Kaiyan Zhang, Manasi Patwardhan, Arman Cohan
| Challenge: | Recent advances in large language models (LLMs) motivated methods that assist or automate different stages of peer review pipeline. |
| Approach: | They synthesize techniques to enhance peer review generation and after-review tasks aligned to reviews. |
| Outcome: | The proposed methods improve the peer review process by fine-tuning strategies, agent-based systems, and emerging paradigms. |
Position Paper: How Should We Responsibly Adopt LLMs in the Peer Review Process? (2026.findings-eacl)
Copied to clipboard
| Challenge: | a recent paper criticizes the current use of Large Language Models (LLMs) for simple review text generation. |
| Approach: | They propose to use Large Language Models to support key aspects of the review process . they argue that this approach overlooks more meaningful applications of LLMs . authors argue that the increased reviewing burden per reviewer is a factor . |
| Outcome: | The proposed approach would support reproducibility, correctness and relevance of citations and ethics review flagging. |
AgentReview: Exploring Peer Review Dynamics with LLM Agents (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods of peer review analysis do not address multivariate nature of the process, account for latent variables, and are constrained by privacy concerns due to the sensitive nature of data. |
| Approach: | They propose a large language model based peer review simulation framework which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. |
| Outcome: | The proposed framework disentangles the impacts of multiple latent factors and addresses privacy concerns. |
ReviewEval: An Evaluation Framework for AI-Generated Reviews (2025.findings-emnlp)
Copied to clipboard
| Challenge: | escalating volume of academic research necessitates innovative approaches to peer review . authors propose reviewEval, ReviewAgent and ReviewEval to improve on existing reviews . |
| Approach: | They propose a framework for AI-generated reviews that measures alignment with human assessments . they propose 'reviewAgent' that iteratively optimizes its intermediate outputs and external improvement loops . |
| Outcome: | The proposed framework improves actionable insights and analytical depth by 6.78% and 47.62% over baselines and expert reviews. |
Autonomous Machine Learning-Based Peer Reviewer Selection System (2025.coling-demos)
Copied to clipboard
| Challenge: | Existing systems that match papers with experts are inefficient and often require long turnaround times. |
| Approach: | They propose an autonomous peer reviewer selection system that employs the natural language processing model to match submitted papers with expert reviewers independently of traditional journals and conferences. |
| Outcome: | The proposed system performs faster and smaller than current models while being more scalable. |
Breaking the Reviewer: Assessing the Vulnerability of Large Language Models in Automated Peer Review Under Textual Adversarial Attacks (2025.findings-emnlp)
Copied to clipboard
Tzu-Ling Lin, Wei-Chih Chen, Teng-Fang Hsiao, Hou-I Liu, Ya-Hsin Yeh, Yu-Kai Chan, Wen-Sheng Lien, Po-Yen Kuo, Philip S. Yu, Hong-Han Shuai
| Challenge: | Large language models (LLMs) are used to review academic papers, but are susceptible to textual adversarial attacks. |
| Approach: | They evaluate the robustness of large language models as automated reviewers in the presence of adversarial attacks. |
| Outcome: | The proposed model is robust against textual adversarial attacks, the authors argue . their findings highlight the importance of addressing adversarials to ensure integrity of scholarly communication. |
BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers? (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evidence suggests that LLMs are not able to detect scientifically unsound work from malicious or poorly designed research agents. |
| Approach: | They develop a framework that evaluates whether fabrication-oriented paper generation agents can deceive multi-model LLM review systems. |
| Outcome: | The proposed framework shows that fabricated papers achieve acceptance rates up to 18% . the framework shows only marginal improvements, with detection accuracy barely exceeding random chance. |
LLMs as Meta-Reviewers’ Assistants: A Case Study (2025.naacl-long)
Copied to clipboard
Eftekhar Hossain, Sanjeev Kumar Sinha, Naman Bansal, R. Alexander Knipper, Souvika Sarkar, John Salvador, Yash Mahajan, Sri Ram Pavan Kumar Guttikonda, Mousumi Akter, Md. Mahadi Hassan, Matthew Freestone, Matthew C. Williams Jr., Dongji Feng, Santu Karmaker
| Challenge: | Meta-reviews are a critical step in the overall scientific peer-reviewed process, which focuses on understanding the consensus of expert opinions on a scholarly work and making informed judgments on its scientific merit. |
| Approach: | They propose to use large language models to generate a controlled multi-perspective-summary (MPS) of their opinions to help meta-reviewers better comprehend multiple experts' perspectives. |
| Outcome: | The proposed model can help meta-reviewers better comprehend multiple experts’ perspectives by generating a controlled multi-perspective-summary (MPS) of their opinions. |
PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality (2026.findings-acl)
Copied to clipboard
| Challenge: | Increasing use of large language models (LLMs) in academic review has raised concerns about quality and fairness. |
| Approach: | They propose a framework to improve the quality of LLM-generated reviews by using retrieval-augmented generation. |
| Outcome: | The proposed framework improves the human-level quality of LLM-generated reviews by adopting prompt engineering and retrieval-augmented generation. |
LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing (2024.emnlp-main)
Copied to clipboard
Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Cheng Jiayang, Zhaowei Wang, Ying Su, Raj Shah, Ruohao Guo, Jing Gu, Haoran Li, Kangda Wei, Zihao Wang, Lu Cheng, Surangika Ranathunga, Meng Fang, Jie Fu, Fei Liu, Ruihong Huang, Eduardo Blanco, Yixin Cao, Rui Zhang, Philip Yu, Wenpeng Yin
| Challenge: | a comparative analysis of paper (meta-)reviews by large language models (LLMs) aims to identify and distinguish LLMs from human activities . |
| Approach: | They present a comparative analysis to identify and distinguish LLM activities from human activities. |
| Outcome: | The proposed analysis aims to improve recognition of instances when someone implicitly uses LLMs for reviewing activities. |