Position Paper: How Should We Responsibly Adopt LLMs in the Peer Review Process? (2026.findings-eacl)
Copied to clipboard
| Challenge: | a recent paper criticizes the current use of Large Language Models (LLMs) for simple review text generation. |
| Approach: | They propose to use Large Language Models to support key aspects of the review process . they argue that this approach overlooks more meaningful applications of LLMs . authors argue that the increased reviewing burden per reviewer is a factor . |
| Outcome: | The proposed approach would support reproducibility, correctness and relevance of citations and ethics review flagging. |
Similar Papers
Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026.acl-long)
Copied to clipboard
Sihong Wu, Owen Jiang, Yilun Zhao, Tiansheng Hu, Yiling Ma, Kaiyan Zhang, Manasi Patwardhan, Arman Cohan
| Challenge: | Recent advances in large language models (LLMs) motivated methods that assist or automate different stages of peer review pipeline. |
| Approach: | They synthesize techniques to enhance peer review generation and after-review tasks aligned to reviews. |
| Outcome: | The proposed methods improve the peer review process by fine-tuning strategies, agent-based systems, and emerging paradigms. |
LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing (2024.emnlp-main)
Copied to clipboard
Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Cheng Jiayang, Zhaowei Wang, Ying Su, Raj Shah, Ruohao Guo, Jing Gu, Haoran Li, Kangda Wei, Zihao Wang, Lu Cheng, Surangika Ranathunga, Meng Fang, Jie Fu, Fei Liu, Ruihong Huang, Eduardo Blanco, Yixin Cao, Rui Zhang, Philip Yu, Wenpeng Yin
| Challenge: | a comparative analysis of paper (meta-)reviews by large language models (LLMs) aims to identify and distinguish LLMs from human activities . |
| Approach: | They present a comparative analysis to identify and distinguish LLM activities from human activities. |
| Outcome: | The proposed analysis aims to improve recognition of instances when someone implicitly uses LLMs for reviewing activities. |
Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Composition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a promising solution to automate literature review writing tasks. |
| Approach: | They propose a framework to automatically evaluate the performance of large language models in three key tasks of literature review writing: reference generation, abstract writing, and literature review composition. |
| Outcome: | The proposed framework assesses the hallucination rates in generated references and measures the semantic coverage and factual consistency of the literature summaries and compositions against human-written counterparts. |
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2025.emnlp-main)
Copied to clipboard
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, Huan Liu
| Challenge: | Recent advances in Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm . traditional methods of assessment and evaluation fail in dynamic and open-ended scenarios . |
| Approach: | They propose a paradigm where LLMs are leveraged to perform scoring, ranking, or selection for machine learning evaluation scenarios. |
| Outcome: | The proposed model-based judgment and evaluation paradigms are based on large language models and are compared to the current model-driven evaluation paradigm. |
Can Large Language Models Be an Alternative to Human Evaluations? (2023.acl-long)
Copied to clipboard
| Challenge: | Human evaluation is indispensable for assessing the quality of texts generated by machine learning models or written by humans. |
| Approach: | They propose to use large language models to evaluate unseen texts using the same instructions and samples . they also use LLMs to generate responses to questions that are used to conduct human evaluation . |
| Outcome: | The proposed model can be used to evaluate texts in open-ended story generation and adversarial attacks. |
Breaking the Reviewer: Assessing the Vulnerability of Large Language Models in Automated Peer Review Under Textual Adversarial Attacks (2025.findings-emnlp)
Copied to clipboard
Tzu-Ling Lin, Wei-Chih Chen, Teng-Fang Hsiao, Hou-I Liu, Ya-Hsin Yeh, Yu-Kai Chan, Wen-Sheng Lien, Po-Yen Kuo, Philip S. Yu, Hong-Han Shuai
| Challenge: | Large language models (LLMs) are used to review academic papers, but are susceptible to textual adversarial attacks. |
| Approach: | They evaluate the robustness of large language models as automated reviewers in the presence of adversarial attacks. |
| Outcome: | The proposed model is robust against textual adversarial attacks, the authors argue . their findings highlight the importance of addressing adversarials to ensure integrity of scholarly communication. |
A Survey on LLMs for Story Generation (2025.findings-emnlp)
Copied to clipboard
Maria Teleki, Vedangi Bengali, Xiangjue Dong, Sai Tejas Janjur, Haoran Liu, Tian Liu, Cong Wang, Ting Liu, Yin Zhang, Frank Shipman, James Caverlee
| Challenge: | Methods for story generation with Large Language Models (LLMs) have come into the spotlight recently. |
| Approach: | They propose a novel taxonomy of LLMs for story generation consisting of two major paradigms: independent story generation by an LLM, and author-assistance for story creation . |
| Outcome: | The proposed taxonomy compares existing work on the topic with those of novel author-assistance models. |
The Impossibility of Fair LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Existing frameworks for evaluating large language models do not extend to general-purpose AI contexts or are infeasible in practice. |
| Approach: | They analyze a variety of technical fairness frameworks to find inherent challenges . they find that each framework does not logically extend to the general-purpose AI context . |
| Outcome: | The proposed frameworks do not logically extend to the general-purpose AI context or are infeasible in practice due to large amounts of unstructured training data and potential combinations of human populations, use cases, and sensitive attributes. |
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)
Copied to clipboard
| Challenge: | introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. |
| Approach: | They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them. |
| Outcome: | The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods. |
LLMs as Meta-Reviewers’ Assistants: A Case Study (2025.naacl-long)
Copied to clipboard
Eftekhar Hossain, Sanjeev Kumar Sinha, Naman Bansal, R. Alexander Knipper, Souvika Sarkar, John Salvador, Yash Mahajan, Sri Ram Pavan Kumar Guttikonda, Mousumi Akter, Md. Mahadi Hassan, Matthew Freestone, Matthew C. Williams Jr., Dongji Feng, Santu Karmaker
| Challenge: | Meta-reviews are a critical step in the overall scientific peer-reviewed process, which focuses on understanding the consensus of expert opinions on a scholarly work and making informed judgments on its scientific merit. |
| Approach: | They propose to use large language models to generate a controlled multi-perspective-summary (MPS) of their opinions to help meta-reviewers better comprehend multiple experts' perspectives. |
| Outcome: | The proposed model can help meta-reviewers better comprehend multiple experts’ perspectives by generating a controlled multi-perspective-summary (MPS) of their opinions. |