Justice in Judgment: Unveiling (Hidden) Bias in LLM-assisted Peer Reviews (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that large language models carry implicit biases across race, gender, and religion . prior studies documented such biase based on text generation and classification tasks . |
| Approach: | They investigate bias in large language models by controlling metadata on author metadata . authors found affiliation bias favoring authors from highly ranked institutions . |
| Outcome: | The proposed model favors authors from highly ranked institutions, the authors show . the model also favors author affiliations from highly-ranked institutions . |
Similar Papers
Humans or LLMs as the Judge? A Study on Judgement Bias (2024.emnlp-main)
Copied to clipboard
| Challenge: | Proprietary models such as GPT-4, Claude, Gemini-Pro and others are being democratized to improve evaluations of LLMs. |
| Approach: | They propose a framework that is free from referencing groundtruth annotations for investigating **Misinformation Oversight Bias**, **Gender Bia**,**Authority Bia* and **Beauty Bia's** on LLM and human judges. |
| Outcome: | The proposed framework investigates **Misinformation Oversight Bias**, **Gender Bia**,**Authority Bia* and **Beauty Bia' on LLM and human judges. |
AgentReview: Exploring Peer Review Dynamics with LLM Agents (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods of peer review analysis do not address multivariate nature of the process, account for latent variables, and are constrained by privacy concerns due to the sensitive nature of data. |
| Approach: | They propose a large language model based peer review simulation framework which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. |
| Outcome: | The proposed framework disentangles the impacts of multiple latent factors and addresses privacy concerns. |
Bias in the Mirror : Are LLMs opinions robust to their own adversarial attacks (2025.acl-long)
Copied to clipboard
| Challenge: | Existing work on large language models lacks robustness, highlighting the limitations of such models. |
| Approach: | They propose a novel approach where two LLMs engage in self-debate to persuade a neutral version of the model. |
| Outcome: | The proposed approach examines whether large language models are robust during interactions and whether they are susceptible to reinforcing misinformation or shifting to harmful viewpoints. |
Position Paper: How Should We Responsibly Adopt LLMs in the Peer Review Process? (2026.findings-eacl)
Copied to clipboard
| Challenge: | a recent paper criticizes the current use of Large Language Models (LLMs) for simple review text generation. |
| Approach: | They propose to use Large Language Models to support key aspects of the review process . they argue that this approach overlooks more meaningful applications of LLMs . authors argue that the increased reviewing burden per reviewer is a factor . |
| Outcome: | The proposed approach would support reproducibility, correctness and relevance of citations and ethics review flagging. |
Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception (2025.coling-main)
Copied to clipboard
| Challenge: | Detecting media bias is critical due to the spread of misinformation and disinformation on social media platforms. |
| Approach: | They investigate the presence and nature of bias within large language models and its consequential impact on media bias detection. |
| Outcome: | The proposed debiasing strategies include prompt engineering and model fine-tuning. |
Don’t Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used as evaluators for code evaluation tasks . however, whether they can handle superficial variations remains unclear . |
| Approach: | They define six types of potential biases in code evaluation and reveal their impact on LLM judges. |
| Outcome: | The proposed method can be used to evaluate semantically equivalent code with superficial variations without reference implementations. |
Prior Beliefs Prejudice LLM-as-Judge: Evidence from Persuasion Evaluation (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models are increasingly used as judges to evaluate text quality, content and assess arguments. |
| Approach: | They propose to exploit belief-conditioned rating inflation by using persuasion-based probing to examine persuasive arguments. |
| Outcome: | The proposed model fails to evaluate persuasive arguments based on belief alignment . the model fails in three of the three tasks, with belief-conditioned rating inflation accounting for 88% of cases. |
Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplification and Resistance in Multi-Agent Based LLM-as-Judge (2025.findings-emnlp)
Copied to clipboard
Chiyu Ma, Enpei Zhang, Yilun Zhao, Wenjun Liu, Yaning Jia, Peijun Qing, Lin Shi, Arman Cohan, Yujun Yan, Soroush Vosoughi
| Challenge: | LLM-as-Judge frameworks provide scalable alternative to human evaluation . but the question of how intrinsic biases manifest in these settings remains unexplored . |
| Approach: | They conduct systematic analysis of four bias types in multi-agent LLM-as-Judge frameworks . they find debate framework amplifies biases sharply after initial debate . |
| Outcome: | The proposed frameworks amplify biases after debate and show they are stronger in meta-judge scenarios. |
Beyond Performance: Quantifying and Mitigating Label Bias in LLMs (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models exhibit undesirable preference toward predicting certain answers over others, despite their adaptability to diverse tasks. |
| Approach: | They propose a label bias calibration method that outperforms recent calibration approaches for improving performance and mitigating label bias. |
| Outcome: | The proposed method outperforms calibration approaches for improving performance and mitigating label bias. |
Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study has shown that LLMs encode social biases and manifest in clinical tasks. |
| Approach: | They use mechanistic interpretability to uncover biases within LLMs . they find gender information is highly localized in MLP layers . |
| Outcome: | The proposed method can reveal biases and representations within LLMs in healthcare. |