A Scalable Entity-Based Framework for Auditing Bias in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to bias evaluation in large language models trade ecological validity for statistical control, or use artificial prompts that lack scale and rigor. |
| Approach: | They propose a framework that uses named entities as probes to measure bias in large language models. |
| Outcome: | The proposed framework reproduces bias patterns observed in natural text, enabling large-scale analysis. |
Similar Papers
A Dual-Layered Evaluation of Geopolitical and Cultural Bias in LLMs (2025.acl-srw)
Copied to clipboard
| Challenge: | Large language models exhibit cultural and geopolitical biases when their outputs shape public opinion or reinforce dominant narratives. |
| Approach: | They define two types of bias in large language models: model bias and inference bias through a two-phase evaluation. |
| Outcome: | The proposed framework evaluates large language models on factual and disputable questions across four languages and question types. |
Beyond Performance: Quantifying and Mitigating Label Bias in LLMs (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models exhibit undesirable preference toward predicting certain answers over others, despite their adaptability to diverse tasks. |
| Approach: | They propose a label bias calibration method that outperforms recent calibration approaches for improving performance and mitigating label bias. |
| Outcome: | The proposed method outperforms calibration approaches for improving performance and mitigating label bias. |
ROBBIE: Robust Bias Evaluation of Large Generative Language Models (2023.emnlp-main)
Copied to clipboard
David Esiobu, Xiaoqing Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi-Yu, Eleonora Presani, Adina Williams, Eric Smith
| Challenge: | generative large language models (LLMs) are becoming more performant and prevalent . we need tools to measure and improve their fairness, authors say . |
| Approach: | They propose to compare 6 different prompt-based bias and toxicity metrics across 12 demographic axes and 5 families of generative large language models. |
| Outcome: | The proposed model can be tested on more datasets to better characterize and mitigate biases . the study compared 6 prompt-based bias and toxicity metrics across 12 demographic axes and 5 families of generative large language models. |
A Monte-Carlo Sampling Framework For Reliable Evaluation of Large Language Models Using Behavioral Analysis (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current approaches to evaluation of large language models ignore high entropy of LLM responses. |
| Approach: | They propose a Monte-Carlo evaluation framework for evaluating large language models . they test multiple LLMs to see if they are susceptible to cognitive biases . |
| Outcome: | The proposed framework shows that LLMs are more human-like and less rational . it also shows that larger LLM models are more susceptible to cognitive biases . |
Who Gets Which Message? Auditing Demographic Bias in LLM-Generated Targeted Text (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models generate demographically conditioned persuasive texts at scale . authors argue that such capabilities raise questions about fairness and representational bias in automated communication. |
| Approach: | They propose a framework for evaluating demographic-conditioned targeted messages . they find gender- and age-based asymmetries in male- and youth-targeted messages a . |
| Outcome: | The proposed framework evaluates generated messages across three dimensions: lexical content, language style, and persuasive framing. |
Large Language Models Are Still Misled by Simple Bias Ensembles (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models are constrained to datasets where each sample is manually injected with only one type of bias. |
| Approach: | They propose a multi-bias benchmark where each sample contains multiple types of biases. |
| Outcome: | The proposed benchmark shows that existing LLMs and debiasing methods perform poorly on this benchmark, highlighting the challenge of eliminating compounded biases. |
A Group Fairness Lens for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods focusing on a few groups lack a comprehensive categorical perspective to evaluate LLMs’ potential biases and unfairness. |
| Approach: | They propose to evaluate LLM biases from a group fairness lens using a hierarchical schema characterizing diverse social groups. |
| Outcome: | The proposed method mitigates biases in LLMs from a group fairness lens and encapsulates target-attribute combinations across multiple dimensions. |
Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that large language models are susceptible to societal biases due to their exposure to human-generated data. |
| Approach: | They propose two strategies to mitigate implicit gender biases in large language models . they create scenarios where implicit gender is present and develop a metric to assess the presence of biase . |
| Outcome: | The proposed methods mitigate implicit biases with self-reflection and fine-tuning. |
Measuring and Mitigating Media Outlet Name Bias in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have explored the potential political biases of large language models, but limited attention has been devoted to the effects of media outlet names. |
| Approach: | They propose to quantify media outlet name biases in large language models and leverage this metric to develop an automated prompt optimization framework. |
| Outcome: | The proposed framework mitigates media outlet name biases, offering a scalable approach to enhancing the fairness of LLMs in news-related applications. |
Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have found evidence of gender bias in machine translation and coreference resolution models using mostly synthetic diagnostic datasets. |
| Approach: | They propose a semi-automatic method to vastly extend synthetic, small diagnostic datasets to include grammatical patterns indicating stereotypical and non-stereotypical gender-role assignments. |
| Outcome: | The proposed method extends the existing dataset to 108K diverse English sentences. |