A Multilingual Social Bias Benchmark Incorporating Thinking Processes (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can learn useful knowledge and harmful stereotypes, making bias evaluation essential. |
| Approach: | They propose a multilingual social bias benchmark that incorporates human-generated reasoning as part of the thinking process. |
| Outcome: | The proposed method demonstrates superior performance over LLM-generated methods . human-generated thinking yields higher-quality evaluations than template-based approaches . |
Similar Papers
Mind the Gap: Multilingual Divide in LLM Bias Detection and Reasoning (2026.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly deployed in multilingual settings . but most bias evaluation remains English-centric and ignores how bias manifests within reasoning . |
| Approach: | They evaluate large language models with supervised fine-tuning and preference optimization . they find that bias varies substantially across languages, with consistent degradation in non-English settings . |
| Outcome: | The proposed model improves in English, Dutch, Spanish, and Turkish using the MBBQ benchmark. |
Social Bias in Multilingual Language Models: A Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained multilingual models exhibit the same social bias as models processing English texts. |
| Approach: | They examine the literature on bias evaluation and mitigation approaches in multilingual and non-English contexts and identify gaps in the field. |
| Outcome: | The proposed models perform well on multilingual language understanding benchmarks and are consistent with the current literature. |
Does Reasoning Introduce Bias? A Study of Social Bias Evaluation and Mitigation in LLM Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large language models have enabled automatic generation of chain-of-thought reasoning . however, when reasoning steps reflect social stereotypes, they can reinforce harmful associations and lead to misleading conclusions. |
| Approach: | They propose a method that detects how model predictions change across incremental reasoning steps. |
| Outcome: | The proposed method outperforms a stereotype-free baseline and improves accuracy. |
Social Bias Benchmark for Generation: A Comparison of Generation and QA-Based Evaluations (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for assessing social bias in large language models (LLMs) do not capture nuanced and context-dependent nature of natural language generation. |
| Approach: | They propose a Bias Benchmark for Generation (BBG) that evaluates social bias in long-form generation by having LLMs generate continuations of story prompts. |
| Outcome: | The proposed benchmark is based on the English BBQ and Korean BBQ datasets and compares it with multiplechoice BBQ evaluation. |
Multi-Persona Thinking for Bias Mitigation in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models exhibit social biases, which can lead to harmful stereotypes and unfair outcomes. |
| Approach: | They propose a simple inference-time framework that encourages reasoning from multiple perspectives. |
| Outcome: | The proposed framework reduces bias by encouraging reasoning from multiple perspectives. |
7 Points to Tsinghua but 10 Points to ? Assessing Large Language Models in Agentic Multilingual National Bias (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models have garnered significant attention for their capabilities in multilingual natural language processing, but studies on risks associated with cross biases are limited to immediate context preferences. |
| Approach: | They investigate multilingual bias in state-of-the-art Large Language Models by analyzing their responses to decision-making tasks across multiple languages. |
| Outcome: | The proposed model can provide personalized advice across university applications, travel, and relocation scenarios. |
TWBias: A Benchmark for Assessing Social Bias in Traditional Chinese Large Language Models through a Taiwan Cultural Lens (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models have shown remarkable capabilities in natural language processing, but concerns about social bias amplification remain. |
| Approach: | They propose a social bias evaluation benchmark for Traditional Chinese LLMs that integrates chat templates and diverse prompts for comprehensive bias assessment. |
| Outcome: | The proposed model incorporates chat templates and diverse prompts for comprehensive bias assessment focusing on Taiwan's cultural context and prioritizing gender and ethnicity bias evaluation. |
Your Stereotypical Mileage May Vary: Practical Challenges of Evaluating Biases in Multiple Languages and Cultural Contexts (2024.lrec-main)
Copied to clipboard
Karen Fort, Laura Alonso Alemany, Luciana Benotti, Julien Bezançon, Claudia Borg, Marthese Borg, Yongjian Chen, Fanny Ducel, Yoann Dupont, Guido Ivetta, Zhijian Li, Margot Mieskes, Marco Naguib, Yuyan Qian, Matteo Radaelli, Wolfgang S. Schmeisser-Nieto, Emma Raimundo Schulz, Thiziri Saci, Sarah Saidi, Javier Torroba Marchante, Shilin Xie, Sergio E. Zanotto, Aurélie Névéol
| Challenge: | Recent studies have identified a gap in the availability of tools and resources to study bias in languages other than English and social contexts outside the north of America. |
| Approach: | They use stereotypes to build a corpus of sentence pairs that cover biases in seven cultural contexts. |
| Outcome: | The proposed resource covers a wide range of languages and cultural settings . it favors sentences that express stereotypes in most bias categories . |
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation (2025.acl-long)
Copied to clipboard
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, Sara Hooker
| Challenge: | Reliable multilingual evaluation is difficult and culturally appropriate evaluation is even harder to achieve. |
| Approach: | They propose a multilingual evaluation framework that aims to mitigate these biases by improving translations and annotation practices. |
| Outcome: | The proposed framework improves translation quality and cultural coverage and is culturally sensitive and culturally agnostic. |
Rethinking Pragmatics in Large Language Models: Towards Open-Ended Evaluation and Preference Tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to assess social-pragmatic inference in large language models are inadequacy, and preferential tuning is the best approach. |
| Approach: | They propose to use free-form models' responses as a measure to assess social-pragmatic reasoning and advocate for preference optimization over supervised finetuning (SFT). |
| Outcome: | The proposed model outperforms supervised finetuning (SFT) and offers a near-free launch in pragmatic abilities without compromising general capabilities. |