Stereotype Bias in a Bilingual Setting: A Culturally Grounded Evaluation in Kazakhstan (2026.acl-long)
Copied to clipboard
Nurkhan Laiyk, Daniil Orel, Ayana Mussabayeva, Maiya Goloburda, Kamila Kuishibekova, Liya Goloburda, Diana Turmakhan, Preslav Nakov, Yuxia Wang, Fajri Koto
| Challenge: | Stereotype bias in language models is largely understudied in English . language models perform strongly on downstream NLP tasks, but they are pre-trained on large text corpora . |
| Approach: | They use a dataset to assess stereotype bias in language models in Kazakhstan . they find that stereotype bias is most pronounced in code-mixed inputs . |
| Outcome: | The proposed dataset shows that stereotype bias is most pronounced in code-mixed inputs. |
Similar Papers
Your Stereotypical Mileage May Vary: Practical Challenges of Evaluating Biases in Multiple Languages and Cultural Contexts (2024.lrec-main)
Copied to clipboard
Karen Fort, Laura Alonso Alemany, Luciana Benotti, Julien Bezançon, Claudia Borg, Marthese Borg, Yongjian Chen, Fanny Ducel, Yoann Dupont, Guido Ivetta, Zhijian Li, Margot Mieskes, Marco Naguib, Yuyan Qian, Matteo Radaelli, Wolfgang S. Schmeisser-Nieto, Emma Raimundo Schulz, Thiziri Saci, Sarah Saidi, Javier Torroba Marchante, Shilin Xie, Sergio E. Zanotto, Aurélie Névéol
| Challenge: | Recent studies have identified a gap in the availability of tools and resources to study bias in languages other than English and social contexts outside the north of America. |
| Approach: | They use stereotypes to build a corpus of sentence pairs that cover biases in seven cultural contexts. |
| Outcome: | The proposed resource covers a wide range of languages and cultural settings . it favors sentences that express stereotypes in most bias categories . |
StereoSet: Measuring stereotypical bias in pretrained language models (2021.acl-long)
Copied to clipboard
| Challenge: | Existing literature on stereotypical biases in language models is limited . current evaluations focus on measuring bias without considering language modeling ability . |
| Approach: | They propose to measure stereotypical biases in four domains: gender, profession, race, and religion . they compare stereotypical and language modeling ability of popular models like BERT, GPT-2, RoBERTa and XLnet . |
| Outcome: | The proposed model shows strong stereotypical biases in gender, profession, race, and religion domains. |
Global Voices, Local Biases: Socio-Cultural Prejudices across Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on human biases are heavily skewed towards Western and European languages . despite growing interest in language models, there are several shortcomings in the literature . |
| Approach: | They scale the Word Embedding Association Test to 24 languages and add culturally relevant information for each language. |
| Outcome: | The proposed language models can reflect and often amplify the effects of bias across linguistic, cultural, and societal borders. |
Blind Men and the Elephant: Diverse Perspectives on Gender Stereotypes in Benchmark Datasets (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for measuring gender stereotypical bias in language models are inconsistencies . lack of explicit standards in data gathering can have detrimental effects on results . |
| Approach: | They propose that currently available benchmarks capture only partial facets of gender stereotypes . they apply a framework from social psychology to balance data across components of gender stereotypes based on stereotypical benchmarks. |
| Outcome: | The proposed framework improves correlation between different benchmarks by using simple balancing techniques. |
Social Bias in Multilingual Language Models: A Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained multilingual models exhibit the same social bias as models processing English texts. |
| Approach: | They examine the literature on bias evaluation and mitigation approaches in multilingual and non-English contexts and identify gaps in the field. |
| Outcome: | The proposed models perform well on multilingual language understanding benchmarks and are consistent with the current literature. |
Intersectional Stereotypes in Large Language Models: Dataset and Analysis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on intersectional stereotypes focus on broader, individual categories . current studies focus on single-group stereotypes, such as racial bias against African Americans . |
| Approach: | They propose to use a dataset of intersectional stereotypes curated with the ChatGPT model to analyze propagation in three contemporary LLMs. |
| Outcome: | The proposed dataset enables analysis of stereotype propagation in three contemporary LLMs. |
SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models (2025.naacl-long)
Copied to clipboard
Margaret Mitchell, Giuseppe Attanasio, Ioana Baldini, Miruna Clinciu, Jordan Clive, Pieter Delobelle, Manan Dey, Sil Hamilton, Timm Dill, Jad Doughman, Ritam Dutt, Avijit Ghosh, Jessica Zosa Forde, Carolin Holtermann, Lucie-Aimée Kaffee, Tanmay Laud, Anne Lauscher, Roberto L Lopez-Davila, Maraim Masoud, Nikita Nangia, Anaelia Ovalle, Giada Pistilli, Dragomir Radev, Beatrice Savoldi, Vipul Raheja, Jeremy Qin, Esther Ploeger, Arjun Subramonian, Kaustubh Dhole, Kaiser Sun, Amirbek Djanibekov, Jonibek Mansurov, Kayo Yin, Emilio Villa Cueva, Sagnik Mukherjee, Jerry Huang, Xudong Shen, Jay Gala, Hamdan Al-Ali, null Tair Djanibekov, Nurdaulet Mukhituly, Shangrui Nie, Shanya Sharma, Karolina Stanczak, Eliza Szczechla, Tiago Timponi Torrent, Deepak Tunuguntla, Marcelo Viridiano, Oskar Van Der Wal, Adina Yakefu, Aurélie Névéol, Mike Zhang, Sydney Zink, Zeerak Talat
| Challenge: | Large Language Models reproduce and exacerbate social biases present in training data, and resources to quantify this issue are limited. |
| Approach: | They propose a multilingual parallel dataset to examine culturally-specific stereotypes that may be learned by LLMs. |
| Outcome: | The proposed dataset includes stereotypes from 20 regions around the world and 16 languages, spanning multiple identity categories subject to discrimination worldwide. |
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation (2025.acl-long)
Copied to clipboard
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, Sara Hooker
| Challenge: | Reliable multilingual evaluation is difficult and culturally appropriate evaluation is even harder to achieve. |
| Approach: | They propose a multilingual evaluation framework that aims to mitigate these biases by improving translations and annotation practices. |
| Outcome: | The proposed framework improves translation quality and cultural coverage and is culturally sensitive and culturally agnostic. |
A Dual-Layered Evaluation of Geopolitical and Cultural Bias in LLMs (2025.acl-srw)
Copied to clipboard
| Challenge: | Large language models exhibit cultural and geopolitical biases when their outputs shape public opinion or reinforce dominant narratives. |
| Approach: | They define two types of bias in large language models: model bias and inference bias through a two-phase evaluation. |
| Outcome: | The proposed framework evaluates large language models on factual and disputable questions across four languages and question types. |
Stereotype Detection as a Catalyst for Enhanced Bias Detection: A Multi-Task Learning Approach (2025.findings-acl)
Copied to clipboard
| Challenge: | a new study addresses bias and stereotypes in language models by exploring how learning them together improves performance. |
| Approach: | They propose a dataset for bias and stereotype detection that integrates religion, gender, socio-economic status, race, profession, and others. |
| Outcome: | The proposed dataset compares encoder-only models and fine-tuned decoder- only models . the results show that learning stereotypes together improves bias detection . |