This Land is Your, My Land: Evaluating Geopolitical Bias in Language Models through Territorial Disputes (2024.naacl-long)
Copied to clipboard
| Challenge: | Pretrained large language models may answer differently in different languages . this contrasts with a multilingual human, who would likely answer consistently . |
| Approach: | They propose a dataset of territorial disputes which includes multiple-choice questions in 49 languages . they propose metrics to quantify bias and consistency in responses across different languages based on their data . |
| Outcome: | The proposed model recalls certain knowledge inconsistently when asked in different languages. |
Similar Papers
A Dual-Layered Evaluation of Geopolitical and Cultural Bias in LLMs (2025.acl-srw)
Copied to clipboard
| Challenge: | Large language models exhibit cultural and geopolitical biases when their outputs shape public opinion or reinforce dominant narratives. |
| Approach: | They define two types of bias in large language models: model bias and inference bias through a two-phase evaluation. |
| Outcome: | The proposed framework evaluates large language models on factual and disputable questions across four languages and question types. |
Navigating the Political Compass: Evaluating Multilingual LLMs across Languages and Nationalities (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are ubiquitous in today’s technological landscape, boasting a plethora of applications, and even endangering human jobs in complex and creative fields. |
| Approach: | They evaluate the political bias of 15 multilingual LLMs using the Political Compass Test and assign a nationality to each model. |
| Outcome: | The models on the 50 most populous countries and their official languages exhibit political bias. |
Bias in the East, Bias in the West: A Bilingual Analysis of LLM Political Bias on U.S.- and China-Related Issues (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can exhibit political biases, which creates a risk of undue influence on LLM users and public opinion. |
| Approach: | They use a dataset of 36k real-time test prompts to measure LLM political bias on U.S. and Chinese issues. |
| Outcome: | The proposed model origin and prompt language influence bias on 60 political issues. |
7 Points to Tsinghua but 10 Points to ? Assessing Large Language Models in Agentic Multilingual National Bias (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models have garnered significant attention for their capabilities in multilingual natural language processing, but studies on risks associated with cross biases are limited to immediate context preferences. |
| Approach: | They investigate multilingual bias in state-of-the-art Large Language Models by analyzing their responses to decision-making tasks across multiple languages. |
| Outcome: | The proposed model can provide personalized advice across university applications, travel, and relocation scenarios. |
Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs (2026.acl-long)
Copied to clipboard
Guy Mor-Lan, Omer Goldman, Matan Eyal, Adi Mayrav Gilady, Sivan Eiger, Idan Szpektor, Avinatan Hassidim, Yossi Matias, Reut Tsarfaty
| Challenge: | Multilingual large language models have minimized the fluency gap between languages, but they are exposed to the risk of biases as knowledge and norms may propagate across languages. |
| Approach: | They propose a test set with 2,156 questions in 12 languages to quantify models' biases . they show a global bias towards answers relevant to the US-locale . |
| Outcome: | The proposed model can answer locale-ambiguous questions in 12 languages. |
I Know, but I Don’t Know! How Persona Conflict Undermines Instruction Adherence in Large Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on persona-grounded dialogue assume idealized scenarios where persona and user utterances are fully aligned. |
| Approach: | They propose a taxonomy that categorizes model behaviors into three response types . they propose sycophantic, adherent, and wavering responses as response types. |
| Outcome: | The proposed framework categorizes model behaviors into three response types and develops a measurement schema grounded in this taxonomy. |
Measuring Geographic Performance Disparities of Offensive Language Classifiers (2022.coling-1)
Copied to clipboard
| Challenge: | Recent work shows that text classifiers are biased regarding different languages and dialects. |
| Approach: | They propose to use a dataset to examine whether language, dialect, and topical content vary across geographical regions to address these gaps. |
| Outcome: | The proposed dataset includes 14 thousand examples across 15 cities and shows that current models do not generalize across locations. |
Towards Practical and Knowledgeable LLMs for a Multilingual World: A Thesis Proposal (2025.naacl-srw)
Copied to clipboard
| Challenge: | a proposed thesis examines the role that multilinguality occupies in the development of practical and knowledgeable LLMs. |
| Approach: | They propose to use multilingual knowledge to improve LLM performance on NLP tasks . they extend the territorial disputes benchmark to retrieval-augmented generation setting . |
| Outcome: | The proposed methods improve LLMs' performance on standard natural language processing tasks by leveraging their existing multilingual knowledge. |
Through the Looking Glass of Multilingual AI: Contrasting Language- and Name Script-Dependent Ethnic Hierarchies in GPT and DeepSeek (2026.acl-srw)
Copied to clipboard
| Challenge: | a recent study found that large language models are biased overwhelmingly Anglocentric . a stereotype perceptual map is a framework for analyzing how ethnic groups are positioned along evaluative dimensions. |
| Approach: | They use a stereotype perceptual map to examine how ethnic groups are positioned along evaluative dimensions. |
| Outcome: | The stereotype perceptual map analyzes model behavior across languages, scripts, evaluative domains and models. |
Mind the Gap: Multilingual Divide in LLM Bias Detection and Reasoning (2026.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly deployed in multilingual settings . but most bias evaluation remains English-centric and ignores how bias manifests within reasoning . |
| Approach: | They evaluate large language models with supervised fine-tuning and preference optimization . they find that bias varies substantially across languages, with consistent degradation in non-English settings . |
| Outcome: | The proposed model improves in English, Dutch, Spanish, and Turkish using the MBBQ benchmark. |