Papers by Paul Röttger
Personalization up to a Point: Why Personalized Content Moderation Needs Boundaries, and How We Can Enforce Them (2025.emnlp-main)
Copied to clipboard
| Challenge: | Personalized content moderation can protect users from harm while facilitating free expression . however, it can also allow highly harmful and even illegal hate speech to spread . |
| Approach: | They propose to enforce legal boundaries on personalized content moderation models to reduce legal violations while maintaining user welfare. |
| Outcome: | The proposed approach reduces legal violations while maintaining user welfare while maintaining a high degree of model performance. |
Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work on persona prompting has shown mixed results on its effectiveness . prior work did not consider when and why personas should affect performance . |
| Approach: | They analyze literature on persona prompting and distill three desiderata for their effectiveness . they propose mitigation strategies to improve robustness but find they only work for the largest, most capable models . |
| Outcome: | The authors find that expert personas usually lead to positive or non-significant performance changes . they propose mitigation strategies to improve robustness but only for the largest models . |
AfriHate: A Multilingual Collection of Hate Speech and Abusive Language Datasets for African Languages (2025.naacl-long)
Copied to clipboard
Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Ayele, David Ifeoluwa Adelani, Ibrahim Said Ahmad, Saminu Mohammad Aliyu, Paul Röttger, Abigail Oppong, Andiswa Bukula, Chiamaka Ijeoma Chukwuneke, Ebrahim Chekol Jibril, Elyas Abdi Ismail, Esubalew Alemneh, Hagos Tesfahun Gebremichael, Lukman Jibril Aliyu, Meriem Beloucif, Oumaima Hourrane, Rooweither Mabuya, Salomey Osei, Samuel Rutunda, Tadesse Destaw Belay, Tadesse Kebede Guge, Tesfa Tegegne Asfaw, Lilian Diana Awuor Wanzare, Nelson Odhiambo Onyango, Seid Muhie Yimam, Nedjma Ousidhoum
| Challenge: | Hate speech and abusive language are global phenomena that need sociocultural background knowledge to be understood, identified, and moderated. |
| Approach: | They propose to use a multilingual dataset to collect hate speech and abusive language in 15 African languages to help improve model performance. |
| Outcome: | The proposed datasets are based on tweets annotated by native speakers familiar with the regional culture and show that they perform well in low-resource settings. |
IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance (2026.tacl-1)
Copied to clipboard
Paul Röttger, Musashi Hinck, Valentin Hofmann, Kobi Hackenburg, Valentina Pyatkin, Faeze Brahman, Dirk Hovy
| Challenge: | Large language models are helping millions of users write texts about diverse issues . issue bias is where an LLM tends to present just one perspective on a given issue . |
| Approach: | They construct a set of 2.49m realistic English-language prompts to measure issue bias in LLM writing assistance using 3.9k templates and 212 political issues from real user interactions. |
| Outcome: | The proposed model aligns more with US Democrat than Republican voter opinion on a subset of issues. |
“My Answer is C”: First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models (2024.findings-acl)
Copied to clipboard
Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, Barbara Plank
| Challenge: | Multiple choice questions are one of the most popular evaluation formats for understanding the capabilities of autoregressive large language models (LLMs). |
| Approach: | They evaluated how aligned first-token evaluation is with the text output along several dimensions, namely final option choice, refusal rate, choice distribution and robustness under prompt perturbation. |
| Outcome: | The proposed evaluation methods are misaligned on all dimensions, reaching mismatch rates over 60%. |
Improving the Detection of Multilingual Online Attacks with Rich Social Media Data from Singapore (2023.acl-long)
Copied to clipboard
Janosch Haber, Bertie Vidgen, Matthew Chapman, Vibhor Agarwal, Roy Ka-Wei Lee, Yong Keong Yap, Paul Röttger
| Challenge: | Toxic content is a global problem, but most resources for detecting toxic content are in English . new datasets and models for non-English languages focus exclusively on one language or dialect . |
| Approach: | They propose to use a multilingual dataset of online attacks to identify code-mixed toxic content in Singapore . they collect reddit comments in Indonesian, Malay, Singlish, and other languages and provide fine-grained hierarchical labels for attacks . |
| Outcome: | The proposed dataset provides fine-grained hierarchical labels for online attacks in Singapore . it shows that the metadata can be used for granular error analysis . |
HateCheck: Functional Tests for Hate Speech Detection Models (2021.acl-long)
Copied to clipboard
| Challenge: | Hate speech detection models are evaluated by measuring their performance on held-out test data using metrics such as accuracy and F1 score. |
| Approach: | They propose a suite of functional tests for hate speech detection models that measure model performance on held-out test data and then craft test cases to validate their quality. |
| Outcome: | The proposed tests show that the proposed models perform poorly on a small set of widely-used hate speech datasets. |
Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQ (2024.findings-acl)
Copied to clipboard
| Challenge: | a global majority of non-English speakers are underrepresented by large language models . however, most open LLMs are limited in their language coverage . |
| Approach: | They propose a silver standard benchmark for basic open-ended question answering with 27.4k test questions across a typologically diverse set of 137 languages. |
| Outcome: | The proposed model can answer questions in 27.4k questions across 137 languages. |
Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations (2025.naacl-long)
Copied to clipboard
| Challenge: | Prior work has focused on using large language models to simulate human behaviors . but, LLMs are known to generate erroneous, stereotypical, or overconfident answers . |
| Approach: | They propose to specialize large language models for simulating survey response distributions by first-token probabilities. |
| Outcome: | The proposed model outperforms other methods and zero-shot classifiers on unseen questions, countries, and a completely unseened survey. |
HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twitter (2025.acl-long)
Copied to clipboard
Manuel Tonneau, Diyi Liu, Niyati Malhotra, Scott A. Hale, Samuel Fraiberger, Victor Orozco-Olvera, Paul Röttger
| Challenge: | Prior work on automated hate speech detection models has been limited due to systematic biases in evaluation datasets and poor performance across geographies. |
| Approach: | They propose to construct a global hate speech dataset representative of social media settings from tweets posted on September 21, 2022. |
| Outcome: | The proposed dataset covers eight languages and four English-speaking countries and covers eight countries where English is the main language on Twitter. |
TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agent (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has focused on the (un)intended leakage of sensitive information through LLM outputs. |
| Approach: | They propose a threat model that embeds context information into natural-looking outputs via linguistic steganography without requiring explicit control over inference inputs. |
| Outcome: | The proposed model transmits 32-bit secrets with 87% accuracy on held-out prompts and can reach over 97% accuracy using majority voting across three generations. |
The Pluralistic Moral Gap: Understanding Moral Judgment and Value Differences between Humans and Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that Large Language Models (LLMs) are not fully aligned with human moral judgments. |
| Approach: | They propose a dataset of 1,618 real-world moral dilemmas paired with a distribution of human moral judgments consisting of a binary evaluation and a free-text rationale to examine how closely LLMs align with human moral judgements. |
| Outcome: | The proposed model reproduces human judgments only under high consensus; alignment deteriorates sharply when human disagreement increases. |
The Ecological Fallacy in Annotation: Modeling Human Label Variation goes beyond Sociodemographics (2023.acl-short)
Copied to clipboard
| Challenge: | Existing work has attempted to model individual annotation behaviour rather than predicting aggregated labels. |
| Approach: | They propose to model individual annotator behaviour rather than predicting aggregated labels by adding group-specific layers to multi-annotator models to account for sociodemographics. |
| Outcome: | The proposed model does not significantly improve on toxic content detection tasks. |
Around the World in 24 Hours: Probing LLM Knowledge of Time and Place (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies have tested language models' ability to reason over time and space in isolation or only in simple or artificial environments. |
| Approach: | They present a dataset of 320k prompts covering 289 cities in 217 countries and 37 time zones to evaluate their ability to jointly reason over time and space. |
| Outcome: | The proposed models perform well on reasoning tasks involving only temporal knowledge, but performance remains constrained on tasks that require connecting temporal and geographic information. |
Bias in the East, Bias in the West: A Bilingual Analysis of LLM Political Bias on U.S.- and China-Related Issues (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can exhibit political biases, which creates a risk of undue influence on LLM users and public opinion. |
| Approach: | They use a dataset of 36k real-time test prompts to measure LLM political bias on U.S. and Chinese issues. |
| Outcome: | The proposed model origin and prompt language influence bias on 60 political issues. |
Improving Adversarial Data Collection by Supporting Annotators: Lessons from GAHD, a German Hate Speech Dataset (2024.naacl-long)
Copied to clipboard
| Challenge: | Hate speech detection models are only as good as the data they are trained on, but adversarial datasets are slow and costly . data sourced from social media suffer from systematic gaps and biases, leading to unreliable models with simplistic decision boundaries. |
| Approach: | They propose a German Adversarial Hate speech Dataset comprising 11k examples . they explore new strategies for supporting annotators and provide manual analysis of disagreements for each strategy . |
| Outcome: | The proposed dataset is challenging even for state-of-the-art hate speech detection models and it significantly improves model robustness. |
Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Language use differs between domains and even within a domain, language use changes over time. |
| Approach: | They propose to use social media comments to study temporal adaptations in pre-trained language models. |
| Outcome: | The proposed model performs better on past than on future test sets, whereas adapting to domain does not improve performance on the downstream task. |
Data-Efficient Strategies for Expanding Hate Speech Detection into Under-Resourced Languages (2022.emnlp-main)
Copied to clipboard
| Challenge: | Hate speech datasets focus on English-language content, hindering effective models . annotating hateful content is expensive, time-consuming and potentially harmful to annotators. |
| Approach: | They propose to use ISO 639-1 codes to fine-tune models on one source language and apply them to another language. |
| Outcome: | The proposed approach performs well on some tasks, but fails on many others. |
XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are now being used by millions of people across the world. |
| Approach: | They propose a test suite called XSTest to identify such eXaggerated Safety behaviours in a systematic way. |
| Outcome: | The proposed test suite identifies eXaggerated Safety behaviours in a systematic way. |
Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals’ Subjective Text Perceptions (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work has shown that LLMs perform poorly when prompted with sociodemographic attributes, suggesting limited inherent sociodemography knowledge. |
| Approach: | They propose to train large language models to be accurate sociodemographic models of annotator variation by using a curated dataset of five tasks with standardized sociodemography. |
| Outcome: | The proposed models improve in sociodemographic prompting when trained but this performance gain is largely due to models learning annotator-specific behaviour rather than sociodemography. |