Papers by Beyza Ermis
The State of Multilingual LLM Safety Research: From Measuring The Language Gap To Mitigating It (2025.emnlp-main)
Copied to clipboard
| Challenge: | a systematic review of 300 publications reveals a language gap in LLM safety research . even high-resource non-English languages receive little attention, authors note . |
| Approach: | They propose to focus on safety evaluation, training data generation, and crosslingual safety generalization based on their findings. |
| Outcome: | The authors suggest that the field can develop more robust, inclusive safety practices for diverse global populations. |
Multilingual Arbitration: Optimizing Data Pools to Accelerate Multilingual Progress (2025.acl-long)
Copied to clipboard
| Challenge: | Synthetic data generation relies on a single oracle teacher model, which can lead to model collapse and bias propagation. |
| Approach: | They propose a multilingual arbitration approach that exploits performance variations among multiple models for each language. |
| Outcome: | The proposed approach surpasses single-teacher distillation with 80% win rates over proprietary and open-weight models with the largest improvements in low-resource languages. |
Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selection (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to improve robustness require changing the fine-tuning process or large-scale data augmentation, which are infeasible or cost prohibitive for closed-source models. |
| Approach: | They propose to prioritize more complex examples or replace existing training examples with LLM-generated data to improve performance on OOD NLI datasets. |
| Outcome: | The proposed methods improve performance on difficult OOD datasets while training with synthetic data leads to substantial improvements on easier OOD data. |
On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research (2023.emnlp-main)
Copied to clipboard
| Challenge: | Perception of toxicity evolves over time and differs between geographies and cultural backgrounds. |
| Approach: | They propose to use a more structured approach to evaluating toxicity over time . they suggest that research that relied on automatic toxicity scores may have resulted in inaccurate results. |
| Outcome: | The Perspective API has been updated to reflect the changes in toxicity scores. |
The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm (2024.emnlp-main)
Copied to clipboard
null Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, Sara Hooker
| Challenge: | Existing approaches to safety alignment focus on homogeneous monolingual settings . preference training and safety measures often overfit to harms common in Western-centric datasets . |
| Approach: | They propose to use human annotated red teaming prompts to identify global and local harms. |
| Outcome: | The proposed approach can address and optimize for a non-homogeneous set of languages and cultural preferences while minimizing both global and local harms. |
Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve? (2024.emnlp-main)
Copied to clipboard
| Challenge: | In the last decade, the generalization and adaptation abilities of deep learning models were evaluated on fixed training and test distributions. |
| Approach: | They propose to train large language models on unlabeled text corpora and train them online. |
| Outcome: | The proposed model training on a text domain could degrade its perplexity on the test portion of the same domain. |
Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for mitigating toxicity require drastic modifications to model parameters or the use of computationally intensive auxiliary models. |
| Approach: | They propose a method that matches current state-of-the-art toxicity mitigation while achieving 43% relative latency reduction during inference. |
| Outcome: | The proposed method achieves 43% latency reduction during inference and is more computationally efficient. |
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation (2025.acl-long)
Copied to clipboard
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, Sara Hooker
| Challenge: | Reliable multilingual evaluation is difficult and culturally appropriate evaluation is even harder to achieve. |
| Approach: | They propose a multilingual evaluation framework that aims to mitigate these biases by improving translations and annotation practices. |
| Outcome: | The proposed framework improves translation quality and cultural coverage and is culturally sensitive and culturally agnostic. |
From One to Many: Expanding the Scope of Toxicity Mitigation in Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | toxicity mitigation in language models has been focused on single-language settings . however, widespread adoption of LLMs has introduced a range of unknown -harms . |
| Approach: | They employ translated data to evaluate and enhance mitigation techniques in the absence of sufficient annotated datasets across languages. |
| Outcome: | The proposed approach compares translation quality and retrieval-augmented mitigation techniques under static and continual toxicity mitigation scenarios. |