Papers by Sara Hooker
Multilingual Arbitration: Optimizing Data Pools to Accelerate Multilingual Progress (2025.acl-long)
Copied to clipboard
| Challenge: | Synthetic data generation relies on a single oracle teacher model, which can lead to model collapse and bias propagation. |
| Approach: | They propose a multilingual arbitration approach that exploits performance variations among multiple models for each language. |
| Outcome: | The proposed approach surpasses single-teacher distillation with 80% win rates over proprietary and open-weight models with the largest improvements in low-resource languages. |
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Extending state-of-the-art language models to low-resource languages requires addressing what we call the low-Resource double bind. |
| Approach: | They propose a low-resource double bind to refer to the co-occurrence of data limitations and compute resource constraints. |
| Outcome: | The proposed model improves performance on frequent sentences but disparates on infrequent ones. |
Locally Differentially Private Document Generation Using Zero Shot Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing privacy protections are limited by pretrained large language models . a new mechanism that leverages the power of large language model de-anonymization is proposed . |
| Approach: | They propose a locally differentially private mechanism that leverages pretrained large language models to counter author de-anonymization attacks by minimizing the impact on downstream utility. |
| Outcome: | The proposed mechanism reduces the success rate of de-anonymization attacks against IMDB datasets while minimizing the impact on downstream utility. |
RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Preference optimization is a widely adopted post-training technique to align large language models with human preferences. |
| Approach: | They propose a method for generating multilingual feedback data to balance data coverage. |
| Outcome: | The proposed method achieves 54.4% win-rate against current state-of-the-art multilingual LLM in its parameter class and 69.5% win- rate or higher against widely used models like Gemma, Mistral and Llama 3. |
Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs (2024.acl-long)
Copied to clipboard
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker
| Challenge: | Proximal Policy Optimization (PPO) is used for RLHF but requires high computational cost and sensitive hyperparameter tuning. |
| Approach: | They propose to use Proximal Policy Optimization to align large language models to human preferences. |
| Outcome: | The proposed method preserves and even increases performance while preserving the motivational principles that led to the development of PPO. |
On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research (2023.emnlp-main)
Copied to clipboard
| Challenge: | Perception of toxicity evolves over time and differs between geographies and cultural backgrounds. |
| Approach: | They propose to use a more structured approach to evaluating toxicity over time . they suggest that research that relied on automatic toxicity scores may have resulted in inaccurate results. |
| Outcome: | The Perspective API has been updated to reflect the changes in toxicity scores. |
Efficient Methods for Natural Language Processing: A Survey (2023.tacl-1)
Copied to clipboard
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
| Challenge: | Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data, but using only scale to improve performance means resource consumption also grows. |
| Approach: | They propose to use data, time, storage, or energy to improve model performance. |
| Outcome: | The proposed methods and findings provide guidance for conducting NLP under limited resources and point towards promising research directions for developing more efficient methods. |
The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm (2024.emnlp-main)
Copied to clipboard
null Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, Sara Hooker
| Challenge: | Existing approaches to safety alignment focus on homogeneous monolingual settings . preference training and safety measures often overfit to harms common in Western-centric datasets . |
| Approach: | They propose to use human annotated red teaming prompts to identify global and local harms. |
| Outcome: | The proposed approach can address and optimize for a non-homogeneous set of languages and cultural preferences while minimizing both global and local harms. |
Nexus: Adaptive Upcycling to Efficiently Pretrain Mixture of Experts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Nevertheless, training from scratch on trillions of tokens remains expensive that most users can only finetune these models. |
| Approach: | They propose to reuse parameters of dense models for the MoE layers with a router that can integrate new experts into existing trained models without hurting performance on previous domains. |
| Outcome: | The proposed router can integrate new experts into existing trained models without hurting the performance on previous domains. |
How Does Quantization Affect Multilingual LLMs? (2024.findings-emnlp)
Copied to clipboard
Kelly Marchisio, Saurabh Dash, Hongyu Chen, Dennis Aumiller, Ahmet Üstün, Sara Hooker, Sebastian Ruder
| Challenge: | Quantization is widely used to improve inference speed and deployment of large language models. |
| Approach: | They conduct a thorough analysis of quantized multilingual LLMs . they find language disparately affected by quantization, non-Latin script languages worst . authors urge consideration of multilingual performance as evaluation criterion for efficient models . |
| Outcome: | The results show that quantization has harmful effects on human evaluation . language performance is disparately affected by quantization, the authors say . |
LLM See, LLM Do: Leveraging Active Inheritance to Target Non-Differentiable Objectives (2024.emnlp-main)
Copied to clipboard
| Challenge: | Historically, high-quality labeled data has been costly to curate due to scarcity of available data and financial cost. |
| Approach: | They characterize the impact of passive inheritance of model properties by studying how the source of synthetic data shapes models’ internal biases, calibration and preferences, and their generations’ textual attributes. |
| Outcome: | The proposed model inheritance can increase lexical diversity or reduce toxicity. |
Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning (2024.acl-long)
Copied to clipboard
Shivalika Singh, Freddie Vargus, Daniel D’souza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O’Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Chien, Sebastian Ruder, Surya Guthikonda, Emad Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, Sara Hooker
| Challenge: | Existing datasets in the English language are mostly in the realm of instruction fine-tuning . aya dataset, the Aya Collection, and the AYa Evaluation Suite are key resources . |
| Approach: | They aim to build a human-curated instruction-following dataset spanning 65 languages . they work with fluent speakers of languages from around the world to collect natural instances of instructions and completions . |
| Outcome: | The goal is to build a human-curated instruction-following dataset spanning 65 languages. |
Intriguing Properties of Compression on Multilingual Models (2022.emnlp-main)
Copied to clipboard
Kelechi Ogueji, Orevaoghene Ahia, Gbemileke Onilude, Sebastian Gehrmann, Sara Hooker, Julia Kreutzer
| Challenge: | Multilingual models are dependent on scaling to generalize to a growing number of languages . compression techniques can have disparate effects on model performance for low-resource languages if used sparsely . |
| Approach: | They propose to characterize the impact of sparsifying multilingual pre-trained language models during fine-tuning. |
| Outcome: | The proposed framework characterizes the impact of sparsifying multilingual pre-trained language models during fine-tuning. |
Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for mitigating toxicity require drastic modifications to model parameters or the use of computationally intensive auxiliary models. |
| Approach: | They propose a method that matches current state-of-the-art toxicity mitigation while achieving 43% relative latency reduction during inference. |
| Outcome: | The proposed method achieves 43% latency reduction during inference and is more computationally efficient. |
IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models (2025.naacl-long)
Copied to clipboard
David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba Oluwadara Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Ijeoma Chukwuneke, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson Koffi Kalipe, Jonathan Mukiibi, Salomon Kabongo Kabenamualu, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuya, Salomey Osei, Shamsuddeen Hassan Muhammad, Sokhar Samb, Tadesse Kebede Guge, Tombekai Vangoni Sherman, Pontus Stenetorp
| Challenge: | Large language models (LLMs) are limited to a few high-resource languages . many low-resourced languages are evaluated only on basic text classification tasks . |
| Approach: | They propose to use IrokoBench to evaluate 17 low-resource African languages . they use human-translated benchmark datasets to evaluate zero-shot, few-shot and translate-test settings . |
| Outcome: | The proposed model performs well in English and French, but the highest performing model perform poorly in proprietary models. |
One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers (2026.acl-long)
Copied to clipboard
Diana Abagyan, Alejandro R. Salamanca, Andres Felipe Cruz-Salinas, Kris Cao, Hangyu Lin, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, Sara Hooker
| Challenge: | Existing approaches to train multilingual large language models for many languages at once are limited due to limited model capacity, scarce high-quality data, and compute constraints. |
| Approach: | They propose to use a universal tokenizer to improve language plasticity and adaptability to new languages by up to 20%. |
| Outcome: | The proposed tokenizer improves language plasticity and improves plasticity towards languages that are completely unseen in the tokenizer and pretraining, by up to 5% win rate gain. |
M-RewardBench: Evaluating Reward Models in Multilingual Settings (2025.acl-long)
Copied to clipboard
Srishti Gureja, Lester James Validad Miranda, Shayekh Bin Islam, Rishabh Maheshwary, Drishti Sharma, Gusti Triandi Winata, Nathan Lambert, Sebastian Ruder, Sara Hooker, Marzieh Fadaee
| Challenge: | Reward models (RMs) are primarily trained and evaluated in English and their capabilities in multilingual settings remain understudied. |
| Approach: | They construct a multilingual RM evaluation benchmark that tests the chat, safety, reasoning, and translation capabilities of RMs in 23 languages. |
| Outcome: | The proposed model performs better for high-resource languages and improves with translation quality. |
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation (2025.acl-long)
Copied to clipboard
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, Sara Hooker
| Challenge: | Reliable multilingual evaluation is difficult and culturally appropriate evaluation is even harder to achieve. |
| Approach: | They propose a multilingual evaluation framework that aims to mitigate these biases by improving translations and annotation practices. |
| Outcome: | The proposed framework improves translation quality and cultural coverage and is culturally sensitive and culturally agnostic. |
From One to Many: Expanding the Scope of Toxicity Mitigation in Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | toxicity mitigation in language models has been focused on single-language settings . however, widespread adoption of LLMs has introduced a range of unknown -harms . |
| Approach: | They employ translated data to evaluate and enhance mitigation techniques in the absence of sufficient annotated datasets across languages. |
| Outcome: | The proposed approach compares translation quality and retrieval-augmented mitigation techniques under static and continual toxicity mitigation scenarios. |
Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning (2024.findings-acl)
Copied to clipboard
| Challenge: | Neural Machine Translation models are extremely data-hungry and require a large dataset to maintain data quality. |
| Approach: | They propose a new data pruning technique that leverages early model training dynamics to identify the most relevant data points for model performance. |
| Outcome: | The proposed technique outperforms the benchmarks on indo-European languages while pruning up to 50% of training data. |
When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have shifted focus toward scaling inference-time compute. |
| Approach: | They propose to scale inference-time compute in a multilingual, multi-task setting . they propose to use m-ArenaHard-v2.0 prompts to sample multiple outputs in parallel . |
| Outcome: | The proposed solutions achieve an average +6.8 jump in win-rates for 8B models on m-ArenaHard-v2.0 prompts in non-English languages against proprietary models like Gemini. |