Papers by Julia Kreutzer
KINNEWS and KIRNEWS: Benchmarking Cross-Lingual Text Classification for Kinyarwanda and Kirundi (2020.coling-main)
Copied to clipboard
| Challenge: | low-resource African languages are traditionally left behind because of the lack of well-annotated data and effective preprocessing. |
| Approach: | They propose two news datasets for multi-class classification of news articles in two low-resource African languages. |
| Outcome: | The proposed datasets show that training embeddings on the higher-resourced Kinyarwanda yields successful cross-lingual transfer to Kirundi. |
Current Advances in LLM Reasoning (2026.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial examines comprehensive evaluation strategies to assess the reasoning abilities of large language models (LLMs) advanced inference time methods and post-training methods that aim to make LLMs think more like humans are discussed in this tutorial. |
| Approach: | This tutorial explores comprehensive evaluation strategies to assess the reasoning abilities of large language models (LLMs) and discusses two types of methods to improve models’ reasoning: advanced inference time methods, structured and self-improvement inference methods, and post-training methods, such as RLHF, DPO, and GRPO. |
| Outcome: | This tutorial examines evaluation strategies to assess the reasoning abilities of large language models and discusses two types of methods to improve models’ reasoning. |
The State of Multilingual LLM Safety Research: From Measuring The Language Gap To Mitigating It (2025.emnlp-main)
Copied to clipboard
| Challenge: | a systematic review of 300 publications reveals a language gap in LLM safety research . even high-resource non-English languages receive little attention, authors note . |
| Approach: | They propose to focus on safety evaluation, training data generation, and crosslingual safety generalization based on their findings. |
| Outcome: | The authors suggest that the field can develop more robust, inclusive safety practices for diverse global populations. |
Bandits Don’t Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Training data for machine translation (MT) is often sourced from multiple large corpora that are multi-faceted in nature. |
| Approach: | They propose to optimize the balance between translationese and natural training data to relieve system developers from manual schedule design. |
| Outcome: | The proposed model relieves system developers from manual schedule design. |
MasakhaNER: Named Entity Recognition for African Languages (2021.tacl-1)
Copied to clipboard
David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, Salomey Osei
| Challenge: | (2020) African languages are underrepresented in existing natural language processing datasets, research, and tools due to lack of datasets and reproducible results. |
| Approach: | They propose to create a dataset for named entity recognition (NER) in ten African languages. |
| Outcome: | The results of the first large dataset for named entity recognition (NER) in ten African languages are released to inform future research on African NLP. |
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Extending state-of-the-art language models to low-resource languages requires addressing what we call the low-Resource double bind. |
| Approach: | They propose a low-resource double bind to refer to the co-occurrence of data limitations and compute resource constraints. |
| Outcome: | The proposed model improves performance on frequent sentences but disparates on infrequent ones. |
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2022.tacl-1)
Copied to clipboard
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, Mofetoluwa Adeyemi
| Challenge: | Lower-resource corpora have systematic issues, including mislabeled or nonstandard/ambiguous language codes. |
| Approach: | They manually audit the quality of 205 language-specific corpora released with five major public datasets. |
| Outcome: | The results show that lower-resource corpora have systematic issues even for non-proficient speakers. |
Self-Regulated Interactive Sequence-to-Sequence Learning (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to learning from different types of feedback have not been explored. |
| Approach: | They propose a machine learning algorithm that uses self-regulation to balance cost and effect of different types of feedback. |
| Outcome: | The proposed model is robust under domain shift and is a promising alternative to active learning. |
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (P18-1)
Copied to clipboard
| Challenge: | Recent work has shown that reinforcement learning (RL) can be scaled to games with large state-action spaces, achieving human-level performance or even superhuman performance. |
| Approach: | They propose to use bandit feedback to improve sequence-to-sequence learning by simulating reward signals by evaluation metrics such as BLEU, F1-score, or ROUGE. |
| Outcome: | The proposed methods improve performance even from small amounts of human feedback, pointing to a great potential for applications at larger scale. |
RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Preference optimization is a widely adopted post-training technique to align large language models with human preferences. |
| Approach: | They propose a method for generating multilingual feedback data to balance data coverage. |
| Outcome: | The proposed method achieves 54.4% win-rate against current state-of-the-art multilingual LLM in its parameter class and 69.5% win- rate or higher against widely used models like Gemma, Mistral and Llama 3. |
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020.findings-emnlp)
Copied to clipboard
Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa Berhe, Mofetoluwa Adeyemi, Masabata Mokgesi-Selinga, Lawrence Okegbemi, Laura Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji, Kathleen Siminyu, Julia Kreutzer, Jason Webster, Jamiil Toure Ali, Jade Abbott, Iroro Orife, Ignatius Ezeani, Idris Abdulkadir Dangana, Herman Kamper, Hady Elsahar, Goodness Duru, Ghollah Kioko, Murhabazi Espoir, Elan van Biljon, Daniel Whitenack, Christopher Onyefuluchi, Chris Chinenye Emezue, Bonaventure F. P. Dossou, Blessing Sibanda, Blessing Bassey, Ayodele Olabiyi, Arshath Ramkilowan, Alp Öktem, Adewale Akinfaderin, Abdallah Bashir
| Challenge: | 'Low-resourced'-ness is a complex problem that goes beyond data availability and reflects systemic problems in society. |
| Approach: | They propose to use machine translation to scale to low-resourced languages by using a dataset and a benchmarking system to measure their resource use. |
| Outcome: | The proposed approach allows participants without formal training to make a unique scientific contribution. |
Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs (2024.acl-long)
Copied to clipboard
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker
| Challenge: | Proximal Policy Optimization (PPO) is used for RLHF but requires high computational cost and sensitive hyperparameter tuning. |
| Approach: | They propose to use Proximal Policy Optimization to align large language models to human preferences. |
| Outcome: | The proposed method preserves and even increases performance while preserving the motivational principles that led to the development of PPO. |
The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm (2024.emnlp-main)
Copied to clipboard
null Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, Sara Hooker
| Challenge: | Existing approaches to safety alignment focus on homogeneous monolingual settings . preference training and safety measures often overfit to harms common in Western-centric datasets . |
| Approach: | They propose to use human annotated red teaming prompts to identify global and local harms. |
| Outcome: | The proposed approach can address and optimize for a non-homogeneous set of languages and cultural preferences while minimizing both global and local harms. |
Inference Strategies for Machine Translation with Conditional Masking (2020.emnlp-main)
Copied to clipboard
| Challenge: | Conditional masked language model training has proven successful for non-autoregressive and semi-auto-regressively sequence generation tasks. |
| Approach: | They propose a conditional masked language model (CMLM) that is a factorization of conditional probabilities of partial sequences and propose heuristics to improve performance. |
| Outcome: | The proposed algorithm is more efficient than the standard “mask-predict” algorithm on machine translation tasks. |
Connecting Language Technologies with Rich, Diverse Data Sources Covering Thousands of Languages (2024.lrec-main)
Copied to clipboard
Daan van Esch, Sandy Ritchie, Sebastian Ruder, Julia Kreutzer, Clara Rivera, Ishank Saxena, Isaac Caswell
| Challenge: | Existing data sources for many thousands of languages are rich and diverse . Efforts are ongoing to extend technology to many more of the world's languages . |
| Approach: | They provide an overview of some of the major online data sources available for thousands of languages. |
| Outcome: | The proposed language technologies are based on the data available for thousands of languages. |
LLM See, LLM Do: Leveraging Active Inheritance to Target Non-Differentiable Objectives (2024.emnlp-main)
Copied to clipboard
| Challenge: | Historically, high-quality labeled data has been costly to curate due to scarcity of available data and financial cost. |
| Approach: | They characterize the impact of passive inheritance of model properties by studying how the source of synthetic data shapes models’ internal biases, calibration and preferences, and their generations’ textual attributes. |
| Outcome: | The proposed model inheritance can increase lexical diversity or reduce toxicity. |
Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning (2024.acl-long)
Copied to clipboard
Shivalika Singh, Freddie Vargus, Daniel D’souza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O’Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Chien, Sebastian Ruder, Surya Guthikonda, Emad Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, Sara Hooker
| Challenge: | Existing datasets in the English language are mostly in the realm of instruction fine-tuning . aya dataset, the Aya Collection, and the AYa Evaluation Suite are key resources . |
| Approach: | They aim to build a human-curated instruction-following dataset spanning 65 languages . they work with fluent speakers of languages from around the world to collect natural instances of instructions and completions . |
| Outcome: | The goal is to build a human-curated instruction-following dataset spanning 65 languages. |
Can Neural Machine Translation be Improved with User Feedback? (N18-3)
Copied to clipboard
| Challenge: | a recent study has focused on the use of explicit and implicit feedback for neural machine translation (NMT) a new study uses explicit and implied feedback to improve performance of NMT with human reinforcement. |
| Approach: | They propose to use real logged feedback to improve neural machine translation with human reinforcement. |
| Outcome: | The proposed method improves translation quality metrics with implicit task-based feedback . the proposed method is based on explicit and implicit feedback collected on the eBay platform . |
Intriguing Properties of Compression on Multilingual Models (2022.emnlp-main)
Copied to clipboard
Kelechi Ogueji, Orevaoghene Ahia, Gbemileke Onilude, Sebastian Gehrmann, Sara Hooker, Julia Kreutzer
| Challenge: | Multilingual models are dependent on scaling to generalize to a growing number of languages . compression techniques can have disparate effects on model performance for low-resource languages if used sparsely . |
| Approach: | They propose to characterize the impact of sparsifying multilingual pre-trained language models during fine-tuning. |
| Outcome: | The proposed framework characterizes the impact of sparsifying multilingual pre-trained language models during fine-tuning. |
Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | In neural sequence-to-sequence learning, Reinforcement Learning (RL) has gained popularity due to the suitability of Policy Gradient (PG) methods for the end-to end training paradigm. |
| Approach: | They propose to let the model explore the output space beyond the reference output that is used for standard cross-entropy minimization by reinforcing model outputs according to their quality, effectively increasing the likelihood of higher-quality samples. |
| Outcome: | The proposed model explores the output space beyond the reference output that is used for cross-entropy minimization, increasing the likelihood of higher-quality samples. |
Joey NMT: A Minimalist NMT Toolkit for Novices (D19-3)
Copied to clipboard
| Challenge: | a recent study shows that novices perform better than experts in a code quiz. |
| Approach: | They present a minimalist neural machine translation toolkit based on PyTorch . they evaluate the accessibility of the toolkit in a user study . |
| Outcome: | The proposed toolkit performs comparable to more complex toolkits on standard benchmarks. |
JoeyS2T: Minimalistic Speech-to-Text Modeling with JoeyNMT (2022.emnlp-demos)
Copied to clipboard
| Challenge: | JoeyS2T is a simple, lightweight NMT extension for speech-to-text tasks such as automatic speech recognition and end-to end speech translation. |
| Approach: | JoeyS2T is a JoeyNMT extension for automatic speech recognition and end-to-end speech translation. |
| Outcome: | JoeyS2T performs competitively on English speech recognition and English-to-German speech translation benchmarks. |
A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation (2022.naacl-main)
Copied to clipboard
David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsuddeen Muhammad, Guyo Jarso, Oreen Yousuf, Andre Niyongabo Rubungo, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Benjamin Ajibade, Tunde Ajayi, Yvonne Gitau, Jade Abbott, Mohamed Ahmed, Millicent Ochieng, Anuoluwapo Aremu, Perez Ogayo, Jonathan Mukiibi, Fatoumata Ouoba Kabore, Godson Kalipe, Derguene Mbaye, Allahsera Auguste Tapo, Victoire Memdjokam Koagne, Edwin Munkoh-Buabeng, Valencia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzaaba, Blessing Sibanda, Andiswa Bukula, Sam Manthalu
| Challenge: | Low-resource languages are left out of large-scale pretraining datasets . authors explore how to leverage existing pre-trained models to create low-resourced translation systems for 16 African languages. |
| Approach: | They investigate how large-scale pre-trained models can be used to create low-resource translation systems for 16 African languages. |
| Outcome: | The proposed models can translate between hundreds of languages even though there is little parallel data available for training. |
Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning (2024.findings-acl)
Copied to clipboard
| Challenge: | Neural Machine Translation models are extremely data-hungry and require a large dataset to maintain data quality. |
| Approach: | They propose a new data pruning technique that leverages early model training dynamics to identify the most relevant data points for model performance. |
| Outcome: | The proposed technique outperforms the benchmarks on indo-European languages while pruning up to 50% of training data. |
When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have shifted focus toward scaling inference-time compute. |
| Approach: | They propose to scale inference-time compute in a multilingual, multi-task setting . they propose to use m-ArenaHard-v2.0 prompts to sample multiple outputs in parallel . |
| Outcome: | The proposed solutions achieve an average +6.8 jump in win-rates for 8B models on m-ArenaHard-v2.0 prompts in non-English languages against proprietary models like Gemini. |