Papers by Maha Elbayad
SpiRit-LM: Interleaved Spoken and Written Language Model (2025.tacl-1)
Copied to clipboard
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoît Sagot, Emmanuel Dupoux
| Challenge: | SpiRit-LM is a foundation multimodal language model that freely mixes text and speech. |
| Approach: | They propose a multimodal language model that freely mixes text and speech . they extend the model to the speech modality by continuously training it on text and language units. |
| Outcome: | The proposed model can learn new tasks in a few-shot fashion across modalities. |
Causes and Cures for Interference in Multilingual Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to reduce interference in multilingual machine translation are often computationally intensive and do not always work. |
| Approach: | They propose to reduce interference in multilingual machine translation models by enlarging the model and tuning the sampling temperature to control the proportion of each language pair in the data. |
| Outcome: | The proposed model size, data size, and proportion of each language pair within the dataset determine interference (or synergy) . |
Online Versus Offline NMT Quality: An In-depth Analysis on English-German and German-English (2020.coling-main)
Copied to clipboard
Maha Elbayad, Michael Ustaszewski, Emmanuelle Esperança-Rodier, Francis Brunet-Manquat, Jakob Verbeek, Laurent Besacier
| Challenge: | Existing studies compare offline and online neural machine translation architectures . we examine the impact of online decoding constraints on the translation quality . |
| Approach: | They evaluate offline and online neural machine translation architectures using human evaluations on English-German and German-English language pairs. |
| Outcome: | The proposed models are particularly sensitive to latency constraints and are well-suited for offline translation tasks. |
Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Sparsely gated Mixture of Experts (MoE) models are a compute-efficient method to scale model capacity for multilingual machine translation tasks. |
| Approach: | They propose a regularization strategy that prevents over-fitting of MoE models on low-resource tasks and conditional MoE Routing and curriculum learning methods that prevent over- fitting. |
| Outcome: | The proposed methods improve the performance of MoE models on low-resource tasks without adversely affecting high-res tasks. |
Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have established that Mixture-of-experts models are parameter-inefficient as the improvement in performance diminishes with an increasing number of experts. |
| Approach: | They propose a mix-of-experts model that uses sparse activation to increase the number of parameters while maintaining low computational requirements per token. |
| Outcome: | The proposed models outperform state-of-the-art models on three multilingual machine translation benchmarks with 4, 15, and 94 language pairs. |
Token-level and sequence-level loss smoothing for RNN language models (P18-1)
Copied to clipboard
| Challenge: | Maximum likelihood estimation treats all sentences that do not match the ground truth as equally poor, ignoring the structure of the output space. |
| Approach: | They propose to extend the reward augmented maximum likelihood approach to token-level loss smoothing by using token-based approaches to improve the model's performance. |
| Outcome: | The proposed model improves on image captioning and machine translation tasks and treats all sentences that do not match the ground truth as poor . |
Less Mature is More Adaptable for Sentence-level Language Modeling (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies fine-tune encoders or contrastive learning approaches to learn sentence representations. |
| Approach: | They propose to use sentence-level models to study how sentence representations influence downstream task performance. |
| Outcome: | The proposed models outperform token-level models in terms of time and data efficiency. |
Mixture of Languages: Improved Multilingual Encoders Through Language Grouping (2025.emnlp-main)
Copied to clipboard
João Maria Janeiro, Belen Alastruey, Francisco Massa, Maha Elbayad, Benjamin Piwowarski, Patrick Gallinari, Loic Barrault
| Challenge: | Recent work in this field relies on training transformer encoders on a large amount of multilingual data, with all parameters shared across all languages. |
| Approach: | They propose a mixture of languages strategy to pretrain largely multilingual encoders using masked language modeling. |
| Outcome: | The proposed architecture outperforms a dense counterpart, MoE models and public multilingual encoders on downstream tasks while minimizing interference. |
Efficiently Upgrading Multilingual Machine Translation Models to Support More Languages (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing multilingual machine translation models need to be upgraded as data becomes available in more languages. |
| Approach: | They propose three techniques that speed up the effective learning of new languages and alleviate catastrophic forgetting . |
| Outcome: | The proposed techniques exceed the performance of a same-sized baseline model with 30% computation and recover the performance a larger model trained from scratch with over 50% reduction in computation. |