Papers by Alexandre Allauzen
Identifying and typifying demographic unfairness in phoneme-level embeddings of self-supervised speech recognition models (2026.findings-acl)
Copied to clipboard
| Challenge: | a number of studies have been done to improve ASR for speaker groups, but there is still room for improvement . authors propose a framework typifying two types of error in phoneme embeddings . |
| Approach: | They propose a framework typifying two types of error that can occur in phoneme modeling . they propose random error/high variance in phonemes embedding vs systematic error/embedding bias . |
| Outcome: | The proposed framework typifies errors in phoneme modeling in ASR systems . it shows that training only on a single, typically disadvantaged SG improves performance . |
FlauBERT: Unsupervised Language Model Pre-training for French (2020.lrec-1)
Copied to clipboard
Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, Didier Schwab
| Challenge: | Language models are a key step to achieve state-of-the-art results in many different Natural Language Processing (NLP) tasks. |
| Approach: | They propose to use a language model that is pre-trained on a large and heterogeneous French corpus to train continuous word representations. |
| Outcome: | The proposed model outperforms existing models on a large and heterogeneous French corpus. |
Learning with Noise-Contrastive Estimation: Easing training by learning to scale (C18-1)
Copied to clipboard
| Challenge: | Neural language models have recently shown great improvement, but they share a common issue: large output vocabulary, computational time, and high dimensional space. |
| Approach: | They propose to make scaling factor a trainable parameter and use noise distribution to initialize output bias. |
| Outcome: | The proposed training strategies yield stable and competitive performances in small and large scale language modelling tasks. |
LOCOST: State-Space Models for Long Document Abstractive Summarization (2024.eacl-long)
Copied to clipboard
Florian Le Bronnec, Song Duong, Mathieu Ravaut, Alexandre Allauzen, Nancy Chen, Vincent Guigue, Alberto Lumbreras, Laure Soulier, Patrick Gallinari
| Challenge: | State-space models are a low-complexity alternative to transformers for text generation . however, the quadratic complexity of the input length restricts the application of large pretrained models to long texts. |
| Approach: | They propose an encoder-decoder architecture based on state-space models for conditional text generation with long context inputs. |
| Outcome: | The proposed model saves memory and memory during training and inference time while saving 50% and 87% of memory. |
Chain and Causal Attention for Efficient Entity Tracking (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to handle entity tracking require at least log2 (n+1) layers to handle n state changes. |
| Approach: | They propose an efficient enhancement to the standard attention mechanism to handle long-term dependencies with a single layer. |
| Outcome: | The proposed model can handle entity tracking with n state changes with a single layer. |
Exploring Precision and Recall to assess the quality and diversity of LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models are limited to specific tasks, but they are now widely available for a wide range of tasks. |
| Approach: | They propose a framework for large language models such as Llama-2 and Mistral that imports precision and recall metrics from image generation to text generation. |
| Outcome: | The proposed framework allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora. |
Measure and Evaluation of Semantic Divergence across Two Languages (2021.acl-long)
Copied to clipboard
| Challenge: | Languages are dynamic systems: word usage may change over time, reflecting various societal factors. |
| Approach: | They propose to build time-varying and bilingual word embeddings using contextualised and non-contextualised embeddables. |
| Outcome: | The proposed methods perform best on newspaper corpora, but are competitive on large corpus. |
Training compute-optimal transformer encoder models (2025.emnlp-main)
Copied to clipboard
| Challenge: | OptiBERT is a family of compute-optimal BERT-style models that matches or surpasses leading baselines while training with dramatically less FLOPS. |
| Approach: | They propose to train OptiBERT models with a Masked Language Model objective . they train a family of compute-optimal BERT-style models that matches or surpasses leading baselines . |
| Outcome: | The proposed model matches or surpasses leading baselines on GLUE and MTEB while training with dramatically less FLOPS. |