Papers by Pierre Andrews
BOUQuET : dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation (2025.emnlp-main)
Copied to clipboard
Pierre Andrews, Mikel Artetxe, Mariano Coria Meglioli, Marta R. Costa-jussà, Joe Chuang, David Dale, Mark Duppenthaler, Nathanial Paul Ekberg, Cynthia Gao, Daniel Edward Licht, Jean Maillard, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Eduardo Sánchez, Ioannis Tsiamas, Arina Turkatenko, Albert Ventayol-Boada, Shireen Yates
| Challenge: | BOUQUET is a multi-way, multicentric and multi-register/domain dataset and benchmark . the dataset is handcrafted in 8 non-English languages . |
| Approach: | They propose to use BOUQuET to collect a multi-way, multicentric and multi-register/domain dataset and benchmark in 8 non-English languages. |
| Outcome: | The proposed dataset is available at https://huggingface.co/datasets/facebook/bouquet. |
Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scale (2023.emnlp-main)
Copied to clipboard
Marta Costa-jussà, Pierre Andrews, Eric Smith, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Daniel Licht, Carleigh Wood
| Challenge: | Multilingual HolisticBias dataset includes 20,459 sentences in 50 languages . dataset is intended to uncover demographic imbalances and quantify mitigations . |
| Approach: | They propose a multilingual extension of the HolisticBias dataset . they use 118 demographic descriptors and three patterns to build multilingual sentences . |
| Outcome: | The proposed model improves translation quality when the source input only differs in gender . it also improves when the masculine human reference is used in the model . |
LCFO: Long Context and Long Form Output Dataset and Benchmarking (2025.findings-acl)
Copied to clipboard
Marta R. Costa-jussà, Pierre Andrews, Mariano Coria Meglioli, Joy Chen, Joe Chuang, David Dale, Christophe Ropers, Alexandre Mourachko, Eduardo Sánchez, Holger Schwenk, Tuan A. Tran, Arina Turkatenko, Carleigh Wood
| Challenge: | Using long text outputs to evaluate progress in summarization and summary expansion tasks is challenging. |
| Approach: | They propose a framework for assessing gradual summarization and summary expansion capabilities across diverse domains. |
| Outcome: | The proposed framework provides alignments between specific QA pairs and corresponding summaries in 7 domains. |
MuTox: Universal MUltilingual Audio-based TOXicity Dataset and Zero-shot Detector (2024.findings-acl)
Copied to clipboard
Marta Costa-jussà, Mariano Meglioli, Pierre Andrews, David Dale, Prangthip Hansanti, Elahe Kalbassi, Alexandre Mourachko, Christophe Ropers, Carleigh Wood
| Challenge: | Existing studies on text-based toxicity detection for other languages are limited, especially for languages other than English. |
| Approach: | They propose a multilingual audio-based toxicity classifier which covers 14 different linguistic families and a dataset of 20,000 audio utterances for English and Spanish. |
| Outcome: | The new classifier improves F1-Score by an average of 100% when compared to existing wordlist-based classifiers. |
stopes - Modular Machine Translation Pipelines (2022.emnlp-demos)
Copied to clipboard
Pierre Andrews, Guillaume Wenzek, Kevin Heffernan, Onur Çelebi, Anna Sun, Ammar Kamran, Yingzhe Guo, Alexandre Mourachko, Holger Schwenk, Angela Fan
| Challenge: | Neural machine translation is a natural language deep learning application that needs data to be trained. |
| Approach: | They describe a framework that empowers scalability and versatility for research use cases. |
| Outcome: | The proposed framework empowers scalability and versatility for research use cases. |
BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric (2023.acl-long)
Copied to clipboard
Mingda Chen, Paul-Ambroise Duquenne, Pierre Andrews, Justine Kao, Alexandre Mourachko, Holger Schwenk, Marta R. Costa-jussà
| Challenge: | End-to-End speech-to speech translation is generally evaluated with text-based metrics . this means generated speech has to be automatically transcribed, making the evaluation dependent on ASR systems. |
| Approach: | They propose a text-free evaluation metric for end-to-end speech-tospeech translation, named BLASER, to avoid the dependency on automatic speech recognition systems. |
| Outcome: | The proposed metric avoids the dependency on automatic speech recognition systems by encoding generated speech segments into a shared embedding space. |
2M-BELEBELE: Highly Multilingual Speech and American Sign Language Comprehension Dataset Download PDF (2025.findings-acl)
Copied to clipboard
Marta R. Costa-jussà, Bokai Yu, Pierre Andrews, Belen Alastruey, Necati Cihan Camgoz, Joe Chuang, Jean Maillard, Christophe Ropers, Arina Turkatenko, Carleigh Wood
| Challenge: | We extend the BELEBELE dataset to speech and sign, and extend the Automatic Speech Recognition Benchmark, FLEURS, by 20%. |
| Approach: | They extend the BELEBELE and FLEURS speech comprehension datasets to speech and sign . they evaluate the datasets for 5-shot and zero-shot settings and find that the accuracy is 10% lower than reading comprehension. |
| Outcome: | The proposed dataset covers 91 spoken languages and one sign language (ASL) it also extends the Automatic Speech Recognition Benchmark, FLEURS, by 20% across languages. |