Papers by Andre Martins
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (2025.acl-short)
Copied to clipboard
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni
| Challenge: | Existing evaluations of NLP models with LLMs are based on human judgments . however, there are concerns about their validity and reproducibility in proprietary models . |
| Approach: | They evaluate 11 current LLMs for their ability to replicate annotations. they show substantial variance across models and datasets. |
| Outcome: | The proposed model can replicate human annotations on 20 NLP datasets and show substantial variance across models and datasets. |
∞-former: Infinite Memory Transformer (2022.acl-long)
Copied to clipboard
| Challenge: | Several efficient transformers have been proposed, but they all have a finite memory capacity and are forced to drop old information. |
| Approach: | They propose an unbounded long-term memory extension that extends the vanilla transformer by using a continuous-space attention mechanism to attend over the long-time memory. |
| Outcome: | The proposed model can model arbitrarily long contexts while keeping the computation budget fixed. |
Different Speech Translation Models Encode and Translate Speaker Gender Differently (2025.acl-short)
Copied to clipboard
| Challenge: | Recent studies on interpreting the hidden states of speech models have shown their ability to capture speaker-specific features, including gender. |
| Approach: | They propose to use probing methods to assess gender encoding across ST models. |
| Outcome: | The proposed models capture speaker-specific features, including gender, while older models do not . low gender encoding capabilities result in systems’ tendency toward a masculine default, a translation bias that is more pronounced in newer architectures. |
Translate Smart, not Hard: Cascaded Translation Systems with Quality-Aware Deferral (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing quality estimation metrics are used to design effective deferral rules for machine translation. |
| Approach: | They propose a simple yet effective approach for machine translation using existing quality estimation metrics as deferral rules. |
| Outcome: | The proposed approach outperforms existing models in large translation tasks while reducing computational costs. |
TOWER+: Bridging Generality and Translation Specialization in Multilingual LLMs (2026.acl-long)
Copied to clipboard
Ricardo Rei, Nuno M Guerreiro, José Pombal, João Alves, Amin Farajian, Pedro Teixeirinha, Andre Martins
| Challenge: | Large Language Models (LLMs) are emerging as the de facto solution for multilingual machine translation. |
| Approach: | They propose a suite of LLMs that can be fine-tuned to deliver strong performance on translation and multilingual general-purpose text capabilities. |
| Outcome: | The proposed models outperform existing models on translation and general-purpose tasks. |
Did Translation Models Get More Robust Without Anyone Even Noticing? (2025.acl-long)
Copied to clipboard
| Challenge: | Neural machine translation models are highly sensitive to “noisy” inputs, such as spelling errors, abbreviations, and formatting issues. |
| Approach: | They revisit this insight in light of recent multilingual MT models and large language models applied to machine translation. |
| Outcome: | The proposed models perform better on clean data than previous models, but none of the open models use robustness techniques. |
mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent multilingual pretrained language models encode strong language-specific signals, which are not explicitly provided during pretraining. |
| Approach: | They propose a language similarity measure that induces similarities across languages from mPLMs using multi-parallel corpora. |
| Outcome: | The proposed measure exhibits moderately high correlations with linguistic similarity measures, and more accurate similarity results on low correlation languages. |
XAMPLER: Learning to Retrieve Cross-Lingual In-Context Examples (2025.findings-naacl)
Copied to clipboard
| Challenge: | XAMPLER: Cross-Lingual Example Retrieval is a cross-lingual example retrieval method . large language models (LLMs) have emerged as effective in-context learning methods . |
| Approach: | They propose a method to train a multilingual model with annotated English examples . they use annotized English data to train the model and use it to train other languages . |
| Outcome: | XAMPLER: Cross-Lingual Example Retrieval improves in-context learning in English . it trains a retriever based on a multilingual small language model using annotated English examples . |
From Tower to Spire: Adding the Speech Modality to a Translation-Specialist LLM (2025.findings-emnlp)
Copied to clipboard
Kshitij Ambilduke, Ben Peters, Sonal Sannigrahi, Anil Keshwani, Tsz Kin Lam, Bruno Martins, Andre Martins, Marcely Zanon Boito
| Challenge: | Spire is a speech-augmented language model capable of translating speech input into 10 languages and transcribing text input in both directions. |
| Approach: | They introduce a speech-augmented language model capable of translating speech input into 10 languages . they integrate the model into existing multilingual LMs via speech discretization . |
| Outcome: | Spire integrates speech-augmented language model into existing multilingual model using speech discretization and pre-training using only 42.5 K hours of speech. |
Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimation (2025.acl-long)
Copied to clipboard
| Challenge: | Qualitative estimation (QE) metrics have been optimized to align with human quality judgments, but whether they encode social biases has been largely overlooked. |
| Approach: | They define and investigate gender bias of QE metrics and discuss its downstream implications for machine translation (MT) when a human entity’s gender in the source is undisclosed, masculine-inflected translations score higher than feminine-infflectes translations are penalized. |
| Outcome: | The proposed measures are based on gender-based quality estimation metrics across multiple domains, datasets, and languages. |
xTower: A Multilingual LLM for Explaining and Correcting Translation Errors (2024.findings-emnlp)
Copied to clipboard
Marcos Treviso, Nuno Guerreiro, Sweta Agrawal, Ricardo Rei, José Pombal, Tania Vaz, Helena Wu, Beatriz Silva, Daan Stigt, Andre Martins
| Challenge: | Neural machine translation systems produce translations with errors and anomalies . understanding these errors can help improve the translation quality and user experience . |
| Approach: | They propose an open large language model (LLM) built on top of TowerBase to provide free-text explanations for translation errors in order to guide the generation of a corrected translation. |
| Outcome: | The proposed model improves translation quality and user experience by allowing translators to provide free-text explanations for errors and anomalies. |
Can Automatic Metrics Assess High-Quality Translations? (2024.emnlp-main)
Copied to clipboard
| Challenge: | a recent human evaluation study found that translations produced by current MT systems achieve very high-quality scores when judged by humans on a direct assessment scale of 0 to 100. |
| Approach: | They stress-test the ability of current translation quality metrics to detect correct translations . they show that current metrics often over or underestimate translation quality . |
| Outcome: | The proposed method overestimates translation quality, the authors show . they show that current metrics often overestimate translation quality . |
Quality-Aware Decoding for Neural Machine Translation (2022.naacl-main)
Copied to clipboard
Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig, Andre Martins
| Challenge: | Despite advances in machine translation quality estimation and evaluation, decoding is mostly oblivious to this. |
| Approach: | They propose to use a decoding framework that is quality-aware for neural machine translation . they compare various methods like N-best reranking and minimum Bayes risk decoding . |
| Outcome: | The proposed quality-aware decoding outperforms MAP-based decoding on four datasets and two model classes. |
Non-Exchangeable Conformal Language Generation with Nearest Neighbors (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to evaluate reliability of generated text are lacking in natural language generation. |
| Approach: | They propose a non-exchangeable conformal prediction method that provides bounds on coverage . they validated their method with k-NN retrieval and show that it produces encouraging results . |
| Outcome: | The proposed method produces encouraging results in machine translation and language modeling tasks. |
An Empirical Study of Translation Hypothesis Ensembling with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are becoming a one-fits-many solution, but they sometimes hallucinate or produce unreliable output. |
| Approach: | They propose to use several LLMs to ensemble translation hypotheses . they use instruction tuning, quality-based reranking, and minimum Bayes risk (MBR) decoding to improve translation quality. |
| Outcome: | The proposed method improves translation quality and instruction tuning improves the quality of the output. |
Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning (2023.findings-emnlp)
Copied to clipboard
Duarte Alves, Nuno Guerreiro, João Alves, José Pombal, Ricardo Rei, José de Souza, Pierre Colombo, Andre Martins
| Challenge: | Large language models (LLMs) are a promising avenue for machine translation (MT) however, their effectiveness depends on the choice of few-shot examples and they often require extra post-processing due to overgeneration. |
| Approach: | They propose a method that incorporates few-shot examples during finetuning to improve performance on MT tasks. |
| Outcome: | The proposed method outperforms few-shot prompting while eliminating the need for in-context examples. |
A Recipe of Parallel Corpora Exploitation for Multilingual Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have highlighted the potential of exploiting parallel corpora to enhance multilingual large language models. |
| Approach: | They investigate the impact of parallel corpora quality and quantity, training objectives, and model size on performance of multilingual large language models enhanced with parallel corporeal. |
| Outcome: | The proposed approach improves performance in bilingual and general-purpose tasks. |
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation (2025.acl-long)
Copied to clipboard
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, Sara Hooker
| Challenge: | Reliable multilingual evaluation is difficult and culturally appropriate evaluation is even harder to achieve. |
| Approach: | They propose a multilingual evaluation framework that aims to mitigate these biases by improving translations and annotation practices. |
| Outcome: | The proposed framework improves translation quality and cultural coverage and is culturally sensitive and culturally agnostic. |
An Interdisciplinary Approach to Human-Centered Machine Translation (2025.emnlp-main)
Copied to clipboard
Marine Carpuat, Omri Asscher, Kalika Bali, Luisa Bentivogli, Fred Blain, Lynne Bowker, Monojit Choudhury, Hal Daumé Iii, Kevin Duh, Ge Gao, Alvin C Grissom II, Marzena Karpinska, Elaine C Khoong, William D. Lewis, Andre Martins, Mary Nurminen, Douglas W. Oard, Maja Popovic, Michel Simard, François Yvon
| Challenge: | Despite progress in MT, a gap persists between how the technology is developed and how it is used in real-world contexts. |
| Approach: | They propose a human-centered approach to machine translation (MT) they argue that MT should be evaluated with diverse goals and contexts of use . |
| Outcome: | The proposed approach emphasizes alignment of evaluation and design with diverse communicative goals and contexts of use. |
Analyzing Context Contributions in LLM-based Machine Translation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved state-of-the-art performance in machine translation . however, the mechanisms by which LLMs use different parts of the input context remain unexplored . |
| Approach: | They propose to analyze how large language models use different parts of the input context . they highlight several key findings: the source part of few-shot examples contributes more than its corresponding targets . |
| Outcome: | The proposed model can leverage in-context learning to perform translation tasks without training . the proposed model is able to perform tasks without being explicitly trained for them . |
Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation (2024.emnlp-main)
Copied to clipboard
Sweta Agrawal, José De Souza, Ricardo Rei, António Farinhas, Gonçalo Faria, Patrick Fernandes, Nuno Guerreiro, Andre Martins
| Challenge: | Existing algorithms for machine translation do not match human preferences, but they can be expensive to obtain and curate at a large scale. |
| Approach: | They propose an approach that leverages the best of both worlds by collecting sentence-level quality assessments from professional linguists on translations generated by multiple high-quality MT systems. |
| Outcome: | The proposed approach improves translation quality on WMT23 and FLORES benchmarks. |