Papers by Julen Etxaniz
Do Multilingual Language Models Think Better in English? (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing studies show that translation-test improves performance of multilingual models by translating the input into English using an external machine translation system. |
| Approach: | They propose a new approach that leverages the few-shot translation capabilities of multilingual language models. |
| Outcome: | The proposed approach outperforms direct inference on 5 tasks. |
Truth Knows No Language: Evaluating Truthfulness Beyond English (2025.acl-long)
Copied to clipboard
Blanca Calvo Figueras, Eneko Sagarzazu, Julen Etxaniz, Jeremy Barnes, Pablo Gamallo, Iria de-Dios-Flores, Rodrigo Agerri
| Challenge: | a new benchmark evaluates the truthfulness of large language models (LLMs) based on imitative falsehoods. |
| Approach: | They propose a professionally translated extension of the TruthfulQA benchmark . it evaluates truthfulness in Basque, Catalan, Galician, and Spanish . |
| Outcome: | The proposed extension of the TruthfulQA benchmark evaluates truthfulness in Basque, Catalan, Galician, and Spanish. |
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for evaluating large language models using annotated benchmarks are in trouble . data contamination can cause wrong scientific conclusions being published . |
| Approach: | They argue that the evaluation of NLP tasks using annotated benchmarks is in trouble . they define different levels of data contamination and propose a community effort . |
| Outcome: | The proposed measures should detect when data from a benchmark was exposed to a model and flag papers with conclusions compromised by data contamination. |
XNLIeu: a dataset for cross-lingual NLI in Basque (2024.naacl-long)
Copied to clipboard
| Challenge: | XNLI is a popular benchmark used to evaluate cross-lingual Natural Language Understanding (NLU) in languages such as English, Basque and other low-resource languages. |
| Approach: | They expand XNLI to include Basque, a low-resource language that can benefit from transfer-learning approaches. |
| Outcome: | The proposed dataset includes Basque, a low-resource language that can benefit from transfer-learning approaches. |
BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Data (2026.eacl-long)
Copied to clipboard
Jaap Jumelet, Abdellah Fourtassi, Akari Haga, Bastian Bunzeck, Bhargav Shandilya, Diana Galvan-Sosa, Faiz Ghifari Haznitrama, Francesca Padovani, Francois Meyer, Hai Hu, Julen Etxaniz, Laurent Prevot, Linyang He, María Grandury, Mila Marcheva, Negar Foroutan, Nikitas Theodoropoulos, Pouya Sadeghi, Siyuan Song, Suchir Salhan, Susana Zhou, Yurii Paniv, Ziyin Zhang, Arianna Bisazza, Alex Warstadt, Leshem Choshen
| Challenge: | prevailing trend in language modeling research is to prioritize scaling, authors say . from infancy to maturity, English learners acquire language through exposure to less than 100M words . |
| Approach: | They propose a multilingual collection of datasets modeling the language a person observes from birth until they acquire a native language. |
| Outcome: | The proposed models outperform models trained on a fixed, developmentally plausible English corpus on various benchmarks. |
Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque (2025.emnlp-main)
Copied to clipboard
Oscar Sainz, Naiara Perez, Julen Etxaniz, Joseba Fernandez de Landa, Itziar Aldabe, Iker García-Ferrero, Aimar Zabala, Ekhi Azurmendi, German Rigau, Eneko Agirre, Mikel Artetxe, Aitor Soroa
| Challenge: | Instructing language models with user intent requires large instruction datasets limited to a limited set of languages. |
| Approach: | They propose to use existing LLMs and synthetically generated instructions to train models with user intent. |
| Outcome: | The proposed model outperforms base non-instructed models on Basque without Basque instructions. |