Papers by Javier Aula-Blasco
Building a Data Infrastructure for a Mid-Resource Language: The Case of Catalan (2024.lrec-main)
Copied to clipboard
Aitor Gonzalez-Agirre, Montserrat Marimon, Carlos Rodriguez-Penagos, Javier Aula-Blasco, Irene Baucells, Carme Armentano-Oller, Jorge Palomar-Giner, Baybars Kulebi, Marta Villegas
| Challenge: | Aina Project aims to provide Catalan with the resources needed to keep its relevance in AI/NLP applications. |
| Approach: | They propose a set of strategies to consider when improving technology support for a mid- or low-resource language . they propose annotated datasets and a framework to make models ready to use . |
| Outcome: | The Aina Project aims to provide Catalan with the necessary resources to keep its relevance in AI/NLP-related industry and research. |
VeritasQA: A Truthfulness Benchmark Aimed at Multilingual Transferability (2025.coling-main)
Copied to clipboard
Javier Aula-Blasco, Júlia Falcão, Susana Sotelo, Silvia Paniagua, Aitor Gonzalez-Agirre, Marta Villegas
| Challenge: | Large Language Models (LLMs) struggle with falsehoods and model hallucination . many efforts struggle to surpass 50% accuracy, with only targeted techniques reaching around 65% . |
| Approach: | They propose a truthfulness benchmark that focuses on imitative falsehoods . they use a set of 353 questions and answers inspired by common misconceptions based on the language . |
| Outcome: | The benchmark is available in Spanish, Catalan, Galician and English . it measures the truthfulness of multilingual LLMs using 353 questions and answers . |
La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America (2025.acl-long)
Copied to clipboard
María Grandury, Javier Aula-Blasco, Júlia Falcão, Clémentine Fourrier, Miguel González Saiz, Gonzalo Martínez, Gonzalo Santamaria Gomez, Rodrigo Agerri, Nuria Aldama García, Luis Chiruzzo, Javier Conde, Helena Gomez Adorno, Marta Guerrero Nieto, Guido Ivetta, Natàlia López Fuertes, Flor Miriam Plaza-del-Arco, María-Teresa Martín-Valdivia, Helena Montoro Zamorano, Carmen Muñoz Sanz, Pedro Reviriego, Leire Rosado Plaza, Alejandro Vaca Serrano, Estrella Vallecillo-Rodríguez, Jorge Vallego, Irune Zubiaga
| Challenge: | La Leaderboard is the first open-source leaderboard to evaluate generative Large Language Models (LLMs) in languages and language varieties of Spain and Latin America. |
| Approach: | They propose to use La Leaderboard to evaluate generative Large Language Models in Spanish and Latin America. |
| Outcome: | La Leaderboard is the first open-source leaderboard to evaluate generative LLMs in languages and language varieties of Spain and Latin America. |
Multi-LMentry: Can Multilingual LLMs Solve Elementary Tasks Across Languages? (2025.emnlp-main)
Copied to clipboard
Luca Moroni, Javier Aula-Blasco, Simone Conia, Irene Baucells, Naiara Perez, Silvia Paniagua Suárez, Anna Sallés, Malte Ostendorff, Júlia Falcão, Guijin Son, Aitor Gonzalez-Agirre, Roberto Navigli, Marta Villegas
| Challenge: | a recent study focused on complex, high-level tasks, but LMentry is limited to English . a multilingual evaluation of large language models is needed to address this gap, authors say . |
| Approach: | They propose a compact benchmark that enables systematic evaluation of large language models . they propose to use tasks that are trivial for humans but remain surprisingly difficult for LLMs . |
| Outcome: | The proposed benchmark is limited to English, leaving its insights linguistically narrow. |
IberoBench: A Benchmark for LLM Evaluation in Iberian Languages (2025.coling-main)
Copied to clipboard
Irene Baucells, Javier Aula-Blasco, Iria de-Dios-Flores, Silvia Paniagua Suárez, Naiara Perez, Anna Salles, Susana Sotelo Docio, Júlia Falcão, Jose Javier Saiz, Robiert Sepulveda Torres, Jeremy Barnes, Pablo Gamallo, Aitor Gonzalez-Agirre, German Rigau, Marta Villegas
| Challenge: | Existing multi-task benchmarks for Large Language Models are limited to English . a new benchmark is needed to evaluate models on a range of tasks . |
| Approach: | They propose a multilingual, multi-task benchmark for Iberian languages built on the LM Evaluation Harness framework. |
| Outcome: | The proposed benchmark covers 62 tasks divided into 179 subtasks and is available in Iberian, Basque, Catalan, Galician, European Spanish and European Portuguese. |