Papers by Eneko Agirre
Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing models of multilingual sentence embeddings require large parallel data resources which are not available for low-resource languages. |
| Approach: | They propose an unsupervised method to derive multilingual sentence embeddings using monolingual data. |
| Outcome: | The proposed method improves on two parallel corpus mining tasks and for other languages. |
Analyzing the Limitations of Cross-lingual Word Embedding Mappings (P19-1)
Copied to clipboard
| Challenge: | Existing methods for cross-lingual word embeddings have limited results . existing methods require little or no cross-linguistic signal to work . |
| Approach: | They compare offline mapping methods to an extension of skip-gram that jointly learns both embedding spaces. |
| Outcome: | The proposed method yields more isomorphic embeddings, is less sensitive to hubness, and achieves stronger results in bilingual lexicon induction. |
Principled Paraphrase Generation with Parallel Corpora (2022.acl-long)
Copied to clipboard
| Challenge: | a popular method for paraphrase generation is round-trip machine translation (MT). |
| Approach: | They propose a method that formalizes the implicit similarity function and relaxes it by requiring the entire translation distribution to match . they propose MT can be used to generate paraphrases by decoding back to the source without having to generate pivot translations. |
| Outcome: | The proposed approach is more principled and efficient than round-trip machine translation (MT) and offers an adjustable parameter to control the fidelity-diversity trade-off. |
CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Methods for adapting language models to new tasks and domains have traditionally assumed white-box access to the model and work by modifying its parameters. |
| Approach: | They propose a method for adapting large language models to new domains and tasks . they fine-tune a small white-box LM and combine it with a large black-box model at the probability level through a network, learned on a smaller validation set. |
| Outcome: | The proposed method improves performance in all cases, while using a domain expert 23x smaller. |
WiCkeD: A Simple Method to Make Multiple Choice Benchmarks More Challenging (2025.acl-short)
Copied to clipboard
| Challenge: | Multiple choice question (MCQ) benchmarks are widely used to evaluate Large Language Models (LLMs). |
| Approach: | They propose a method to increase the complexity of existing multiple-choice benchmarks by randomly replacing a choice with “None of the above”. |
| Outcome: | The proposed method can be applied to 6 popular benchmarks and evaluate 18 open-weight LLMs. |
PixT3: Pixel-based Table-To-Text Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Table-to-text generation is a visual recognition task that uses textual descriptions from structured inputs. |
| Approach: | They propose to rethink data-to-text generation as a visual recognition task by removing the need for rendering the input in a string format. |
| Outcome: | The proposed model overcomes the challenges of linearization and input size limitations and is applicable to open-ended and controlled generation settings. |
ZS4IE: A toolkit for Zero-Shot Information Extraction with simple Verbalizations (2022.naacl-demo)
Copied to clipboard
| Challenge: | Information Extraction (IE) analysts use supervised machine learning to define the schema and build a training corpus with annotated examples. |
| Approach: | They propose a workflow where the analyst verbalizes the entities/relations, which are then used by a Textual Entailment model to perform zero-shot IE. |
| Outcome: | The proposed workflow performs very well on four IE tasks with a single user interface and a video demonstration is available on vimeo. |
Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work shows that Relation Extraction tasks can be recasted as Textual Entailment tasks using verbalizations. |
| Approach: | They propose to recasted RE tasks as Textual Entailment tasks using verbalizations . they show that entailment reduces the need for manual annotation to 50% and 20% . |
| Outcome: | The proposed method reduces the need for manual annotation to 50% and 20% in event argument extraction tasks while achieving the same performance as with full training. |
Label Verbalization and Entailment for Effective Zero and Few-Shot Relation Extraction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Relation extraction systems require large amounts of labeled examples which are costly to annotate. |
| Approach: | They propose to use hand-made relation extraction tasks to refine a pretrained textual entailment engine which is run as-is or further fine-tuned on labeled examples. |
| Outcome: | The proposed system achieves 63% F1 zero-shot, 69% with 16 examples per relation and 4 points short of the state-of-the-art system on the same conditions. |
A Methodology for Creating Question Answering Corpora Using Inverse Data Annotation (2020.acl-main)
Copied to clipboard
Jan Deriu, Katsiaryna Mlynchyk, Philippe Schläpfer, Alvaro Rodrigo, Dirk von Grünigen, Nicolas Kaiser, Kurt Stockinger, Eneko Agirre, Mark Cieliebak
| Challenge: | Existing methods to efficiently construct corpus for question answering over structured data are time-consuming and cost-intensive. |
| Approach: | They propose a method to efficiently construct a corpus for question answering over structured data. |
| Outcome: | The proposed method triples the annotation speed while maintaining complexity of queries. |
An Effective Approach to Unsupervised Machine Translation (P19-1)
Copied to clipboard
| Challenge: | a recent research line has managed to train both unsupervised and unsupervised machine translation systems using monolingual corpora only. |
| Approach: | They propose to use monolingual corpora to train both unsupervised and unsupervised machine translation systems. |
| Outcome: | The proposed system achieves 22.5 BLEU points in English-to-German WMT 2014, 5.5 points more than the previous best unsupervised system, and 0.5 points more in the (supervised) shared task winner back in 2014. |
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for evaluating large language models using annotated benchmarks are in trouble . data contamination can cause wrong scientific conclusions being published . |
| Approach: | They argue that the evaluation of NLP tasks using annotated benchmarks is in trouble . they define different levels of data contamination and propose a community effort . |
| Outcome: | The proposed measures should detect when data from a benchmark was exposed to a model and flag papers with conclusions compromised by data contamination. |
Conversational Question Answering in Low Resource Scenarios: A Dataset and Case Study for Basque (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing datasets for conversational question answering systems are expensive and limited in resources . a dataset of thousands of dialogues and tens of thousands question answering turns is available for free . |
| Approach: | They aim to test the performance of Conversational Question Answering systems in non-English languages . they use a dataset built on top of Wikipedia sections about popular people and organizations . |
| Outcome: | The results show that the system can handle low-resource conditions comparable to English . the results also show that dialogue history models are not directly transferable to another language . |
Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems (2020.emnlp-main)
Copied to clipboard
Jan Deriu, Don Tuggener, Pius von Däniken, Jon Ander Campos, Alvaro Rodrigo, Thiziri Belkacem, Aitor Soroa, Eneko Agirre, Mark Cieliebak
| Challenge: | Lack of time efficient and reliable evalu-ation methods is hampering the development of conversational dialogue systems (chatbots). |
| Approach: | They propose a framework that replaces human-bot conversations with conversations between bots and an annotation tool that ranks chatbots based on their ability to mimic human behaviour. |
| Outcome: | The proposed evaluation framework replaces human-bot conversations with bot conversations and allows for frequent evaluations of chatbots during their evaluation cycle. |
Event Extraction in Basque: Typologically Motivated Cross-Lingual Transfer-Learning Analysis (2024.lrec-main)
Copied to clipboard
| Challenge: | Using a multilingual language model, Event Extraction tasks require humans to follow complicated guidelines and follow complicated rules. |
| Approach: | They propose a multilingual multilingual language model that is trained in a source language and applied to a target language. |
| Outcome: | The proposed model is based on a multilingual event extraction dataset for Basque . it shows that the shared linguistic characteristic between source and target languages does have an impact on transfer quality. |
Probing for Semantic Classes: Diagnosing the Meaning Content of Word Embeddings (P19-1)
Copied to clipboard
| Challenge: | Empirical analysis of word embeddings of ambiguous words is limited by the small size of manually annotated resources and by the fact that word senses are treated as unrelated individual concepts. |
| Approach: | They present a large dataset based on manual Wikipedia annotations and word senses, where word sense from different words are related by semantic classes. |
| Outcome: | The proposed method can predict whether a word is single-sense or multi-sensor, if the sense is frequent, and it can predict rare senses. |
Improving Factuality in Clinical Abstractive Multi-Document Summarization by Guided Continued Pre-training (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing methods for enhancing the factual accuracy of abstractive summarization models are not effective in fact-critical domains like clinical document summarizing. |
| Approach: | They propose a guided continued pre-training stage for encoder-decoder models followed by supervised fine-tuning on summarization. |
| Outcome: | The proposed approach improves the quality and factuality of the summaries and achieves the best-published results on the clinical document summarization task. |
A Call for More Rigor in Unsupervised Cross-lingual Learning (2020.acl-main)
Copied to clipboard
| Challenge: | Existing research on unsupervised cross-lingual learning has focused on purely unsupervised learning without any parallel data for most of the world's languages. |
| Approach: | They propose to define "multilingual learning" as learning a common model for two or more languages from raw text, without any downstream task labels. |
| Outcome: | The proposed model is based on a model with no parallel data and abundant monolingual data. |
A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings (P18-1)
Copied to clipboard
| Challenge: | Existing methods to learn cross-lingual word embeddings have failed in more realistic scenarios . a fully unsupervised initialization and a robust self-learning algorithm are needed to improve the existing methods. |
| Approach: | They propose an unsupervised initialization method that exploits structural similarity of embeddings and a robust self-learning algorithm that iteratively improves it. |
| Outcome: | The proposed method achieves the best published results in standard datasets even surpassing previous supervised systems. |
Translation Artifacts in Cross-lingual Transfer Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing cross-lingual transfer learning techniques involve human and machine translations. |
| Approach: | They propose to use machine translation to translate test set or training set to introduce subtle artifacts that have a notable impact in existing cross-lingual models. |
| Outcome: | The proposed translation process reduces the lexical overlap between the premise and hypothesis by 4.3 and 2.8 points . the proposed translation-test and zero-shot approaches improve on previous work . |
PoeLM: A Meter- and Rhyme-Controllable Language Model for Unsupervised Poetry Generation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating formal verse poetry use existing poems for supervision, which are difficult to obtain for most languages and poetic forms. |
| Approach: | They propose an unsupervised approach to generate formal verse poetry without supervision . they use control codes to describe meter and rhyme scheme constraints, and train a transformer language model . |
| Outcome: | The proposed method generates poems that follow any given meter and rhyme scheme without training . it is comparable to those written by humans and generates comparable quality poems . |
DoQA - Accessing Domain-Specific FAQs via Conversational QA (2020.acl-main)
Copied to clipboard
| Challenge: | a dataset of 2,437 dialogues and 10,917 QA pairs is used to access domain-specific FAQ information. |
| Approach: | They present a dataset with 2,437 dialogues and 10,917 QA pairs for FAQs . they use the Wizard of Oz method with crowdsourcing to create dialogues using the original post and the original reply. |
| Outcome: | The proposed system can access domain-specific FAQ information without training data. |
GUIDEX: Guided Synthetic Data Generation for Zero-Shot Information Extraction (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing domain-specific IE systems require expert schema design, data annotation, and model training. |
| Approach: | They propose a method that automatically defines domain-specific schemas and infers guidelines and generates synthetically labeled instances. |
| Outcome: | The proposed method improves on seven zeroshot Named Entity Recognition benchmarks. |
Give your Text Representation Models some Love: the Case for Basque (2020.lrec-1)
Copied to clipboard
Rodrigo Agerri, Iñaki San Vicente, Jon Ander Campos, Ander Barrena, Xabier Saralegi, Aitor Soroa, Eneko Agirre
| Challenge: | Word embeddings and pre-trained language models are expensive to train and are often used by small companies and research groups to build their own. |
| Approach: | They propose to use word embeddings and pre-trained language models to build rich representations of text and improve NLP tasks. |
| Outcome: | The proposed models perform better than publicly available versions in downstream NLP tasks for Basque. |
EnerGIZAr: Leveraging GIZA++ for Effective Tokenizer Initialization (2025.findings-acl)
Copied to clipboard
| Challenge: | Continual pre-training has long been considered the default strategy for adapting models to non-English languages, but struggles with initializing new embeddings, especially for non-Latin scripts. |
| Approach: | They propose a method that leverages statistical word alignment techniques to improve continual pre-training by leveraging word alignment matrix between source and target tokens. |
| Outcome: | The proposed method outperforms existing methods on key NLP tasks including POS tagging, Sentiment Analysis, NLI, and NER in Hindi, Basque, Arabic and Korean. |
Beyond Offline Mapping: Learning Cross-lingual Word Embeddings through Context Anchoring (2021.acl-long)
Copied to clipboard
| Challenge: | Recent research on cross-lingual word embeddings has been dominated by unsupervised mapping approaches that align monolingual embedders. |
| Approach: | They propose an unsupervised mapping approach that fixes fixed embeddings and learns new ones for the source language that are aligned with them. |
| Outcome: | The proposed method outperforms conventional mapping methods on bilingual lexicon induction and obtains competitive results in the downstream XNLI task. |
Unsupervised Statistical Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) systems can be trained from monolingual corpora without supervision. |
| Approach: | They propose a phrase-based approach that trains from monolingual corpora . their method is based on phrase-driven Statistical Machine Translation (SMT) they propose to train NMT systems without supervision from monolinguistic corpors . |
| Outcome: | The proposed approach improves on the existing supervised systems by combining a phrase table with an n-gram language model and fine-tuning hyperparameters through an unsupervised MERT variant. |
Bilingual Lexicon Induction through Unsupervised Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Existing methods for bilingual lexicon induction use nearest neighbor or related retrieval methods to induce word translation pairs. |
| Approach: | They propose a method that aligns word embeddings in two languages and uses them to build a phrase-table and a language model to extract the bilingual lexicon. |
| Outcome: | The proposed method improves accuracy 6 points over nearest neighbor and 4 points over CSLS retrieval on the same cross-lingual embeddings. |
Improving Conversational Question Answering Systems after Deployment using Feedback-Weighted Learning (2020.coling-main)
Copied to clipboard
| Challenge: | In most applications, users are not able to provide the correct answer to the system, but they are able provide binary (correct, incorrect) feedback. |
| Approach: | They propose feedback-weighted learning based on importance sampling to improve upon an initial supervised system using binary user feedback. |
| Outcome: | The proposed method improves on an initial supervised system, getting close to a fully-supervised system that has access to the same labeled examples in in-domain experiments (QuAC) and matching in out-of-domain experiment (DoQA). |
Emergent Abilities of Large Language Models under Continued Pre-training for Language Adaptation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing large language models are notoriously English-centric, and their performance has been reported to drop significantly in lessresourced languages. |
| Approach: | They propose a language-agnostic benchmark for in-context learning that reveals catastrophic forgetting early on CPT when English is not included. |
| Outcome: | The proposed method does not impact validation perplexity but is critical for emergence of downstream capabilities in the target language. |
Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque (2025.emnlp-main)
Copied to clipboard
Oscar Sainz, Naiara Perez, Julen Etxaniz, Joseba Fernandez de Landa, Itziar Aldabe, Iker García-Ferrero, Aimar Zabala, Ekhi Azurmendi, German Rigau, Eneko Agirre, Mikel Artetxe, Aitor Soroa
| Challenge: | Instructing language models with user intent requires large instruction datasets limited to a limited set of languages. |
| Approach: | They propose to use existing LLMs and synthetically generated instructions to train models with user intent. |
| Outcome: | The proposed model outperforms base non-instructed models on Basque without Basque instructions. |