Papers by Roberto Navigli
How Much Do Encoder Models Know About Word Senses? (2025.acl-long)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a key task in Natural Language Processing (NLP) however, how well these models inherently disambiguate word senses remains uncertain. |
| Approach: | They evaluate several encoder-only PLMs across WordNet and ODE sense inventories to evaluate their ability to separate word senses without any task-specific fine-tuning. |
| Outcome: | The proposed model outperforms output layer on WordNet and ODE sense inventories by 15 percentage points. |
Building Semantic Grams of Human Knowledge (2020.lrec-1)
Copied to clipboard
| Challenge: | Word senses are typically defined with textual definitions and put in context via lexical-semantic relations such as synonymy, antonymy, hypernymy, etc. |
| Approach: | They propose a slot-filler structure to define the meaning of words in terms of their prototypical semantic information. |
| Outcome: | The proposed model improves on a semantic similarity task and shows significant improvements over state-of-the-art embeddings. |
SRL4E – Semantic Role Labeling for Emotions: A Unified Evaluation Framework (2022.acl-long)
Copied to clipboard
| Challenge: | Existing datasets for emotion detection are heterogeneous in size, domain, format, splits, emotion categories and role labels, hampering progress in this area. |
| Approach: | They propose a framework for annotating emotions manually using a common labeling scheme to unify several datasets tagged with emotions and semantic roles. |
| Outcome: | The proposed framework unifies datasets tagged with emotions and semantic roles by using a common labeling scheme. |
Incorporating Graph Information in Transformer-based AMR Parsing (2023.findings-acl)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a semantic graph abstraction for text representations. |
| Approach: | They propose a model and method that incorporates graph information into the learned representations of AMR by word-to-node alignment. |
| Outcome: | The proposed model improves AMR parsing performance by embedding graph information into the encoder at training time. |
A Tour of Explicit Multilingual Semantics: Word Sense Disambiguation, Semantic Role Labeling and Semantic Parsing (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | a recent advent of pretrained language models has sparked a revolution in NLP . but, there are still questions about whether current approaches capture explicit, symbolic meaning . this tutorial will review efforts to tackle three key open problems in lexical and sentence-level semantics . |
| Approach: | This tutorial reviews recent efforts to shed light on meaning in NLP . it will focus on three key open problems in lexical and sentence-level semantics . |
| Outcome: | This tutorial reviews recent efforts to shed light on meaning in NLP . it focuses on three key open problems in lexical and sentence-level semantics . |
Personalized PageRank with Syntagmatic Information for Multilingual Word Sense Disambiguation (2020.acl-demos)
Copied to clipboard
| Challenge: | SyntagRank is a knowledge-based WSD system that exploits syntagmatic information to perform state-of-the-art knowledge-driven WSD in a multilingual setting. |
| Approach: | They propose to exploit syntagmatic information to perform state-of-the-art knowledge-based WSD in a multilingual setting by using a Web interface and a RESTful API. |
| Outcome: | SyntagRank exploits disambiguated pairs of words in SyntagNet to perform state-of-the-art knowledge-based WSD in a multilingual setting. |
Generationary or “How We Went beyond Word Sense Inventories and Learned to Gloss” (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to Word Sense Disambiguation use discrete word senses . however, many language users have different understandings of words . |
| Approach: | They propose a unified computational lexical semantics model that can produce contextually appropriate definitions. |
| Outcome: | The proposed model outperforms existing models in lexical semantics and discriminative tasks. |
Language Pivoting from Parallel Corpora for Word Sense Disambiguation of Historical Languages: A Case Study on Latin (2024.lrec-main)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is an important task in NLP . most of the work on this task has been done on contemporary English or other modern languages, leaving challenges posed by low-resource languages and diachronic change open. |
| Approach: | They propose to use existing bilingual corpora instead of native English datasets to generate a Latin WSD model. |
| Outcome: | The proposed approach achieves state-of-the-art on a standard benchmark for Latin WSD. |
ESC: Redesigning WSD with Extractive Sense Comprehension (2021.naacl-main)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a historical NLP task aimed at linking words in contexts to discrete sense inventories. |
| Approach: | They propose a transformer-based neural architecture for extractive Sense Comprehension to solve a span extraction problem and a new state of the art English WSD task. |
| Outcome: | The proposed model outdoes all of its competitors while relying on three times fewer annotations. |
Huge Automatically Extracted Training-Sets for Multilingual Word SenseDisambiguation (L18-1)
Copied to clipboard
| Challenge: | Word Sense Disambiguation is a crucial task in Natural Language Processing . supervised systems need to be trained on word-by-word basis, a problem that is beyond reach for resource-rich languages like English. |
| Approach: | They release six large-scale sense-annotated datasets in multiple languages to pave the way for supervised multilingual Word Sense Disambiguation. |
| Outcome: | The results show that large-scale sense annotations can be used as training sets for supervised systems. |
SGL: Speaking the Graph Languages of Semantic Parsing via Multilingual Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Graph-based semantic parsing is one of the most promising general-purpose meaning representations . owing to this heterogeneity, most research focused on solutions specific to a given formalism . |
| Approach: | They propose a multilingual neural machine translation framework for Graph-based semantic parsing . they propose Graph2seq architecture that trains with an MNMT objective . |
| Outcome: | The proposed framework outperforms all competitors on cross-lingual parsing tasks. |
Fatality Killed the Cat or: BabelPic, a Multimodal Dataset for Non-Concrete Concepts (2020.acl-main)
Copied to clipboard
| Challenge: | Existing datasets for image understanding do not cover concepts denoting concrete, tangible things such as CAT, TRAFFIC LIGHT and so on. |
| Approach: | They present a hand-labeled image-synset dataset cleaning the BabelNet Lexical Knowledge Base (LKB) they explicitly target non-concrete concepts, thus providing refreshing new data for the community . |
| Outcome: | The proposed dataset cleans the image-synset association within the BabelNet Lexical Knowledge Base (LKB) it explicitly targets non-concrete concepts, providing refreshing new data for the community. |
BOOKCOREF: Coreference Resolution at Book Scale (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for coreference resolution systems are limited in length and do not adequately assess system capabilities at the book scale. |
| Approach: | They propose a novel pipeline that produces high-quality coreference resolution annotations on full narrative texts and a book-scale benchmark, BOOKCOREF. |
| Outcome: | The proposed pipeline produces high-quality coreference resolution annotations on full texts with an average document length of more than 200,000 tokens. |
Nibbling at the Hard Core of Word Sense Disambiguation (2022.acl-long)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a task that is based on a set of pre-trained language models. |
| Approach: | They propose to use Word Sense Disambiguation to test whether systems can handle ambiguous words. |
| Outcome: | The proposed benchmarks show that seven of the most representative state-of-the-art systems make trivial errors on traditional evaluation benchmarks. |
Neuralign: A Context-Aware, Cross-Lingual and Fully-Neural Sentence Alignment System for Long Texts (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing sentence alignment systems focus on auxiliary information such as document metadata and hyperparameter-sensitive techniques, and neglect the crucial role that context plays in the alignment process. |
| Approach: | They propose a context-aware, end-to-end and fully-neural architecture for sentence alignment that maps source and target sentences in long documents by contextualizing their sentence embeddings with respect to the other sentences in the document. |
| Outcome: | The proposed system maps source and target sentences in long documents by contextualizing their sentence embeddings with respect to the other sentences in the document. |
Breaking Through the 80% Glass Ceiling: Raising the State of the Art in Word Sense Disambiguation by Incorporating Knowledge Graph Information (2020.acl-main)
Copied to clipboard
| Challenge: | Neural architectures are the current state of the art in Word Sense Disambiguation (WSD) however, they make limited use of the vast amount of relational information encoded in Lexical Knowledge Bases (LKBs). |
| Approach: | They propose a neural supervised architecture that embeds Lexical Knowledge Bases and exploits pretrained synset embeddings to predict synsets that are not in the training set. |
| Outcome: | The proposed architecture breaks through the 80% ceiling on the concatenation of all standard all-words English evaluation benchmarks. |
Analyzing Homonymy Disambiguation Capabilities of Pretrained Language Models (2024.lrec-main)
Copied to clipboard
Lorenzo Proietti, Stefano Perrella, Simone Tedeschi, Giulia Vulpis, Leonardo Lavalle, Andrea Sanchietti, Andrea Ferrari, Roberto Navigli
| Challenge: | Word Sense Disambiguation (WSD) is a key task in Natural Language Processing (NLP) but current pretrained language models lack the granularity to perform disambiguation . |
| Approach: | They propose a large-scale resource that leverages homonymy relations to cluster WordNet senses and train Homonymy Disambiguation systems. |
| Outcome: | The proposed model can distinguish homonyms with up to 95% accuracy even without fine-tuning the underlying PLM. |
Code-Switching with Word Senses for Pretraining in Neural Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Lexical ambiguity is a significant and pervasive challenge in Neural Machine Translation (NMT) many state-of-the-art (SOTA) NMT systems struggle to handle polysemous words . |
| Approach: | They propose an end-to-end approach for pretraining multilingual NMT models leveraging word sense-specific information from Knowledge Bases. |
| Outcome: | The proposed approach improves translation quality and scales to various data and resource-strapped scenarios. |
Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that MT metrics return assessments as scalar scores that are difficult to interpret, posing a challenge to making informed design choices. |
| Approach: | They propose an interpretable evaluation framework that evaluates MT metrics in two scenarios that serve as proxies for filtering and translation re-ranking use cases. |
| Outcome: | The proposed framework offers clearer insights than correlation with human judgments. |
AMRs Assemble! Learning to Ensemble with Autoregressive Models for AMR Parsing (2023.acl-short)
Copied to clipboard
| Challenge: | et al., 2013) examines the current state-of-the-art in AMR parsing . current models violate structural constraints, but they can corrupt graphs . |
| Approach: | They propose two new ensemble strategies to improve AMR parsing robustness and reduce computational time. |
| Outcome: | The proposed methods improve robustness to structural constraints while reducing computational time. |
CNER: Concept and Named Entity Recognition (2024.naacl-long)
Copied to clipboard
| Challenge: | Concept and Named Entity Recognition (CNER) is a new unified task that handles concepts and entities mentioned in unstructured texts seamlessly. |
| Approach: | They propose a new unified task that handles concepts and entities mentioned in unstructured texts seamlessly. |
| Outcome: | The proposed task gains +5.4 and +8 macro F1 points when performed as a unified task compared to specialized named entity and concept recognition systems. |
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code (2025.coling-industry)
Copied to clipboard
Taishi Nakamura, Mayank Mishra, Simone Tedeschi, Yekun Chai, Jason T. Stillerman, Felix Friedrich, Prateek Yadav, Tanmay Laud, Vu Minh Chien, Terry Yue Zhuo, Diganta Misra, Ben Bogin, Xuan-Son Vu, Marzena Karpinska, Arnav Varma Dantuluri, Wojciech Kusa, Tommaso Furlanello, Rio Yokota, Niklas Muennighoff, Suhas Pai, Tosin Adewumi, Veronika Laippala, Xiaozhe Yao, Adalberto Barbosa Junior, Aleksandr Drozd, Jordan Clive, Kshitij Gupta, Liangyu Chen, Qi Sun, Ken Tsui, Nour Moustafa-Fahmy, Nicolo Monti, Tai Dang, Ziyang Luo, Tien-Tung Bui, Roberto Navigli, Virendra Mehta, Matthew Blumberg, Victor May, Hiep Nguyen, Sampo Pyysalo
| Challenge: | Pretrained language models are integral part of AI applications, but their high computational cost limits accessibility. |
| Approach: | They evaluate Aurora-M, a 15B parameter multilingual open-source model trained on English, Finnish, Hindi, Japanese, Vietnamese, and code. |
| Outcome: | The proposed model outperforms existing models on English, Finnish, Hindi, Japanese, Vietnamese, and code. |
GeneSis: A Generative Approach to Substitutes in Context (2021.emnlp-main)
Copied to clipboard
| Challenge: | lexical substitution tasks require a system to provide adequate replacements for a word in a given context. |
| Approach: | They propose a generative approach to lexical substitution using a seq2seq model to generate suitable replacements for a word in context. |
| Outcome: | The proposed approach achieves state-of-the-art on different benchmarks and human evaluation of the generated substitutes. |
ID10M: Idiom Identification in 10 Languages (2022.findings-naacl)
Copied to clipboard
| Challenge: | Identifying and understanding idioms in context is a key goal and challenge in Natural Language Understanding tasks. |
| Approach: | They propose a multilingual Transformer-based system for the identification of idioms and a manually-curated evaluation benchmark. |
| Outcome: | The proposed system performs well in 10 languages and is released on github. |
ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Recent work in language modeling has led to effective SLMs with impressive performance levels across various benchmarks. |
| Approach: | They propose a benchmark that introduces process-level evaluation for commonsense reasoning tasks. |
| Outcome: | The proposed benchmarks show that large language models provide correct answers despite flawed reasoning processes in a substantial portion of cases. |
What’s the Meaning of Superhuman Performance in Today’s NLU? (2023.acl-long)
Copied to clipboard
Simone Tedeschi, Johan Bos, Thierry Declerck, Jan Hajič, Daniel Hershcovich, Eduard Hovy, Alexander Koller, Simon Krek, Steven Schockaert, Rico Sennrich, Ekaterina Shutova, Roberto Navigli
| Challenge: | Recent research has focused on developing larger pretrained language models and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities. |
| Approach: | They propose to use benchmarks such as SuperGLUE and SQUAD to evaluate PLMs' abilities in language understanding, reasoning, and reading comprehension to assess their performance. |
| Outcome: | The proposed benchmarks have serious limitations affecting comparison between humans and PLMs and provide recommendations for fairer and more transparent benchmarks. |
ConSeC: Word Sense Disambiguation as Continuous Sense Comprehension (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems for word Sense Disambiguation assume that each word can be disambiguated individually . a novel approach to WSD is proposed to address this limitation . |
| Approach: | They propose a supervised semantics-based approach to Word Sense Disambiguation that takes into account the senses assigned to nearby words. |
| Outcome: | The proposed approach surpasses all its competitors and sets a new state of the art on English WSD. |
ExtEnD: Extractive Entity Disambiguation (2022.acl-long)
Copied to clipboard
| Challenge: | Entity disambiguation (ED) is a task in natural language processing that requires a large pre-trained language model to perform. |
| Approach: | They propose a local formulation for Entity Disambiguation (ED) that frames this task as a text extraction problem and propose two Transformer-based architectures that implement it. |
| Outcome: | The proposed model outperforms all its competitors in terms of data efficiency and raw performance on 4 out of 4 benchmarks. |
Unifying Cross-Lingual Semantic Role Labeling with Heterogeneous Linguistic Resources (2021.naacl-main)
Copied to clipboard
| Challenge: | Using cross-lingual techniques to perform Semantic Role Labeling (SRL) has been limited by the fact that each language adopts its own linguistic formalism . |
| Approach: | They propose a unified model to perform cross-lingual SRL over heterogeneous linguistic resources. |
| Outcome: | The proposed model is able to annotate a sentence in a single forward pass with all the inventories it was trained with, providing a tool for the analysis and comparison of linguistic theories across different languages. |
WikiNEuRal: Combined Neural and Knowledge-based Silver Data Creation for Multilingual NER (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a key intermediate task in NLP. |
| Approach: | They propose a method which uses knowledge-based approaches and neural models to produce high-quality training corpora for NER. |
| Outcome: | The proposed method improves on standard benchmarks and yields significant improvements up to 6 span-based F1-score points over previous state-of-the-art systems for data creation. |
With More Contexts Comes Better Performance: Contextualized Sense Embeddings for All-Round Word Sense Disambiguation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Contextualized word embeddings have been used effectively across several tasks in Natural Language Processing, but it is difficult to link them to structured sources of knowledge. |
| Approach: | They propose a semi-supervised approach to producing sense embeddings for the lexical meanings within a lexicon that is comparable to that of contextualized word vectors. |
| Outcome: | The proposed approach outperforms state-of-the-art models in the English Word Sense Disambiguation task and in the multilingual one while training on sense-annotated data in English only. |
xCoRe: Cross-context Coreference Resolution (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current coreference resolution systems are limited to short-to-medium-sized documents and struggle to scale to very long documents due to architectural limitations and implied memory costs. |
| Approach: | They propose a unified approach to coreference resolution that unifies two challenging settings . they use a pipeline that first identifies mentions, then creates clusters within individual contexts . |
| Outcome: | The proposed model achieves state-of-the-art results on cross-document benchmarks and strong performance on long-document data while retaining top-tier results on traditional datasets. |
Probing for Predicate Argument Structures in Pretrained Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Recent proposed approaches have achieved impressive results in dependency- and span-based, multilingual and cross-lingual Semantic Role Labeling (SRL) |
| Approach: | They propose to probe for predicate argument structures in pretrained language models . they show that PLMs encode semantic structures directly into contextualized representations . |
| Outcome: | The proposed models have achieved impressive results in dependency- and span-based, multilingual and cross-lingual Semantic Role Labeling (SRL) |
RAED: Retrieval-Augmented Entity Description Generation for Emerging Entity Linking and Disambiguation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Entity Linking and Entity Disambiguation systems assume static knowledge bases are incomplete and up-to-date, rendering them incapable of handling entities not yet included in the knowledge base. |
| Approach: | They propose a model that retrieves external knowledge to improve factual grounding in entity descriptions. |
| Outcome: | The proposed model outperforms systems that require fixed knowledge sets on Entity Disambiguation and Wikipedia to improve factual grounding in entity descriptions. |
IR like a SIR: Sense-enhanced Information Retrieval for Multiple Languages (2021.emnlp-main)
Copied to clipboard
Rexhina Blloshmi, Tommaso Pasini, Niccolò Campolungo, Somnath Banerjee, Roberto Navigli, Gabriella Pasi
| Challenge: | Recent advances in contextualized embeddings have made ranking on non-English documents cumbersome . a novel multilingual query expansion mechanism provides sense definitions as additional semantic information for the query. |
| Approach: | They propose a multilingual query expansion mechanism that leverages word sense information to enhance the model's performance. |
| Outcome: | The proposed model performs better than its supervised and unsupervised alternatives across languages while being trained on English Robust04 data. |
REBEL: Relation Extraction By End-to-end Language generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to extract relation triplets from text often involve multiple-step pipelines that propagate errors or are limited to a small number of relation types. |
| Approach: | They propose to use autoregressive seq2seq models to simplify Relation Extraction by expressing triplets as a sequence of text and a model that performs end-to-end relation extraction for more than 200 different relation types. |
| Outcome: | The proposed model achieves state-of-the-art on an array of Relation Extraction and Relation Classification benchmarks and achieves top performance in most of them. |
VerbAtlas: a Novel Large-Scale Verbal Semantic Resource and Its Application to Semantic Role Labeling (D19-1)
Copied to clipboard
| Challenge: | VerbAtlas is a lexical-semantic resource that combines WordNet synsets into semantically-coherent frames. |
| Approach: | They propose a lexical-semantic resource that brings together WordNet synsets into semantically-coherent frames. |
| Outcome: | The proposed resource brings together all WordNet synsets into semantically-coherent frames. |
Do Large Language Models Understand Word Senses? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have set new performance standards in a wide range of tasks. |
| Approach: | They evaluate the Word Sense Disambiguation capabilities of instruction-tuned LLMs and their ability to understand word senses in three generative settings: definition generation, free-form explanation, and example generation. |
| Outcome: | The proposed models can explain the meaning of words in context with 98% accuracy, while demonstrating greater robustness across domains and levels of difficulty. |
Cross-lingual AMR Aligner: Paying Attention to Cross-Attention (2023.findings-acl)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) graphs embed the semantics of a sentence in a directed acyclic graph, where concepts are represented by nodes, semantic relations between concepts by edges, and the co-references by reentrant nodes. |
| Approach: | They propose a novel aligner for Abstract Meaning Representation graphs that scales cross-lingually and can align units and spans in sentences of different languages. |
| Outcome: | The proposed aligner achieves state-of-the-art in the benchmarks and can scale cross-lingually. |
MOSAICo: a Multilingual Open-text Semantically Annotated Interlinked Corpus (2024.naacl-long)
Copied to clipboard
Simone Conia, Edoardo Barba, Abelardo Carlos Martinez Lorenzo, Pere-Lluís Huguet Cabot, Riccardo Orlando, Luigi Procopio, Roberto Navigli
| Challenge: | Existing approaches to integrate semantics into Natural Language Understanding (NLP) systems are cost-effective and environmental impact-related. |
| Approach: | They propose to provide semantically-annotated corpora for four NLU tasks across five languages and to drop the requirement of closed datasets. |
| Outcome: | The proposed model provides hundreds of millions of silver yet high-quality annotations for four NLU tasks across five languages. |
Efficient AMR Parsing with CLAP: Compact Linearization with an Adaptable Parser (2024.lrec-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) parsers face efficiency challenges because of their large model size and computational time, which limit their accessibility within the research community. |
| Approach: | They propose a novel linearization system that simplifies encoding and reduces the number of tokens by between 40% and 50%. |
| Outcome: | The proposed system reduces the number of tokens by 40% and 50% while maintaining high performance while reducing training and inference times. |
Sense-Annotated Corpora for Word Sense Disambiguation in Multiple Languages and Domains (2020.lrec-1)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a field of NLP where data is usually tied to a specific language. |
| Approach: | They propose to release five large datasets annotated with word-senses in five different languages and 5 datasets in English for a different semantic domain. |
| Outcome: | The study shows that supervised models trained on the data achieve higher performance than those trained on other corpora. |
Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trends (2024.acl-long)
Copied to clipboard
| Challenge: | Large autoregressive generative models have emerged as the cornerstone for achieving the highest performance across several Natural Language Processing tasks. |
| Approach: | They propose a pipeline that trains a state-of-the-art Coreference Resolution system within the constraints of an academic budget and trains with up to 0.006x the memory resources. |
| Outcome: | The proposed framework outperforms encoder-based discriminative systems on the CoNLL-2012 benchmark, training with up to 0.006x the memory resources and obtaining 170x faster inference compared to previous state-of-the-art systems. |
XL-AMR: Enabling Cross-Lingual AMR Parsing with Transfer Learning Techniques (2020.emnlp-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a popular formalism of natural language. |
| Approach: | They develop a cross-lingual AMR parser that can be trained on the produced data . they use transfer learning techniques to produce automatic AMR annotations across languages . |
| Outcome: | The proposed parser significantly surpasses those reported in Chinese, German, Italian and Spanish. |
Estimating Machine Translation Difficulty (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Despite the high-quality outputs, it is difficult to distinguish between state-of-the-art models and identify areas for future improvement. |
| Approach: | They propose a new metric to evaluate difficulty estimators and use it to assess both baselines and novel approaches. |
| Outcome: | The proposed models outperform both heuristic-based methods and LLM-as-a-judge approaches, with sentinel-src achieving the best performance. |
ZEBRA: Zero-Shot Example-Based Retrieval Augmentation for Commonsense Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current Large Language Models (LLMs) have shown strong reasoning capabilities in commonsense question answering benchmarks, but the process underlying their success remains largely opaque. |
| Approach: | They propose a zero-shot question answering framework that combines retrieval, case-based reasoning and introspection to improve the model's performance and interpretability. |
| Outcome: | The proposed framework outperforms existing LLMs and previous knowledge integration approaches in commonsense reasoning benchmarks and achieves an average accuracy improvement of 4.5 points. |
DiBiMT: A Novel Benchmark for Measuring Word Sense Disambiguation Biases in Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Lexical ambiguity poses one of the greatest challenges in the field of Machine Translation. |
| Approach: | They propose a new benchmark to study semantic biases in Machine Translation of nominal and verbal words in five different languages. |
| Outcome: | The proposed benchmark tests state-of-the-art machine translation systems against the new test bed and provides a statistical and linguistic analysis of the results. |
Concept-pedia: a Wide-coverage Semantically-annotated Multimodal Dataset (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current evaluations for Vision-language Models remain heavily anchored to ImageNet . |
| Approach: | They propose a large-scale semantically-annotated multimodal resource that extends the range of visual concepts, including diverse abstract categories. |
| Outcome: | The proposed model expands the range of visual concepts, including diverse abstract categories. |
FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in text summarization have shown remarkable performance, but a significant number of summaries exhibit factual inconsistencies, such as hallucinations. |
| Approach: | They propose a factuality-oriented metric that evaluates text summarization for accuracy . they use a human annotation process to examine the accuracy of automatically generated summaries . |
| Outcome: | The proposed metric sets a new state-of-the-art on AGGREFACT, the de-facto benchmark for factuality evaluation. |
DMLM: Descriptive Masked Language Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | Descriptive Masked Language Modeling (DMLM) is a knowledge-enhanced reading comprehension objective that requires the model to predict the most likely word in a context, being provided with the word’s definition. |
| Approach: | They propose a knowledge-enhanced reading comprehension objective where the model is required to predict the most likely word in a context, being provided with the word’s definition. |
| Outcome: | The proposed model improves on a number of well-established NLU benchmarks and other semantic-focused tasks, e.g., Semantic Role Labeling. |
Entity Disambiguation with Entity Definitions (2023.eacl-main)
Copied to clipboard
| Challenge: | Entity Disambiguation (ED) is a crucial problem in Natural Language Processing (NLP). |
| Approach: | They propose to use Wikipedia titles as the textual representation of each candidate to improve the generalization capability over unseen patterns. |
| Outcome: | The proposed model improves on 2 out of 6 benchmarks and is generalized over unseen patterns. |
Exploring Non-Verbal Predicates in Semantic Role Labeling: Challenges and Opportunities (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing systems for SRL are incapable of transferring knowledge across different predicate types. |
| Approach: | They propose a new PropBank dataset which boasts wide coverage of multiple predicate types and a manually-annotated challenge set which gives equal importance to verbal, nominal, and adjectival predicates. |
| Outcome: | The proposed dataset shows that standard benchmarks do not provide an accurate picture of the current situation in SRL and that state-of-the-art systems are still incapable of transferring knowledge across different predicate types. |
SyntagNet: Challenging Supervised Word Sense Disambiguation with Lexical-Semantic Combinations (D19-1)
Copied to clipboard
| Challenge: | Current research in knowledge-based Word Sense Disambiguation (WSD) indicates that performances depend heavily on the Lexical Knowledge Base (LKB) employed. |
| Approach: | They propose to use a Lexical Knowledge Base to capture syntagmatic relations to enable knowledge-based WSD systems to achieve a new state of the art. |
| Outcome: | The proposed resource captures syntagmatic relations and is the first large-scale manually-curated resource of this kind made available to the community. |
Fully-Semantic Parsing and Generation: the BabelNet Meaning Representation (2022.acl-long)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is the most popular formalism for Semantic Parsing. |
| Approach: | They propose a language-independent representation of meaning using BabelNet and VerbAtlas. |
| Outcome: | The proposed framework outperforms existing frameworks thanks to fully-semantic framing, the authors show . the proposed dataset is labeled entirely according to the proposed framework, and is available on github. |
Word Sense Linking: Disambiguating Outside the Sandbox (2024.findings-acl)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) systems have performed well on several evaluation benchmarks, but it still struggles to find downstream applications. |
| Approach: | They propose a task where systems have to identify which spans to disambiguate and link them to their most suitable meaning. |
| Outcome: | The proposed task performs above the estimated inter-annotator agreement on a set of words . the proposed system is based on 'transformer-based' architectures and iteratively relaxes the assumptions . |
UniteD-SRL: A Unified Dataset for Span- and Dependency-Based Multilingual and Cross-Lingual Semantic Role Labeling (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual and cross-lingual Semantic Role Labeling (SRL) has attracted increasing attention as multilingual text representation techniques have become more effective and widely available. |
| Approach: | They propose a benchmark for multilingual and cross-lingual, span- and dependency-based SRL that provides expert-curated parallel annotations using a common predicate-argument structure inventory. |
| Outcome: | The proposed benchmark provides expert-curated parallel annotations using a common predicate-argument structure inventory, allowing direct comparisons across languages and encouraging studies on cross-lingual transfer in SRL. |
Conception: Multilingually-Enhanced, Human-Readable Concept Vector Representations (2020.coling-main)
Copied to clipboard
| Challenge: | Word vector representations suffer from a monolingual bias due to the amount of data available across languages. |
| Approach: | They propose a technique for building language-independent vector representations of concepts which places multilinguality at its core while retaining explicit relationships between concepts. |
| Outcome: | The proposed representations outperform the state-of-the-art in multilingual and cross-lingual Semantic Word Similarity and Word Sense Disambiguation on low-resource languages. |
ReLiK: Retrieve and LinK, Fast and Accurate Entity Linking and Relation Extraction on an Academic Budget (2024.findings-acl)
Copied to clipboard
| Challenge: | Entity Linking and Relation Extraction (EL) are fundamental tasks in Natural Language Processing. |
| Approach: | They propose a Retriever-Reader architecture for Entity Linking and Relation Extraction . they propose an input representation that incorporates the candidate entities alongside the text . |
| Outcome: | The proposed architecture achieves state-of-the-art in in- and out-of domain benchmarks while using academic budget training and with 40x inference speed compared to competitors. |
Multi-LMentry: Can Multilingual LLMs Solve Elementary Tasks Across Languages? (2025.emnlp-main)
Copied to clipboard
Luca Moroni, Javier Aula-Blasco, Simone Conia, Irene Baucells, Naiara Perez, Silvia Paniagua Suárez, Anna Sallés, Malte Ostendorff, Júlia Falcão, Guijin Son, Aitor Gonzalez-Agirre, Roberto Navigli, Marta Villegas
| Challenge: | a recent study focused on complex, high-level tasks, but LMentry is limited to English . a multilingual evaluation of large language models is needed to address this gap, authors say . |
| Approach: | They propose a compact benchmark that enables systematic evaluation of large language models . they propose to use tasks that are trivial for humans but remain surprisingly difficult for LLMs . |
| Outcome: | The proposed benchmark is limited to English, leaving its insights linguistically narrow. |
Universal Semantic Annotator: the First Unified API for WSD, SRL and Semantic Parsing (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing approaches to understanding textual information are still far from achieving true natural language understanding (NLU). |
| Approach: | They propose a unified API for high-quality automatic annotations of texts in 100 languages through state-of-the-art systems for Word Sense Disambiguation, Semantic Role Labeling and Semantics Parsing. |
| Outcome: | The proposed system can provide users with rich and diverse semantic information, help second-language learners, and integrate explicit semantic knowledge into downstream tasks and real-world applications. |
Mitigating Data Scarcity in Semantic Parsing across Languages with the Multilingual Semantic Layer and its Dataset (2024.findings-acl)
Copied to clipboard
Abelardo Carlos Martinez Lorenzo, Pere-Lluís Huguet Cabot, Karim Ghonim, Lu Xu, Hee-Soo Choi, Alberte Fernández-Castro, Roberto Navigli
| Challenge: | Large Language Models (LLMs) have advanced significantly in understanding human text, but semantic representations remain crucial for various applications. |
| Approach: | They introduce a multilingual semantic layer which decouples from disambiguation and external inventories and simplifies the task. |
| Outcome: | The proposed model reduces performance gap between languages and annotators by enabling them to understand semantic relations between concepts in any language. |
InVeRo-XL: Making Cross-Lingual Semantic Role Labeling Accessible with Intelligible Verbs and Roles (2021.emnlp-demo)
Copied to clipboard
| Challenge: | InVeRo-XL is an off-the-shelf system capable of annotating text with predicate sense and semantic role labels from 7 predicated-argument structure inventories in more than 40 languages. |
| Approach: | They propose to use RESTful API and Web interface to integrate sentence-level semantics into cross-lingual downstream tasks. |
| Outcome: | The proposed system can annotate text with predicate sense and semantic role labels from 7 predicated-argument structure inventories in more than 40 languages. |
Named Entity Recognition for Entity Linking: What Works and What’s Next (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Entity Linking (EL) systems have achieved impressive results on standard benchmarks thanks to the contextualized representations provided by recent pretrained language models. |
| Approach: | They propose to exploit Named Entity Recognition (NER) to narrow the gap between EL systems trained on high and low amounts of labeled data. |
| Outcome: | The proposed model can be exploited to narrow the gap between EL systems trained on high and low amounts of labeled data. |
Semantic Role Labeling Meets Definition Modeling: Using Natural Language to Describe Predicate-Argument Structures (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to Semantic Role Labeling rely on discrete labels to classify predicate senses and their arguments. |
| Approach: | They propose a generalized formulation of Semantic Role Labeling that leverages Definition Modeling to describe predicate-argument structures using natural language definitions instead of discrete labels. |
| Outcome: | The proposed model can describe predicate-argument structures using natural language definitions instead of discrete labels. |
Game Theory Meets Embeddings: a Unified Framework for Word Sense Disambiguation (D19-1)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is an open problem in Natural Language Processing (NLP). |
| Approach: | They propose a game-theoretic model that embeds ambiguous words as players of a non cooperative game and their senses as strategies that the players can select in order to play the games. |
| Outcome: | The proposed model performs well on standard benchmarks and different tests on standard datasets. |
AMuSE-WSD: An All-in-one Multilingual System for Easy Word Sense Disambiguation (2021.emnlp-demo)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a task of associating a word in context with its most appropriate sense from a predefined sense inventory. |
| Approach: | They propose to use a state-of-the-art neural model to integrate WSD into real-world applications. |
| Outcome: | The proposed system offers high-quality sense information in 40 languages through a state-of-the-art neural model for WSD. |
Interpretable Coreference Resolution Evaluation Using Explicit Semantics (2026.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods for coreference resolution are limited by semantic and contextual information. |
| Approach: | They propose a semantically-enhanced evaluation framework for coreference resolution that assigns semantic labels to nominal mentions and propagates them to entire coreference clusters. |
| Outcome: | The proposed framework uncovers systematic weaknesses obscured by standard metrics. |
Reducing Disambiguation Biases in NMT by Leveraging Explicit Word Sense Information (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that Neural Machine Translation models struggle to disambiguate polysemous words without lapsing into their most frequent senses. |
| Approach: | They propose a way to automatically create high-precision sense-annotated parallel corpora . they then propose 'fine-tuning' strategies to exploit these sense annotations during training . |
| Outcome: | The proposed approach achieves higher BLEU scores than its vanilla counterpart in 3 language pairs. |
Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering (2025.findings-acl)
Copied to clipboard
| Challenge: | Multiple-choice question answering tasks are one of the most commonly used tasks for evaluating Large Language Models (LLMs). |
| Approach: | They analyze whether existing answer extraction methods are aligned with human judgment and how they are influenced by answer constraints in the prompt across different domains. |
| Outcome: | The proposed evaluation strategies can be inconsistent with human judgment, and can lead to inaccurate and misleading comparisons. |
LSTMEmbed: Learning Word and Sense Representations from a Large Semantically Annotated Corpus with Long Short-Term Memories (P19-1)
Copied to clipboard
| Challenge: | Recent work has focused on vector representations which capture different meanings, i.e., senses, of words. |
| Approach: | They propose a bidirectional LSTM model which learns word senses from semantically annotated corpora by focusing on word order. |
| Outcome: | The proposed model achieves state-of-the-art on the SemEval-2014 word-to-sense similarity task and is available online at http://lcl.uniroma1.it/LSTMEmbed. |
Just “OneSeC” for Producing Multilingual Sense-Annotated Data (P19-1)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is one of the most affected research areas . annotated data are scarce in English and almost absent in other languages . |
| Approach: | They propose a language-independent method for the automatic extraction of thousands of sentences in which a target word is tagged with its meaning. |
| Outcome: | The proposed method outperforms existing methods on multilingual and domain-specific settings. |
REDFM: a Filtered and Multilingual Relation Extraction Dataset (2023.acl-long)
Copied to clipboard
| Challenge: | Existing Relation Extraction models rely on small datasets with low coverage of relation types . current systems rely only on small data sets with limited coverage of relationship types - especially when working with languages other than english. |
| Approach: | They propose to use an automatic annotated dataset to train relation extraction systems. |
| Outcome: | The proposed model can extract triplets in multiple languages from a human-revised dataset. |
Echoes from Alexandria: A Large Resource for Multilingual Book Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent research in text summarization has focused on news stories, where texts are typically short and have strong layout features. |
| Approach: | They propose a resource for multilingual book summarization that uses a new extractive-then-abstractive baseline to compare the results. |
| Outcome: | The proposed resource is the largest and first to be multilingual, featuring 5 languages and 25 language pairs. |
LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing Question Answering systems are limited by noisy documents and flawed QA pairs. |
| Approach: | They propose a high-quality subset of NarrativeQA focused on literary works . they identify and correct low-quality QA samples while removing extraneous text . |
| Outcome: | The proposed subset of NarrativeQA is based on literary works. |
Integrating Personalized PageRank into Neural Word Sense Disambiguation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Neural Word Sense Disambiguation (WSD) uses pre-existing knowledge, but only close neighbors influence prediction. |
| Approach: | They propose to exploit WordNet graphs to improve a classification model by recomputing logits . they incorporate an online neural approximated PageRank to refine edge weights . |
| Outcome: | The proposed method improves the current state of the art in the field of Neural Word Sense Disambiguation (WSD) the proposed method exploits the global graph structure while keeping space requirements linear in the number of edges. |
MultiNERD: A Multilingual, Multi-Genre and Fine-Grained Dataset for Named Entity Recognition (and Disambiguation) (2022.findings-naacl)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a process of identifying named entities in unstructured texts and classifying them through specific semantic categories. |
| Approach: | They propose a method for automatically producing NER annotations and introduce a manually-annotated test set. |
| Outcome: | The proposed method covers 10 languages, 15 NER categories and 2 textual genres and a manually-annotated test set. |
Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation (2025.findings-naacl)
Copied to clipboard
Luca Moroni, Giovanni Puccetti, Pere-Lluís Huguet Cabot, Andrei Stefan Bejgu, Alessio Miaschi, Edoardo Barba, Felice Dell’Orletta, Andrea Esuli, Roberto Navigli
| Challenge: | Pretrained Large Language Models (LLMs) are mainly designed for the English language, but are not optimized for non-English languages due to language contamination or multilingual pretraining data. |
| Approach: | They propose a method that leverages neural mapping for vocabulary substitution to optimize LLMs for the Italian language. |
| Outcome: | The proposed method reduces token fertility by 25% and improves grounded alignment strategies. |
InVeRo: Making Semantic Role Labeling Accessible with Intelligible Verbs and Roles (2020.emnlp-demos)
Copied to clipboard
| Challenge: | Semantic Role Labeling (SRL) is dependent on complex linguistic resources and sophisticated neural models, which makes the task difficult to approach for non-experts. |
| Approach: | They propose a platform for semantic role labeling that provides verb sense and semantic role information with an easy to use Web interface and RESTful APIs. |
| Outcome: | The proposed system provides human-readable verb sense and semantic role information with an easy to use Web interface and RESTful APIs. |
Bridging the Gap in Multilingual Semantic Role Labeling: a Language-Agnostic Approach (2020.coling-main)
Copied to clipboard
| Challenge: | Recent research indicates that taking advantage of complex syntactic features leads to favorable results in Semantic Role Labeling. |
| Approach: | They propose a language-agnostic model that does away with morphological and syntactic features to achieve robustness across languages. |
| Outcome: | The proposed model outperforms the state-of-the-art in all languages of the CoNLL-2009 benchmark dataset. |
Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress (2025.acl-short)
Copied to clipboard
| Challenge: | In machine translation evaluation, metric performance is assessed based on agreement with human judgments. |
| Approach: | They incorporate human baselines into the MT meta-evaluation to gain a clearer understanding of metric performance and establish an upper bound. |
| Outcome: | The results suggest human parity, but there are several reasons to caution . |
SPRING Goes Online: End-to-End AMR Parsing and Generation (2021.emnlp-demo)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a formalism for representing the semantics of natural language in a readable and hierarchical way. |
| Approach: | They present SPRING Online Services, a Web interface and RESTful APIs for their AMR parsing and generation system, SPRING (Symmetric PaRsIng aNd Generation). |
| Outcome: | The proposed system provides a highly interactive visualization platform and feedback mechanism to obtain user suggestions for further improvements of the system’s output. |
Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards (2026.acl-long)
Copied to clipboard
| Challenge: | Existing PRM datasets are expensive to construct and limited to the mathematical domain. |
| Approach: | They propose a method to generate a corpus of one million reasoning steps using the Planning Domain Definition Language. |
| Outcome: | The proposed model generates a corpus of approximately one million reasoning steps across various PDDL domains and trains them. |
Framing Word Sense Disambiguation as a Multi-Label Problem for Model-Agnostic Knowledge Integration (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent studies treat Word Sense Disambiguation (WSD) as a single-label classification problem, but gold data suggests that maximizing the probability of a singular sense is not the most suitable training objective for WSD. |
| Approach: | They propose to use Word Sense Disambiguation (WSD) as a multi-label classification problem in which multiple senses can be assigned to each target word. |
| Outcome: | The proposed method bears closer resemblance to how human annotators disambiguate text and can be extended to exploit structured knowledge from semantic networks. |