Understanding Figurative Meaning through Explainable Visual Entailment (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing models for visual entailment and visual question-answering have limited ability to understand figurative meaning in images and captions. |
| Approach: | They propose a task framing the figurative meaning understanding problem as an explainable visual entailment task where the model has to predict whether the image entitles a caption and justify the predicted label with a textual explanation. |
| Outcome: | The proposed dataset contains 6,027 image, caption, label, explanation instances covering five diverse figurative phenomena. |
Similar Papers
Looking Beyond the Pixels: Evaluating Visual Metaphor Understanding in VLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Visual metaphors are a complex vision–language phenomenon that requires both perceptual and conceptual reasoning to understand. |
| Approach: | They introduce a visual metaphor dataset featuring 2177 synthetic and 350 human-annotated images and benchmark several SOTA VLMs on two tasks: Visual Metaphor Captioning (VMC) and Visual Metamorphosis VQA (VM-VQA). |
| Outcome: | The proposed model outperforms standard few-shot baselines on visual metaphors and VM-VQA tasks. |
FLUTE: Figurative Language Understanding through Textual Explanations (2022.emnlp-main)
Copied to clipboard
| Challenge: | Figurative language understanding is a recognizing textual entailment task, but lacks data for figurative language. |
| Approach: | They propose to use a dataset to analyze figurative NLI instances with explanations to improve models' performance. |
| Outcome: | The proposed dataset can scale up models even for figurative language using human annotations. |
Figurative Language in Recognizing Textual Entailment (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing RTE models struggle to capture figurative language, despite its ubiquity, it remains a bottleneck in automatic text understanding. |
| Approach: | They propose to frame five existing figurative language datasets into over 12,500 RTE examples. |
| Outcome: | The proposed models struggle to perform pragmatic inference and reasoning about world knowledge. |
IRFL: Image Recognition of Figurative Language (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Figures of speech are ubiquitous in many forms of discourse, allowing people to convey complex, abstract ideas and evoke emotion. |
| Approach: | They develop a dataset for multimodal figurative language understanding using human annotation and an automatic pipeline to generate a multimodal dataset. |
| Outcome: | The proposed dataset performs better than human vision and language models compared with a human dataset . |
Figurative Language Processing: A Linguistically Informed Feature Analysis of the Behavior of Language Models and Humans (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent years have witnessed a growing interest in investigating what Transformer-based language models (TLMs) actually learn from training data. |
| Approach: | They propose to use a black-box TLM and two intrinsically transparent white-box models to investigate the performance of figurative language models on sarcasm, similes, idioms, and metaphors. |
| Outcome: | The proposed models perform better than other models on figurative language classification tasks. |
Testing the Ability of Language Models to Interpret Figurative Language (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing work on figurative language has not been done on literal language models. |
| Approach: | They propose a Winograd-style task to evaluate figurative phrases with divergent meanings by interpreting paired figurativ phrases with a human input. |
| Outcome: | The proposed task outperforms state-of-the-art models on a nonliteral language understanding task in zero-shot settings. |
It’s not Rocket Science: Interpreting Figurative Language in Narratives (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing text representations by design rely on compositionality, while figurative language is often non-compositional. |
| Approach: | They propose to use a pre-trained language model to interpret figurative language types to adopt human strategies for interpreting figurativ language types: inferring meaning from context and relying on constituent words’ literal meanings. |
| Outcome: | The proposed models perform significantly worse than humans on discriminative and generative tasks, bridging the gap from human performance. |
Towards Explainable Evaluation of Language Models on the Semantic Similarity of Visual Concepts (2022.coling-1)
Copied to clipboard
Maria Lymperaiou, George Manoliadis, Orfeas Menis Mastromichalakis, Edmund G. Dervakos, Giorgos Stamou
| Challenge: | Recent advances in NLP research have focused on robustness and explainability issues of their evaluation strategies. |
| Approach: | They propose to use pre-trained transformers to evaluate semantic similarity for visual vocabularies . they propose to provide explainable metrics for understanding the quality of retrieved instances . |
| Outcome: | The proposed metrics highlight inabilities of widely used evaluation methods and highlight weaknesses in learned linguistic representations. |
Reasoning Beyond Literal: Cross-style Multimodal Reasoning for Figurative Language Understanding (2026.findings-eacl)
Copied to clipboard
| Challenge: | figurative language is essential for expressing intent, emotion, and perspective . figural language is often dependent on Styles Reasoning, causing incongruities between expressions . |
| Approach: | They propose a framework that induces reasoning capabilities to compact vision–language models . figurative language is essential in expressing intent, emotion, and perspective . |
| Outcome: | The proposed framework can interpret multimodal figurative language, provide transparent reasoning traces, and generalize across multiple figurativ styles. |
Grounded Textual Entailment (C18-1)
Copied to clipboard
Hoa Trong Vu, Claudio Greco, Aliia Erofeeva, Somayeh Jafaritazehjan, Guido Linders, Marc Tanti, Alberto Testoni, Raffaella Bernardi, Albert Gatt
| Challenge: | Existing models for entailment analysis are not performing well in visual information-based models. |
| Approach: | They propose to use a visual representation of the Textual Entailment task to compare visual-grounded models with a multimodal version of the SNLI dataset. |
| Outcome: | The proposed model performs better when there is an image of the “world” or “situation” . |