AnaDE1.0: A Novel Data Set for Benchmarking Analogy Detection and Extraction (2024.eacl-long)
Copied to clipboard
| Challenge: | Textual analogies that make comparisons between two concepts are often used for explaining complex ideas, creative writing, and scientific discovery. |
| Approach: | They propose a task that includes three synergistic tasks: detecting documents containing analogies, extracting text segments that make up the analogy, and identifying the (source and target) concepts being compared. |
| Outcome: | The proposed task performs well on all sub-tasks and smaller models perform better than non-finetuned ChatGPT, suggesting high task difficulty. |
Similar Papers
AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies (2024.emnlp-main)
Copied to clipboard
Xiao Ye, Andrew Wang, Jacob Choi, Yining Lu, Shreya Sharma, Lingfeng Shen, Vijay Murari Tiyyala, Nicholas Andrews, Daniel Khashabi
| Challenge: | Analogical reasoning is an important part of human communication, says a new study . a benchmark to determine analogical reasoning ability in language models is needed . |
| Approach: | They propose to benchmark analogical reasoning ability in language models by collecting 340 analogies from human writings. |
| Outcome: | The proposed benchmark aims to determine analogical reasoning ability in language models. |
ANALOGICAL - A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models (2023.findings-acl)
Copied to clipboard
Thilini Wijesiriwardene, Ruwan Wickramarachchi, Bimal Gajera, Shreeyash Gowaikar, Chandan Gupta, Aman Chadha, Aishwarya Naresh Reganti, Amit Sheth, Amitava Das
| Challenge: | Modern large language models are evaluated on extrinsic measures based on benchmarks such as GLUE and SuperGLUE. |
| Approach: | They propose a benchmark to intrinsically evaluate large language models across a taxonomy of analogies of long text with six levels of complexity. |
| Outcome: | The proposed benchmark evaluates LLMs across a taxonomy of analogies of long text with six levels of complexity. |
Automatic Extraction of Metaphoric Analogies from Literary Texts: Task Formulation, Dataset Construction, and Evaluation (2025.coling-main)
Copied to clipboard
Joanne Boisson, Zara Siddique, Hsuvas Borkakoty, Dimosthenis Antypas, Luis Espinosa Anke, Jose Camacho-Collados
| Challenge: | Recent advances in large language models (LLMs) have shown to be difficult to extract metaphors from free text because they can involve some implicit concepts and link dissimilar concepts. |
| Approach: | They compare the ability of large language models to extract metaphors from literary texts using domain experts. |
| Outcome: | The proposed models can extract metaphors from literary texts without using domain experts. |
Multilingual Culture-Independent Word Analogy Datasets (2020.lrec-1)
Copied to clipboard
| Challenge: | In text processing, deep neural networks use word embeddings as an input. |
| Approach: | They propose to use benchmark datasets to compare the quality of word embeddings in text processing . they use a word analogy task in Croatian, English, Estonian, Finnish, Latvian, Lithuanian, Russian, Slovenian, and Swedish . |
| Outcome: | The proposed datasets are culturally independent and cross-lingual for the languages used. |
ParallelPARC: A Scalable Pipeline for Generating Natural-Language Analogies (2024.naacl-long)
Copied to clipboard
| Challenge: | Analogy-making is a central to human cognition, allowing us to abstract information and understand novel situations in terms of familiar ones. |
| Approach: | They propose a pipeline to generate paragraph-based analogies using large language models and large language distractors. |
| Outcome: | The proposed pipeline outperforms existing models in binary and multiple-choice settings and shows that humans outperformed the best models after a light supervision. |
Interpretable Text Embeddings and Text Similarity Explanation: A Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text embeddings are a fundamental component in many NLP tasks, but their interpretation and explanation remain challenging. |
| Approach: | They propose a framework for interpretable text embeddings and text similarity explanation . they characterize the main ideas, approaches, and trade-offs and discuss lessons learned . |
| Outcome: | The proposed methods are compared with existing models and compare them with existing ones. |
MGAD: Multilingual Generation of Analogy Datasets (L18-1)
Copied to clipboard
| Challenge: | Existing methods for word embedding evaluation are computationally expensive and task-specific. |
| Approach: | They propose a minimally supervised method for generating word embedding evaluation datasets for a large number of languages using existing dependency treebanks and parsers. |
| Outcome: | The proposed method evaluates three popular word embedding algorithms against these datasets and shows that their performance varies between syntactic categories. |
AnaScore: Understanding Semantic Parallelism in Proportional Analogies (2025.naacl-long)
Copied to clipboard
| Challenge: | AnaScore metric aims to evaluate the strength of semantic parallelism in sentence analogies. |
| Approach: | They propose an automatic metric to evaluate the strength of semantic parallelism in sentence analogies. |
| Outcome: | The proposed metric shows that formally explainable examples are more beneficial for analogical reasoning, whereas ambiguous analogies with no clear criterion tend to hinder inference. |
Scientific and Creative Analogies in Pretrained Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing analogy datasets focus on a limited set of analogical relations with a high similarity of the two domains between which the analogy holds. |
| Approach: | They propose a dataset that encodes analogy in pretrained language models . they use a system that maps attributes and relational structures across dissimilar domains . |
| Outcome: | The proposed dataset shows that state-of-the-art models achieve low performance on analogy tasks . |
StoryAnalogy: Deriving Story-level Analogies from Large Language Models to Unlock Analogical Understanding (2023.emnlp-main)
Copied to clipboard
Cheng Jiayang, Lin Qiu, Tsz Chan, Tianqing Fang, Weiqi Wang, Chunkit Chan, Dongyu Ru, Qipeng Guo, Hongming Zhang, Yangqiu Song, Yue Zhang, Zheng Zhang
| Challenge: | Analogy-making between narratives is crucial for human reasoning . despite its importance, there has been limited research on story analogies . |
| Approach: | They construct a large-scale story-level analogy corpus with 24K story pairs . they find that the tasks are incredibly difficult for large language models such as ChatGPT . |
| Outcome: | The proposed corpus contains 24K story pairs from diverse domains with human annotations on two similarities from the extended Structure-Mapping Theory. |