ID10M-JAM: Stress-Testing Idiom Identification Under Challenging Context (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) achieve strong performance on idiom identification benchmarks, yet their robustness to misleading contextual signals remains largely untested. |
| Approach: | They propose an adversarial extension of the ID10M dataset that jams idiom understanding by injecting coherent but conflicting context before each target sentence. |
| Outcome: | The proposed benchmark exposes systematic vulnerabilities in LLMs’ contextual reasoning, pushing idiom identification to its breaking point. |
Similar Papers
ID10M: Idiom Identification in 10 Languages (2022.findings-naacl)
Copied to clipboard
| Challenge: | Identifying and understanding idioms in context is a key goal and challenge in Natural Language Understanding tasks. |
| Approach: | They propose a multilingual Transformer-based system for the identification of idioms and a manually-curated evaluation benchmark. |
| Outcome: | The proposed system performs well in 10 languages and is released on github. |
Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context (2025.acl-long)
Copied to clipboard
| Challenge: | Existing models fail to resolve idiomaticity when it depends on contextual understanding . idiom frequency influences performance but does not guarantee accurate interpretation. |
| Approach: | They propose a novel contrastive dataset to assess whether large language models can effectively leverage context to disambiguate idiomatic meanings. |
| Outcome: | The proposed model performs better on sentences deemed more likely by the model . collocational frequency and sentence probability influence performance but not accuracy . |
No Context Needed: Contextual Quandary In Idiomatic Reasoning With Pre-Trained Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | idiomatic expressions (IEs) are a non-compositional aspect of a text that makes it difficult for a model to comprehend . general purpose PTLMs are negatively affected by the context, as performance increases with its removal. |
| Approach: | They propose to use idiomatic expressions to infer additional meaning from IEs . they argue that only IE-aware models are suitable for idiom- matic reasoning tasks . |
| Outcome: | The proposed models can reason in the presence of idiomatic expressions, the authors show . they show that general purpose PTLMs are negatively affected by the context . |
Easy as PIE? Identifying Multi-Word Expressions with LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multiword expressions (MWEs) are a semantically non-compositional subclass of multiword expression . authors show that prompt-based LLMs can perform competitively with supervised models . |
| Approach: | They propose a prompt-based approach to identify idiomatic expressions in running text . they find prompt-driven LLMs can perform competitively with supervised models . |
| Outcome: | The proposed approach can perform well with supervised models on annotated data. |
Getting BART to Ride the Idiomatic Train: Learning to Represent Idiomatic Expressions (2022.tacl-1)
Copied to clipboard
| Challenge: | Prior work has identified deficiencies in their contextualized representation stemming from the underlying compositional paradigm of representation. |
| Approach: | They propose to use an adapter as a lightweight non-compositional language expert trained on idiomatic sentences to build idiomity into BART. |
| Outcome: | The proposed approach improves idiomaticity over baselines and up to 25% higher sequence accuracy on idiom processing tasks. |
Deconstructing In-Context Learning: Understanding Prompts via Corruption (2024.lrec-main)
Copied to clipboard
| Challenge: | Prior work examined how modifying different elements of the prompt can affect model performance, but this limited number of elements made replication challenging. |
| Approach: | They decompose the entire prompt into four components: task description, demonstration inputs, labels, and inline instructions provided for each demonstration. |
| Outcome: | The proposed model is robust to minor prompt modifications, but its underlying pre-trained backbone is brittle . previous studies focused on models with fewer than 15 billion parameters or exclusively examined black-box models like GPT-3 or PaLM, making replication challenging. |
AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing datasets are limited to providing the degree of idiomaticity of expressions along with the literal and, where applicable, (a single) non-literal interpretation of MWEs. |
| Approach: | They propose to use a dataset to test the effectiveness of a language model in generating representations of sentences containing idioms. |
| Outcome: | The proposed model performs reasonably well on the one-shot and few-shot scenarios, but there is scope for improvement in the zero-shot scenario. |
Memorization or Reasoning? Exploring the Idiom Understanding of LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | idioms have long posed a challenge due to their unique linguistic properties, which set them apart from other common expressions. |
| Approach: | They propose to use a large-scale dataset of idioms in six languages to evaluate LLMs' idiomatic processing ability. |
| Outcome: | The proposed model integrates contextual cues and reasoning to improve idiom understanding in LLMs, suggesting that their performance is influenced by memorization and reasoning. |
Idiomatic Expression Identification using Semantic Compatibility (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to localize idiomatic expressions have limited views of their generalizability to new idioms. |
| Approach: | They propose a multi-stage neural architecture to detect whether a sentence has an idiomatic expression and localize it when it occurs in a figurative sense. |
| Outcome: | The proposed model achieves state-of-the-art on three of the largest datasets with idiomatic expressions of varied syntactic patterns and degrees of non-compositionality. |
Out-of-Context Reasoning in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a lightweight technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
| Approach: | They propose a lightweight technique that trains only new token embeddings on axioms . they train only new embeddables and evaluate them on unseen tasks . |
| Outcome: | The proposed technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |