Drivel-ology: Challenging LLMs with Interpreting Nonsense with Depth (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite excelling at many natural language processing tasks, large language models fail to grasp the layered semantics of Drivelological text. |
| Approach: | They construct a benchmark dataset of over 1,200+ carefully curated and diverse examples across English, Mandarin, Spanish, French, Japanese, and Korean to examine their Drivelological characteristics. |
| Outcome: | The proposed models lack conceptual understanding and lack conceptual and semantic accuracy. |
Similar Papers
Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers? (2025.coling-main)
Copied to clipboard
Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, Yongfeng Zhang
| Challenge: | Large language models have shown remarkable performances across a wide range of tasks, but mechanisms by which they encode tasks of varying complexity remain poorly understood. |
| Approach: | They propose to explore the possibility that LLMs process concepts in different layers . they propose to categorize concepts based on their level of abstraction . |
| Outcome: | The proposed model can process complex concepts in shallow layers, the authors show . the proposed model could be used to prob complex tasks in shallow ones . |
Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current Large Language Models (LLMs) are unparalleled in their ability to generate grammatically correct, fluent text. |
| Approach: | They argue that LLMs only parrot statistical patterns in training data and that language learning in LLM cannot inform human language learning. |
| Outcome: | The proposed model can generate grammatically correct, fluent text without requiring human intervention. |
Metaphor and Large Language Models: When Surface Features Matter More than Deep Understanding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on metaphor processing have focused on single datasets and specific task settings, often using artificially constructed data through lexical replacement. |
| Approach: | They propose to evaluate the capabilities of Large Language Models (LLMs) in metaphor interpretation across multiple datasets, tasks, and prompt configurations. |
| Outcome: | The proposed frameworks are more realistic and efficient than current models and are more efficient than existing models. |
Measuring Psychological Depth in Language Models (2024.emnlp-main)
Copied to clipboard
Fabrice Harel-Canada, Hanyu Zhou, Sreya Muppalla, Zeynep Yildiz, Miryung Kim, Amit Sahai, Nanyun Peng
| Challenge: | Current evaluations of creative stories focus on objective properties of the text, such as its style, coherence, diversity, and creativity. |
| Approach: | They propose a framework that measures an LLM's ability to produce authentic and narratively complex stories that provoke emotion, empathy, and engagement. |
| Outcome: | The proposed framework shows that humans can consistently evaluate stories based on the PDS (0.72 Krippendorff’s alpha). |
The Model Agreed, But Didn’t Learn: Diagnosing Surface Compliance in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models internalize vast world knowledge as parametric memory, yet inherit the staleness and errors of their source corpora. |
| Approach: | They propose a framework that subjects models to discriminative self-assessment under diverse contextual pressures to scrutinize subtle behavioral nuances induced by memory modifications. |
| Outcome: | The proposed framework achieves high benchmarks without overwriting internal beliefs, while recursive modifications accumulate representational residues, triggering cognitive instability and permanently diminishing the reversibility of the model’s memory state. |
LingGym: How Far Are LLMs from Thinking Like Field Linguists? (2025.emnlp-main)
Copied to clipboard
| Challenge: | LINGGYM is a benchmark that evaluates LLMs’ capacity for meta-linguistic reasoning using Interlinear Glossed Text and grammatical descriptions extracted from 18 typologically diverse reference grammars. |
| Approach: | They propose a benchmark that evaluates LLMs’ capacity for meta-linguistic reasoning using Interlinear Glossed Text and grammatical descriptions extracted from 18 typologically diverse reference grammars. |
| Outcome: | The proposed model can generalize linguistic inference across low-resource languages and structures not seen during training. |
DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | Retrievalaugmented LLMs have been used to ground LLM in external knowledge . a gap exists in the current landscape regarding the effectiveness of grounding LLM on heterogeneous knowledge sources. |
| Approach: | They propose a model that uses symbolic language to generate symbolic queries . they use a dataset that is generated using predefined reasoning chains and human annotation . |
| Outcome: | The proposed model outperforms previous approaches by a significant margin in QA tasks over text. |
Under the Shadow of Babel: How Language Shapes Reasoning in LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | linguistic relativity suggests that the structure of language shapes cognitive patterns . large language models internalize the habitual logical structures embedded in different languages, authors say . |
| Approach: | a study introduces a bilingual dataset for causal reasoning in Chinese and English. |
| Outcome: | a new study shows that large language models internalize reasoning biases shaped by language . the model internalizes language-specific preferences and rigidly applies them to atypical inputs, the study shows . |
Factuality of Large Language Models: A Survey (2024.emnlp-main)
Copied to clipboard
Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Das, Preslav Nakov
| Challenge: | Large language models (LLMs) are factually incorrect, which limits their applicability in real-world scenarios. |
| Approach: | They analyze existing work to identify major challenges and their associated causes . they propose to evaluate LLMs using a variety of measures to mitigate factual errors . |
| Outcome: | The proposed methods are based on a variety of datasets and proposed strategies to mitigate factual errors. |
Logic Haystacks: Probing LLMs’ Long-Context Logical Reasoning (Without Easily Identifiable Unrelated Padding) (2026.eacl-short)
Copied to clipboard
| Challenge: | Recent large language models claim long context windows, but evaluations often involve simple retrieval tasks or synthetic tasks padded with irrelevant text. |
| Approach: | They use grammars to generate simplified English with logical representations to create long input text while controlling its semantics. |
| Outcome: | The proposed model performs better with realistic distractors than with standard models. |