| Challenge: | a novel approach to detecting noun abstraction within a large language model is proposed . a first step towards the explainability of conceptual abstraction in LLMs is shown . |
| Approach: | They propose a method to detect noun abstraction within a large language model . they instantiate taxonomic relationships and analyze attention matrices produced by BERT . |
| Outcome: | The proposed approach can detect hypernymy in a large language model . the results are a first step towards the explainability of conceptual abstraction in LLMs . |
Similar Papers
Towards Concept-Aware Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Concepts play a pivotal role in various human cognitive functions, including reasoning and communication. |
| Approach: | They analyze how well contemporary large language models capture human concepts and their structure . they propose a method for pretraining LLMs using concepts and a simpler approach . |
| Outcome: | The proposed method matches human intuition and improves robustness of predictions. |
What Does Parameter-free Probing Really Uncover? (2024.acl-short)
Copied to clipboard
| Challenge: | Probing large language models (LLMs) has been criticized for using pre-defined label-laden target labels. |
| Approach: | They extend a parameter-free probing technique called perturbed masking applied to BERT to examine the relationship between UD and BERT. |
| Outcome: | The proposed method is compared to the UD formalism for English and shows that it lacks correlations with linguistic theory. |
LLMs meet Bloom’s Taxonomy: A Cognitive View on Large Language Model Evaluations (2025.coling-main)
Copied to clipboard
| Challenge: | Existing evaluation approaches for Large Language Models lack a structured approach that reflects the underlying cognitive abilities required for solving the tasks. |
| Approach: | They propose a hierarchical approach to evaluation of Large Language Models that leverages Bloom’s Taxonomy to identify how well they cover the levels of Bloom’ s taxonomies. |
| Outcome: | The proposed evaluation frameworks cover the Bloom’s Taxonomy, a hierarchical framework for categorizing cognitive skills, on the most widely used benchmarks. |
SCOP: Evaluating the Comprehension Process of Large Language Models from a Cognitive View (2025.acl-long)
Copied to clipboard
| Challenge: | despite the potential of large language models, it is difficult to fully count on them in real-world scenarios. |
| Approach: | They propose to examine how LLMs perform during the comprehension process from a cognitive perspective. |
| Outcome: | The proposed model analyzes how LLMs perform during the comprehension process from a cognitive perspective. |
Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflection (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to self-detection only retrospectively evaluate LLM-generated answers, leading to over-trust in incorrectly generated answers. |
| Approach: | They propose a self-detection paradigm that considers the comprehensive answer space beyond LLM-generated answers to mitigate the over-trust in LLM generated incorrect answers. |
| Outcome: | The proposed framework can be integrated with existing approaches for superior self-detection. |
Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers? (2025.coling-main)
Copied to clipboard
Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, Yongfeng Zhang
| Challenge: | Large language models have shown remarkable performances across a wide range of tasks, but mechanisms by which they encode tasks of varying complexity remain poorly understood. |
| Approach: | They propose to explore the possibility that LLMs process concepts in different layers . they propose to categorize concepts based on their level of abstraction . |
| Outcome: | The proposed model can process complex concepts in shallow layers, the authors show . the proposed model could be used to prob complex tasks in shallow ones . |
Can Large Language Models Interpret Noun-Noun Compounds? A Linguistically-Motivated Study on Lexicalized and Novel Compounds (2024.acl-long)
Copied to clipboard
| Challenge: | Noun-noun compounds represent an important challenge for Natural Language Understanding . correct interpretation of noun-nomin compounds is essential for many applications . |
| Approach: | They test whether Large Language Models can interpret the semantic relation between nouns . they also test whether they can abstract from such knowledge to predict the relation . |
| Outcome: | The proposed models can interpret the semantic relation between nouns and compounds using analogical comparisons. |
Current Advances in LLM Reasoning (2026.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial examines comprehensive evaluation strategies to assess the reasoning abilities of large language models (LLMs) advanced inference time methods and post-training methods that aim to make LLMs think more like humans are discussed in this tutorial. |
| Approach: | This tutorial explores comprehensive evaluation strategies to assess the reasoning abilities of large language models (LLMs) and discusses two types of methods to improve models’ reasoning: advanced inference time methods, structured and self-improvement inference methods, and post-training methods, such as RLHF, DPO, and GRPO. |
| Outcome: | This tutorial examines evaluation strategies to assess the reasoning abilities of large language models and discusses two types of methods to improve models’ reasoning. |
Forget What You Know about LLMs Evaluations - LLMs are Like a Chameleon (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) excel on public benchmarks, but high scores may mask overreliance on dataset-specific surface cues rather than true language understanding. |
| Approach: | They propose a meta-evaluation framework that systematically rephrases benchmark inputs to detect overfitting. |
| Outcome: | The proposed framework detects performance degradation indicative of superficial pattern reliance on dataset-specific cues and distortion levels. |
Cross-Modal Taxonomic Generalization in (Vision-) Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that language models learn from surface form to learn from more grounded evidence. |
| Approach: | They propose to use a vision-language model to learn hypernyms from images . they find that the model can recover this knowledge and generalize even when there is no hypernomia in the image. |
| Outcome: | The proposed model can recover this knowledge and generalize even when the model receives no evidence of hypernyms during training. |