Mechanistic Interpretability of Large-Scale Counting in LLMs through a System-2 Strategy

Challenge:	Large language models exhibit systematic limitations in counting tasks due to depth constraints.
Approach:	They propose a method that decomposes large counting tasks into smaller, independent sub-problems that the model can reliably solve.
Outcome:	The proposed method surpasses architectural limitations and achieves higher accuracy on large-scale counting tasks.

Similar Papers

Overview

Challenge

Approach

Outcome

LLM2: Let Large Language Models Harness System 2 Reasoning (2025.naacl-short)

Copied to clipboard

Cheng Yang, Chufan Shi, Siheng Li, Bo Shui, Yujiu Yang, Wai Lam

Language Models

Challenge:	Empirical results on mathematical reasoning benchmarks substantiate the efficacy of Large language models (LLMs).
Approach:	They propose a framework that combines an LLM with a process-based verifier to generate plausible candidates and provide timely process-driven feedback to distinguish desirable and undesirable outputs.
Outcome:	Empirical results show that LLM2 improves accuracy on GSM8K and self-consistency increases major@20 accuracy.

The Role of Deductive and Inductive Reasoning in Large Language Models (2025.acl-long)

Copied to clipboard

Chengkun Cai, Xu Zhao, Haoliang Liu, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang, Lei Li

Language Models

Challenge:	Large Language Models (LLMs) have demonstrated impressive capabilities in reasoning tasks, yet their reliability in problem-solving remains debatable.
Approach:	They propose a framework that integrates both deductive and inductive reasoning approaches to enhance LLM reasoning by progressively adapting its reasoning pathways based on problem complexity.
Outcome:	The proposed framework achieves 70.3% accuracy on AIW, compared to 62.2% for Tree of Thought, while maintaining lower computational costs.

Working Memory Identifies Reasoning Limits in Language Models (2024.emnlp-main)

Copied to clipboard

Chunhui Zhang, Yiren Jian, Zhongyu Ouyang, Soroush Vosoughi

Language Models

Challenge:	Using large language models, we examine the limitations of their cognitive capabilities and their working memory.
Approach:	They examine the limitations of large language models from a scaling perspective . they also assess various prompting strategies, revealing their diverse impacts on LLM performance.
Outcome:	The proposed models perform poorly on n-back tasks and on prompting strategies.

LogiDynamics: Unraveling the Dynamics of Inductive, Abductive and Deductive Logical Inferences in LLM Reasoning (2025.emnlp-main)

Copied to clipboard

Tianshi Zheng, Cheng Jiayang, Chunyang Li, Haochen Shi, Zihao Wang, Jiaxin Bai, Yangqiu Song, Ginny Wong, Simon See

Multimodality

Challenge:	Modern large language models (LLMs) employ diverse logical inference mechanisms for reasoning.
Approach:	They analyze the comparative dynamics of inductive (System 1) versus abductive/deductive (system 2) inference in large language models by using a controlled analogical reasoning environment and a MCQ/free-text task format.
Outcome:	The proposed methods can significantly scale LLM reasoning.

FAC2E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition (2024.emnlp-main)

Copied to clipboard

Xiaoqiang Wang, Lingfei Wu, Tengfei Ma, Bang Liu

Semantic Text Processing

Challenge:	Large language models (LLMs) are evaluated by overall performance on various text understanding and generation tasks.
Approach:	They propose a framework for Fine-grAined and Cognition-grounded LLMs’ Capability Evaluation that dissociates the language-related capabilities from cognition-related ones.
Outcome:	The proposed framework dissociates the language-related capabilities from cognition-related ones and breaks down the process of applying a specific capability into three sub-steps: recalling relevant knowledge, utilizing knowledge, and solving problems.

Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLMs (2025.emnlp-main)

Copied to clipboard

Zhen Xiong, Yujun Cai, Zhecheng Li, Yiwei Wang

Responsible & Trustworthy NLP

Challenge:	Large Reasoning Models (LRMs) often display unstable behaviors, e.g., hallucinating unsupported premises, overthinking simple tasks, and displaying higher sensitivity to prompt variations.
Approach:	They propose a graph-based analytical framework that clusters long, verbose CoT outputs into semantically coherent reasoning steps, then constructs directed reasoning graphs to capture contextual and logical dependencies among these steps.
Outcome:	The proposed framework enables quantitative evaluation of internal reasoning structure and quality beyond conventional metrics and provides practical insights for prompt engineering and cognitive analysis of LLMs.

Do Large Language Models Know How Much They Know? (2024.emnlp-main)

Copied to clipboard

Gabriele Prato, Jerry Huang, Prasanna Parthasarathi, Shagun Sodhani, Sarath Chandar

Semantic Text Processing

Challenge:	Large Language Models are highly capable systems, but their capabilities and limitations are unclear.
Approach:	They develop a benchmark that challenges LLMs to recall all information they possess on specific topics.
Outcome:	The proposed model can recall excessive, insufficient, or the precise amount of information they possess on a given topic, indicating their awareness of how much they know about the given topic.

On Generalization across Measurement Systems: LLMs Entail More Test-Time Compute for Underrepresented Cultures (2025.acl-long)

Copied to clipboard

Minh Duc Bui, Kyung Eun Park, Goran Glavaš, Fabian David Schmidt, Katharina Von Der Wense

Reasoning

Challenge:	Large Language Models (LLMs) should be able to provide accurate information irrespective of the measurement system at hand .
Approach:	They use newly compiled datasets to test if this is true for seven open-source LLMs.
Outcome:	The proposed model can provide accurate information regardless of the measurement system at hand.

LM2: A Simple Society of Language Models Solves Complex Reasoning (2024.emnlp-main)

Copied to clipboard

Gurusha Juneja, Subhabrata Dutta, Tanmoy Chakraborty

Language Models

Challenge:	Existing studies show that providing guidance via decomposing the original question into multiple subproblems elicits more robustness in LLM reasoning.
Approach:	They propose a language-based decomposition, solution and verification framework that modularizes the decomposer, solution, and verification into three different language models.
Outcome:	The proposed model outperforms existing methods on in- and out-domain reasoning problems, outperforming the best baselines by 8.1% on MATH, 7.71% on JEEBench, and 9.7% on MedQA problems.

Rethinking Tabular Data Understanding with Large Language Models (2024.naacl-long)

Copied to clipboard

Tianyang Liu, Fei Wang, Muhao Chen

Semantic Text Processing

Challenge:	Large Language Models (LLMs) are capable of various tasks, yet their capability in interpreting and reasoning over tabular data remains an underexplored area.
Approach:	They propose a method for table structure normalization to improve model performance . they propose aggregation of multiple reasoning pathways to improve performance based on textual and symbolic reasoning.
Outcome:	The proposed method improves performance on symbolic reasoning tasks with textual reasoning slightly outperforming symbolic reasoning on tables.