Interpreting Arithmetic Mechanism in Large Language Models through Comparative Neuron Analysis (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have found that arithmetic ability is limited to a few attention heads . existing studies do not elaborate on the mechanisms of these heads or how they influence FFN layers. |
| Approach: | They propose a method that identifies an internal logic chain consisting of four stages from input to prediction. |
| Outcome: | The proposed method improves prediction probabilities by amplifying coefficient scores of FFN neurons related to predictions. |
Similar Papers
An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Prior work has focused on ablating components in the CoT prompt, but the reason why these components are important to LLM reasoning is not explored. |
| Approach: | They investigate "neuron activation" as a lens to provide a unified explanation to previous work . they propose an approach to automatically identify neurons that imply arithmetic reasoning . |
| Outcome: | The proposed approach can explain the importance of components in a CoT prompt . it also automatically identifies neurons that imply arithmetic reasoning . |
Interpretability Analysis of Arithmetic In-Context Learning in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) solve arithmetic with only a few in-context examples, yet the computations that connect those examples to the answer remain opaque. |
| Approach: | They propose to use in-context examples to illustrate how large language models process ICEs to isolate partial-sum representations in three-operand tasks and investigate their influence on final logits. |
| Outcome: | The proposed model performs better than previous models on three-operand tasks. |
Memory or Reasoning? Explore How LLMs Compute Mixed Arithmetic Expressions (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) can solve complex multi-step math reasoning problems, but their internal implementation is limited. |
| Approach: | They propose to use a "C**ausal **E**ffect **D**riven **F**ine-tuning method" to improve LLMs' reasoning ability. |
| Outcome: | The proposed method improves the model's reasoning ability by enhancing key components that are used to execute mixed arithmetic calculations. |
Neuron-Level Knowledge Attribution in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for attribution of knowledge in large language models struggle to operate at neuron level due to computational constraints. |
| Approach: | They propose a static method for pinpointing significant neurons using three metrics . they also propose identifying "query neurons" which activate these "value neurons" |
| Outcome: | The proposed method shows superior performance across three metrics compared to seven other methods . it analyzes six types of knowledge across attention and feed-forward network layers . |
Neuron-Level Differentiation of Memorization and Generalization in Large Language Models (2025.emnlp-main)
Copied to clipboard
Ko-Wei Huang, Yi-Fu Fu, Ching-Yu Tsai, Yu-Chieh Tu, Tzu-ling Cheng, Cheng-Yu Lin, Yi-Ting Yang, Heng-Yi Liu, Keng-Te Liao, Da-Cheng Juan, Shou-De Lin
| Challenge: | Existing models exhibit memorization and generalization behaviors in ways that are not easily interpretable or controllable. |
| Approach: | They propose to use a GPT-2 and LLaMA-3.2 model to identify distinct neuron subsets responsible for each behavior to steer the model toward memorization or generalization. |
| Outcome: | The proposed models show that inference-time interventions on these neurons can steer the model’s behavior toward memorization or generalization. |
Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy Arithmetic Tasks (2024.acl-short)
Copied to clipboard
| Challenge: | Despite the generality and far-reaching consequences of large language models, there are still significant limitations making it difficult to apply them to certain tasks. |
| Approach: | They show that large language models can perform arithmetic tasks more robustly when conditioned on all of the correct higher-order digits. |
| Outcome: | The proposed model can predict the first digit of n-digit by m-digit multiplication without chain of thought reasoning, but in practice it fails to correctly predict the last digit on n digit by 1-digit multiplikation . |
The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have explored methods to enhance self-correction in large language models, but little attention has been given to understanding the models’ internal mechanisms underlying error detection. |
| Approach: | They propose to use a large language model to analyze arithmetic errors in four smaller-sized LLMs and identify their internal mechanisms. |
| Outcome: | The proposed models heavily rely on consistency headstextemdashattention heads that assess surface-level alignment of numerical values in arithmetic solutions. |
A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on how large language models process and store information related to arithmetic tasks have shown their behavior inconsistent and context-dependent. |
| Approach: | They propose to mechanize the processing of arithmetic queries by a causal mediation framework. |
| Outcome: | The proposed model improves the performance of arithmetic queries with a set of MLP modules. |
Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals (2024.acl-long)
Copied to clipboard
| Challenge: | Existing interpretability research focused on analyzing a single mechanism . et al., 2023) focused on finding how models copy or recall factual knowledge . |
| Approach: | They propose a competition of mechanisms that focuses on the interplay of multiple mechanisms instead of individual mechanisms . they uncover how and where the competition of mechanism happens within LLMs using logit inspection and attention modification methods. |
| Outcome: | The proposed model is based on two interpretability methods, logit inspection and attention modification. |
Fine-Grained Analysis of Shared Syntactic Mechanisms in Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in language models have demonstrated their ability to process linguistic expressions with complex syntactic structures. |
| Approach: | They investigate whether language models employ shared neural mechanisms across different constructions by applying causal interpretability methods at a granular level. |
| Outcome: | The proposed model performance improves on acceptability judgment benchmarks. |