From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts? (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to evaluate features disentangle concepts from activations of neural networks are limited by their quality . current methods for concept identification and steering are sparse autoencoders, but they are not reliable. |
| Approach: | They propose to evaluate how well featurization methods disentangle one concept from another . they use sentiment, domain, voice, and tense to steer these features . |
| Outcome: | The proposed evaluations show that featurization methods are insufficient to establish steering selectivity . the results suggest that steering a feature affects many concepts despite a near absence of interaction effects. |
Similar Papers
RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to disentangle individual neurons from multiple high-level concepts are not yet benchmarked. |
| Approach: | They propose a method of Multi-task Distributed Alignment Search that allows to find distributed representations satisfying multiple causal criteria. |
| Outcome: | The proposed method achieves state-of-the-art on the target language model with Llama2-7B . |
Constructing Interpretable Features from Compositional Neuron Groups (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for analyzing LLMs rely on dictionary learning with sparse autoencoders (SAEs) however, SAEs struggle in causal evaluations and lack intrinsic interpretability, as their learning is not explicitly tied to the computations of the model. |
| Approach: | They propose to decompose MLP activations with semi-nonnegative matrix factorization (SNMF) such that the learned features are mapped to their activating inputs, making them directly interpretable. |
| Outcome: | Experiments on Llama 3.1, Gemma 2 and GPT-2 show that SNMF derived features outperform SAEs and a strong supervised baseline on causal steering while aligning with human-interpretable concepts. |
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020.acl-main)
Copied to clipboard
| Challenge: | Current approaches to interpretability evaluation focus on faithfulness criteria . current approaches focus on readability, plausibility and faithfulness . |
| Approach: | They argue that current binary definition of faithfulness sets unrealistic standards . they argue that a more graded definition would be of greater practical utility . |
| Outcome: | The proposed approach is based on three assumptions and lacks a graded definition of faithfulness. |
Interpretable Multi-dataset Evaluation for Named Entity Recognition (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation methods for named entity recognition tasks are difficult to interpret . authors present a general methodology for interpretable evaluation for named entities . |
| Approach: | They propose a general methodology for interpretable evaluation for named entity recognition task. |
| Outcome: | The proposed evaluation method enables researchers to interpret differences in models and datasets . it makes it easy for future researchers to run similar analyses and drive progress in this area . |
Evaluating Readability and Faithfulness of Concept-based Explanations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluating concepts from different perspectives lack a unified formalization. |
| Approach: | They propose a formal definition of concepts generalizing to diverse concept-based explanations’ settings and apply it to other types of explanations or tasks. |
| Outcome: | Extensive experimental analysis was carried out to determine the evaluation measures for explanation evaluation measures. |
Unveiling Decision-Making in LLMs for Text Classification : Extraction of influential and interpretable concepts with Sparse Autoencoders (2026.findings-eacl)
Copied to clipboard
| Challenge: | Concept-based explanations for large language models are not well understood in text classification. |
| Approach: | They propose a model with a specialized classifier head and activation rate sparsity loss for sentence classification . they compare it to existing models with HI-Concept and ConceptShap . |
| Outcome: | The proposed model improves both the causality and interpretability of the extracted features. |
Is Multi-Hop Reasoning Really Explainable? Towards Benchmarking Reasoning Interpretability (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for multi-hop reasoning are not able to evaluate their interpretability . a recent study found that many paths are unreasonable . |
| Approach: | They propose a framework to evaluate the interpretability of multi-hop reasoning models . they annotate all possible rules and establish a benchmark . |
| Outcome: | The proposed framework outperforms existing models in terms of performance and interpretability. |
Neuron-level Interpretation of Deep NLP Models: A Survey (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing work on deep neural networks has focused on representation analysis, but recent work focused on analyzing neurons within these models. |
| Approach: | They propose to analyze neural networks to uncover linguistic concepts captured by the network . they propose to use a granular approach to analyze neurons within these models . |
| Outcome: | The proposed method combines methods to discover and understand neurons in a network with evaluation methods. |
On the Lack of Robust Interpretability of Neural Text Classifiers (2021.findings-acl)
Copied to clipboard
Muhammad Bilal Zafar, Michele Donini, Dylan Slack, Cedric Archambeau, Sanjiv Das, Krishnaram Kenthapadi
| Challenge: | Several models have been proposed to interpret models with feature-based interpretability methods. |
| Approach: | They propose to quantify the robustness of neural text classifiers by using two randomization tests to compare models with identical initializations. |
| Outcome: | The proposed methods show surprising deviations from expected behavior . the results raise questions about the extent of insights that practitioners may draw from interpretations. |
When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent studies have found that Large Language Models process multilingual inputs in shared concept spaces, thought to support generalization and cross-lingual transfer. |
| Approach: | They investigate the development of language-agnostic concept spaces during pretraining of EuroLLM using the causal interpretability method of activation patching. |
| Outcome: | The proposed model is language-agnostic and enables cross-lingual transfer . the model is able to process multilingual inputs, but lacks cross-linguistic alignment . |