Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit impressive abilities in various domains such as text generation, instruction following, and reasoning. |
| Approach: | They propose a method to decompose the activations of Large Language Models into a sparse linear combination of SAE features. |
| Outcome: | The proposed method shows that some features are strongly related to specific languages, while others are unaffected by ablating them. |
Similar Papers
Deciphering Cultural Representations in Large Language Models via Sparse Autoencoders (2026.findings-acl)
Copied to clipboard
| Challenge: | Prior work has identified so-called cultural neurons, but individual neurons are often polysemous, conflating abstract cultural knowledge with surface-level lexical cues due to superposition. |
| Approach: | They apply Sparse Autoencoders to decompose LLM activations into sparse, interpretable feature representations that disentangle culturally selective features. |
| Outcome: | The proposed model disentangles culturally selective features from paraphrasing and task formats, indicating abstraction beyond lexical correlations. |
Evaluating the Impact of SAE-based Language Steering on LLM Performance (2026.eacl-srw)
Copied to clipboard
| Challenge: | Recent advances in Sparse Autoencoders (SAEs) have revealed interpretable features within large language models (LLMs) however, the impact of SAE-based language steering on output quality and task performance remains unclear. |
| Approach: | They apply language-specific SAE feature steering to three LLMs from two model families and evaluate it on a translation task and a multilingual question-answering task. |
| Outcome: | The proposed approach outperforms prompting and language neuron-based steering on translation and multilingual question-answering tasks. |
Sparse Autoencoder Features for Classifications and Transferability (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) provide potential for uncovering structured, human-interpretable representations in Large Language Models (LLMs). |
| Approach: | They analyze SAEs for interpretable feature extraction from Large Language Models in safety-critical classification tasks. |
| Outcome: | The proposed framework outperforms hidden-state and BoW models while demonstrating cross-lingual toxicity detection and visual classification tasks. |
A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) can disentangle complex features into more interpretable components. |
| Approach: | They propose to use Sparse Autoencoders to disentangle LLM features into more interpretable components. |
| Outcome: | The proposed method disentangles complex features into more interpretable components. |
Toward Efficient Sparse Autoencoder-Guided Steering for Improved In-Context Learning in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse autoencoders (SAEs) have emerged as a powerful analytical tool in mechanistic interpretability for large language models (LLMs). |
| Approach: | They propose a novel approach that leverages SAEs to enhance the general in-context learning performance of large language models (LLMs). |
| Outcome: | The proposed method yields a 3.5% improvement across diverse text classification tasks and exhibits greater robustness to hyperparameter variations compared to standard steering approaches. |
LinguaLens: Towards Interpreting Linguistic Mechanisms of Large Language Models via Sparse Auto-Encoder (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior research on linguistic mechanisms of large language models is limited by coarse granularity, limited analysis scale, and narrow focus. |
| Approach: | They propose a framework for analyzing the linguistic mechanisms of large language models based on Sparse Auto-Encoders. |
| Outcome: | The proposed framework extracts Chinese and English linguistic features across four dimensions . it uncovers intrinsic representations of linguistic knowledge in LLMs and can control outputs . |
On the Versatility of Sparse Autoencoders for In-Context Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Sparse autoencoders (SAEs) are emerging as a key analytical tool in interpretability for large language models. |
| Approach: | They propose to use SAEs to extract knowledge from billions of tokens for sparse reconstruction. |
| Outcome: | The proposed model can extract knowledge from billions of tokens for sparse reconstruction. |
Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages (2025.naacl-long)
Copied to clipboard
| Challenge: | In the brains of human bilinguals, syntax processing may occur in similar regions for their first and second language, depending on factors like when the second language was learned and language proficiency. |
| Approach: | They propose to use sparse autoencoders to train Llama-3-8B and Aya-23-8B models to train multilingual models that share morphsyntactic representations of grammatical concepts. |
| Outcome: | The proposed model can predict plural verbs in different languages by activating the same plural feature. |
Route Sparse Autoencoder to Interpret Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse autoencoders (SAEs) extract interpretable and monosemantic features in large language models . prior work focused on feature extraction from a single layer, failing to capture activations that span multiple layers. |
| Approach: | They propose a framework that integrates a routing mechanism with a shared SAE to efficiently extract features from multiple layers. |
| Outcome: | The proposed framework extracts features from multiple layers while incurring minimal parameter overhead while achieving high interpretability and flexibility. |
Group-SAE: Efficient Training of Sparse Autoencoders for Large Language Models via Layer Groups (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) are a promising unsupervised approach for understanding the representations of layers of Large Language Models (LLMs). |
| Approach: | They propose a method that groups similar models and trains a single SAE per group based on representational similarity across layers. |
| Outcome: | Experiments on Pythia family models show that the proposed method significantly accelerates training with minimal impact on reconstruction quality and comparable downstream task performance and interpretability over baseline SAEs trained layer by layer. |