SAE-SSV: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have impressive capabilities in natural language understanding and generation, but controlling their behavior remains a challenge. |
| Approach: | They propose a supervised steering approach that operates in sparse, interpretable representation spaces. |
| Outcome: | The proposed approach achieves higher success rates with minimal degradation in generation quality compared to existing methods. |
Similar Papers
Evaluating the Impact of SAE-based Language Steering on LLM Performance (2026.eacl-srw)
Copied to clipboard
| Challenge: | Recent advances in Sparse Autoencoders (SAEs) have revealed interpretable features within large language models (LLMs) however, the impact of SAE-based language steering on output quality and task performance remains unclear. |
| Approach: | They apply language-specific SAE feature steering to three LLMs from two model families and evaluate it on a translation task and a multilingual question-answering task. |
| Outcome: | The proposed approach outperforms prompting and language neuron-based steering on translation and multilingual question-answering tasks. |
Toward Efficient Sparse Autoencoder-Guided Steering for Improved In-Context Learning in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse autoencoders (SAEs) have emerged as a powerful analytical tool in mechanistic interpretability for large language models (LLMs). |
| Approach: | They propose a novel approach that leverages SAEs to enhance the general in-context learning performance of large language models (LLMs). |
| Outcome: | The proposed method yields a 3.5% improvement across diverse text classification tasks and exhibits greater robustness to hyperparameter variations compared to standard steering approaches. |
Sparse Autoencoder Features for Classifications and Transferability (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) provide potential for uncovering structured, human-interpretable representations in Large Language Models (LLMs). |
| Approach: | They analyze SAEs for interpretable feature extraction from Large Language Models in safety-critical classification tasks. |
| Outcome: | The proposed framework outperforms hidden-state and BoW models while demonstrating cross-lingual toxicity detection and visual classification tasks. |
Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit impressive abilities in various domains such as text generation, instruction following, and reasoning. |
| Approach: | They propose a method to decompose the activations of Large Language Models into a sparse linear combination of SAE features. |
| Outcome: | The proposed method shows that some features are strongly related to specific languages, while others are unaffected by ablating them. |
Denoising Concept Vectors with Sparse Autoencoders for Improved Language Model Steering (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for steering concept vectors suffer from noisy features in diverse datasets that undermine steering robustness. |
| Approach: | They propose a Sparse Autoencoder-Denoised Concept Vector (SDCV) which selectively keeps the most discriminative SAE latents while reconstructing hidden representations. |
| Outcome: | The proposed method improves steering success rates by 4-16% across six challenging concepts while maintaining topic relevance. |
SAEs Are Good for Steering – If You Select the Right Features (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) can learn a decomposition of a model’s latent space by analyzing the input tokens that activate them. |
| Approach: | They propose an unsupervised approach to learn a decomposition of a model’s latent space by analyzing the input tokens that activate them. |
| Outcome: | The proposed approach matches the performance of existing supervised methods by identifying features with low output scores and identifying them with input and output scores. |
A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) can disentangle complex features into more interpretable components. |
| Approach: | They propose to use Sparse Autoencoders to disentangle LLM features into more interpretable components. |
| Outcome: | The proposed method disentangles complex features into more interpretable components. |
On the Versatility of Sparse Autoencoders for In-Context Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Sparse autoencoders (SAEs) are emerging as a key analytical tool in interpretability for large language models. |
| Approach: | They propose to use SAEs to extract knowledge from billions of tokens for sparse reconstruction. |
| Outcome: | The proposed model can extract knowledge from billions of tokens for sparse reconstruction. |
Model Unlearning via Sparse Autoencoder Subspace Guided Projections (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing unlearning strategies lack interpretability or fail to provide robust defense against adversarial prompts. |
| Approach: | They propose a framework that leverages SAE features to drive targeted updates in the model’s parameter space. |
| Outcome: | The proposed framework reduces harmful knowledge accuracy by 3.22% compared to baselines and improves adversarial robustness under jailbreak prompts. |
Group-SAE: Efficient Training of Sparse Autoencoders for Large Language Models via Layer Groups (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) are a promising unsupervised approach for understanding the representations of layers of Large Language Models (LLMs). |
| Approach: | They propose a method that groups similar models and trains a single SAE per group based on representational similarity across layers. |
| Outcome: | Experiments on Pythia family models show that the proposed method significantly accelerates training with minimal impact on reconstruction quality and comparable downstream task performance and interpretability over baseline SAEs trained layer by layer. |