Activation Scaling for Steering and Interpreting Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a successful intervention should flip the correct with the wrong token, while remaining sparse. |
| Approach: | They propose to use activation scaling to flip the correct with the wrong token . they use gradient-based optimization to learn and evaluate a specific kind of efficient intervention . |
| Outcome: | The proposed method performs comparable with steering vectors but is much less minimal. |
Similar Papers
From Weights to Activations: Is Steering the Next Frontier of Adaptation? (2026.acl-long)
Copied to clipboard
Simon Ostermann, Daniil Gurgurov, Tanja Baeumel, Michael A. Hedderich, Sebastian Lapuschkin, Wojciech Samek, Vera Schmitt
| Challenge: | Pre-trained large language models are the basis of a wide range of NLP tasks. |
| Approach: | They propose to use parameter updates and parameter-efficient adaptation to modify behavior of large language models. |
| Outcome: | The proposed method enables local and reversible behavioral change without parameter updates. |
Can Activation Steering Generalize Across Languages? A Study on Syllogistic Reasoning in Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Prior work has focused on activation steering for Large Language Models (LLMs) this technique can be used to improve reasoning accuracy and transferability across languages. |
| Approach: | They propose to use activation steering to steer models towards a cross-lingual reasoning space. |
| Outcome: | The proposed techniques generalise well to multilingual datasets while minimizing language modelling performance. |
Style Vectors for Steering Generative Large Language Models (2024.findings-eacl)
Copied to clipboard
Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, Tobias Hecking
| Challenge: | Large language models (LLMs) can be trained on vast corpora and can generate text in a nuanced and parameterisable way. |
| Approach: | They propose to add style vectors to the activations of hidden layers during text generation to steer output towards specific styles. |
| Outcome: | The proposed approach differs from prompt engineering in that it can be nuanced and parameterisable. |
Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspective (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to edit LLMs' activations are limited by their magnitude and direction consistency. |
| Approach: | They propose a method that edits activations to alter their magnitudes and directions to preserve activation norms. |
| Outcome: | The proposed method preserves activation norm and improves safety benchmarks. |
Embracing Anisotropy: Turning Massive Activations into Interpretable Control Knobs for Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work shows that Large Language Models exhibit highly anisotropic internal representations . prior work shows specialized dimensions capture domain-specific features . |
| Approach: | They propose a simple magnitude-based criterion to identify Domain-Critical Dimensions in a training-free manner. |
| Outcome: | The proposed method outperforms whole-dimension steering in domain adaptation and jailbreaking scenarios. |
RISER: Orchestrating Latent Reasoning Skills for Adaptive Activation Steering (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for domain-specific reasoning with large language models require updating parameter updates. |
| Approach: | They propose a plug-and-play intervention framework that adaptively steers LLM reasoning in activation space. |
| Outcome: | The proposed framework achieves zero-shot accuracy improvements of 3.4–6.5% over the base model while outperforming chain-of-thought-style reasoning with 2–3 higher token efficiency and robust accuracy gains. |
DeCoVec: Building Decoding Space based Task Vector for Large Language Models via In-Context Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to steering large language models require fine-tuning or manipulation of internal states, limiting their flexibility and scalability. |
| Approach: | They propose a framework that constructs task vectors directly in the decoding space by leveraging in-context learning. |
| Outcome: | The proposed framework outperforms standard few-shot baselines on TruthfulQA, Math-500, and AQUA-RAT with gains up to +5.50 accuracy. |
SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | SteerVLM is a lightweight steering module designed to guide Vision-Language Models (VLMs) towards outputs that better adhere to desired instructions. |
| Approach: | They propose a lightweight steering module that learns from latent embeddings of paired prompts encoding target and converse behaviors to dynamically adjust activations connecting the language modality with image context. |
| Outcome: | The proposed steering module outperforms existing intervention techniques on steering and hallucination mitigation benchmarks for VLMs. |
GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models often ignore token-level causal influence and underutilize model logits. |
| Approach: | They propose a novel approach that uses a gradient-based approach to identify influential tokens and construct directional steering vectors based on their contribution to preferred over dispreferred outputs. |
| Outcome: | The proposed approach outperforms fine-tuning and prior steering methods on both LLM and VLM tasks without degrading fluency or general capabilities. |
Pitfalls of Scale: Investigating the Inverse Task of Redefinition in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable results in several linguistic, reasoning and knowledge retrieval tasks. |
| Approach: | They propose to scale Large Language Models (LLMs) to scale up to reveal potential reasoning gaps as LLMs scale up. |
| Outcome: | The proposed redefinition task shows that model performance degrades with scale, and false confidence rises. |