Feature Drift: How Fine-Tuning Repurposes Representations in LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Sparse autoencoders (SAEs) are a powerful tool for interpreting neural networks by extracting concepts (features) represented in their activations. |
| Approach: | They propose to use Sparse Autoencoders to extract concepts from their activations to explain how fine-tuning changes model capabilities. |
| Outcome: | The proposed model recombines existing concepts rather than learning new ones, and shows that it is a better explanation for how fine-tuning changes model capabilities. |
Similar Papers
From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have achieved remarkable success in aligning with user intentions. |
| Approach: | They develop local and global explanation methods and a feed-forward-based method for input-output attribution to investigate the impact of instruction tuning on user intentions. |
| Outcome: | The proposed method compares explanations from pre-trained and instruction-tuned models . it empowers LLMs to recognize the instruction parts of user prompts, it encourages response generation . |
On the Impact of Fine-Tuning on Chain-of-Thought Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models have emerged as powerful tools for general intelligence, showcasing advanced natural language processing capabilities. |
| Approach: | They propose to use supervised fine-tuning and Quantized Low-Rank Adapters to improve LLMs' task-specific performance to address privacy and safety risks. |
| Outcome: | The proposed model improves the accuracy of the chain-of-thought reasonings across four datasets and demonstrates that the faithfulness of CoT reasoning decreases. |
Understanding Emergent Misalignment via Feature Superposition Geometry (2026.acl-long)
Copied to clipboard
| Challenge: | Emergent misalignment is a problem for large language models (LLMs) fine-tuning on narrow tasks can induce harmful behaviors despite no explicit supervision. |
| Approach: | They propose a mechanistic account based on the geometry of feature superposition . they propose to use sparse autoencoders to identify misalignment-inducing features . |
| Outcome: | The proposed model outperforms random removal and stronger mitigations than LLM-as-a-judge filtering. |
Intention-Adaptive LLM Fine-Tuning for Text Revision Generation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing work on large language models (LLMs) has demonstrated impressive capabilities in context-based text generation tasks, such as summarization and reasoning. |
| Approach: | They propose an intention-adaptive layer-wise LLM fine-tuning framework that dynamically selects a subset of LLM layers to learn intentions and transfers them to revision generation. |
| Outcome: | The proposed framework outperforms PEFT baselines on small revision corpora while maintaining fast convergence and accuracy. |
Exploring Memorization in Fine-tuned Language Models (2024.acl-long)
Copied to clipboard
Shenglai Zeng, Yaxin Li, Jie Ren, Yiding Liu, Han Xu, Pengfei He, Yue Xing, Shuaiqiang Wang, Jiliang Tang, Dawei Yin
| Challenge: | Existing studies have shown that pre-trained langauge models tend to memorize and regenerate segments of their pre-training corpus when prompted appropriately. |
| Approach: | They conduct the first comprehensive analysis to explore language models’ memorization during fine-tuning across tasks. |
| Outcome: | The proposed analysis shows that memorization presents a strong disparity among different fine-tuning tasks. |
SAEs Are Good for Steering – If You Select the Right Features (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) can learn a decomposition of a model’s latent space by analyzing the input tokens that activate them. |
| Approach: | They propose an unsupervised approach to learn a decomposition of a model’s latent space by analyzing the input tokens that activate them. |
| Outcome: | The proposed approach matches the performance of existing supervised methods by identifying features with low output scores and identifying them with input and output scores. |
Efficient Layer-wise LLM Fine-tuning for Revision Intention Prediction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models have shown extraordinary success across text generation tasks . however, their potential for simple yet essential text classification remains underexplored . |
| Approach: | a plug-and-play layer-wise parameter-efficient fine-tuning framework is proposed . it fine- tunes a subset of important LLM layers while freezing redundant ones . |
| Outcome: | a plug-and-play framework fine-tunes a subset of important LLM layers while freezing redundant layers. |
Deciphering Cultural Representations in Large Language Models via Sparse Autoencoders (2026.findings-acl)
Copied to clipboard
| Challenge: | Prior work has identified so-called cultural neurons, but individual neurons are often polysemous, conflating abstract cultural knowledge with surface-level lexical cues due to superposition. |
| Approach: | They apply Sparse Autoencoders to decompose LLM activations into sparse, interpretable feature representations that disentangle culturally selective features. |
| Outcome: | The proposed model disentangles culturally selective features from paraphrasing and task formats, indicating abstraction beyond lexical correlations. |
Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) encapsulate a vast amount of factual information within their pre-trained weights. |
| Approach: | They compare unsupervised fine-tuning and retrieval-augmented generation approaches to learning new factual information. |
| Outcome: | The proposed models outperform unsupervised fine-tuning and retrieval-augmented generation (RAG) on knowledge-intensive tasks across different topics. |
Understanding Refusal in Language Models with Sparse Autoencoders (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a study of refusal in instruction-tuned language models identifies latent features that causally mediate refusal behaviors. |
| Approach: | They conduct a mechanistic study of refusal in instruction-tuned LLMs using sparse autoencoders . they identify latent features that causally mediate refusal behaviors using sparsed autoencoding . |
| Outcome: | The proposed method validates refusal-related features across multiple datasets. |