Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuning

Challenge:	Recent approaches to fine-tuning of large language models suffer from task interference and catastrophic forgetting.
Approach:	They propose a fine-tuning framework that adapts isolation decisions based on online estimates of parameter importance.
Outcome:	The proposed framework reduces interference and forgetting while releasing outdated parameters to recover plasticity.

Similar Papers

Overview

Challenge

Approach

Outcome

Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performance (2025.emnlp-main)

Copied to clipboard

Yao Wang, Di Liang, Minlong Peng

Language Models

Challenge:	Extensive experiments demonstrate that our approach significantly alleviates task interference and forgetting.
Approach:	They propose a framework for supervised fine-tuning for large language models . they first fine-tail the model on each task to identify its core parameter regions .
Outcome:	The proposed framework outperforms vanilla fine-tuning and baselines on multiple public benchmarks on reasoning, dialogue, instruction following, and more.

Rehearsal-free Continual Language Learning via Efficient Parameter Isolation (2023.acl-long)

Copied to clipboard

Zhicheng Wang, Yufang Liu, Tao Ji, Xiaoling Wang, Yuanbin Wu, Congcong Jiang, Ye Chao, Zhencong Han, Ling Wang, Xu Shao, Wenqiu Zeng

Green & Sustainable NLP

Challenge:	Existing methods for learning continual tasks do not cache history data, which makes the problem more challenging.
Approach:	They propose a method that allocates a small portion of private parameters and learns them with a shared pre-trained model.
Outcome:	The proposed method is comparable to existing methods and comparable to those using historical data.

Dissecting Fine-Tuning Unlearning in Large Language Models (2024.emnlp-main)

Copied to clipboard

Yihuai Hong, Yuelin Zou, Lijie Hu, Ziqian Zeng, Di Wang, Haiqin Yang

Language Models

Challenge:	Existing methods for fine-tuning-based unlearning are ineffective at completely erasing model-embedded knowledge, but their true effectiveness remains unclear.
Approach:	They propose to use activation patching and parameter restoration experiments to examine the limitations of fine-tuning-based unlearning methods for erasing harmful, sensitive, or copyrighted information within large language models.
Outcome:	The proposed methods alter the model’s knowledge retrieval process rather than genuinely erasing the problematic knowledge embedded in the model parameters.

Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging (2024.emnlp-main)

Copied to clipboard

Yiming Ju, Ziyi Ni, Xingrun Xing, Zhixiong Zeng, Hanyu Zhao, Siqi Fan, Zheng Zhang

Language Models

Challenge:	Existing studies suggest that the order of training samples can affect model performance, but this is not the case.
Approach:	They propose to merge supervised fine-tuning models with different data orders to mitigate this imbalance by parameter merging.
Outcome:	The proposed method outperforms the weighted-average method on five datasets.

Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models (2023.acl-long)

Copied to clipboard

Zhong Zhang, Bang Liu, Junming Shao

Language Models

Challenge:	Pre-trained language models are overly parameterized and have significant redundancy . recent studies show that PLMs are highly over-parameterized and robust to pruning .
Approach:	They propose to re-parameter and fine-tune pre-trained language models from a new perspective: Discovery of intrinsic task-specific subspace.
Outcome:	The proposed model can be fine-tuned in the subspace with a small number of free parameters.

Compatibility-Aware Dynamic Fine-Tuning for Large Language Models (2026.acl-long)

Copied to clipboard

Yucheng Zhou, Junwei Sheng, Qianning Wang, Jianbing Shen

Language Models

Challenge:	Recent work attributes optimization instability to the low probability of demonstrations being incompatible with the sample level.
Approach:	They propose a Dynamic Fine-Tuning extension of DFT that controls sample-level optimization variance.
Outcome:	The proposed model can generalize token-level stabilization to the sample level while remaining fully supervised and free of reward modeling.

Parameter-Efficient Fine-Tuning: Is There An Optimal Subset of Parameters to Tune? (2024.findings-eacl)

Copied to clipboard

Max Ploner, Alan Akbik

Language Models

Challenge:	Recent research has illuminated the possibility of selective parameter-efficient fine-tuning, which retains the inference speed of the original model and comes at no additional computational cost.
Approach:	They propose to selectively update only a small subset of parameters during the fine-tuning process, keeping the remaining parameters frozen during training.
Outcome:	The proposed methods retain the inference speed of the original model and come at no additional computational cost.

A Layer-wise Analysis of Supervised Fine-Tuning (2026.acl-long)

Copied to clipboard

Qinghua Zhao, Xueling Gong, Xinyu Chen, Zhongfeng Kang, Xinlu Li

Language Models

Challenge:	Existing methods for fine-tuning ignore depth-dependent heterogeneity of instruction-following . a critical gap remains in understanding where these changes occur across the model's depth and which layers are essential for instruction- following.
Approach:	They propose a method which selectively updates critical intermediate layers . they show that effective alignment is architecturally localized rather than distributed .
Outcome:	The proposed method outperforms standard LoRA up to 10.2% on GSM8K with reduced parameter overhead.

Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models (2025.naacl-long)

Copied to clipboard

Hyegang Son, Yonglak Son, Changhoon Kim, Young Geun Kim

Semantic Text Processing

Challenge:	Existing methods for fine-tuning pre-trained models impose substantial resource usage.
Approach:	They propose a parameter-efficient fine-tuning method that freezes adapters early to reduce resource usage while maintaining performance.
Outcome:	The proposed method reduces memory usage, computation amount, and training time by 42.85%, 34.59%, and 11.82% while maintaining performance.

ATLANTIS: Weak-to-Strong Learning via Importance Sampling (2025.acl-long)

Copied to clipboard

Yi Liu, Guoyin Wang, Shicheng Li, Feifan Song, Xu Sun

Challenge:	ATLANTIS is a new technique to improve the performance of large language models.
Approach:	They propose a new technique to bridge the gap between the distribution of current datasets and the real-world data distribution by using importance sampling.
Outcome:	The proposed technique can bring consistent and significant improvements to models’ performance and can be flexibly transferred among models with different structures.