FroM: Frobenius Norm-Based Data-Free Adaptive Model Merging (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new adaptive merging method is proposed to improve fine-tuning performance . traditional methods often encounter task interference when merging full fine-uning models . |
| Approach: | They propose an adaptive merging method that directly measures model parameters using the Frobenius norm . |
| Outcome: | The proposed method outperforms baseline methods in various fine-tuning scenarios. |
Similar Papers
RankMean: Module-Level Importance Score for Merging Fine-tuned LLM Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Traditionally, developing new language models involves fine-tuning pre-trained LMs . model merging is a cost-effective alternative to fine-timing LM models for multiple tasks . |
| Approach: | They propose an algorithm for merging fine-tuned language models without additional training. |
| Outcome: | The proposed algorithm outperforms baseline methods on multiple benchmarks. |
To See a World in a Spark of Neuron: Disentangling Multi-Task Interference for Training-Free Model Merging (2025.emnlp-main)
Copied to clipboard
Zitao Fang, Guodong Du, Shuyang Yu, Yifei Guo, Yiwei Zhang, Yiyao Cao, Jing Li, Ho-Kin Tang, Sim Kuan Goh
| Challenge: | Existing approaches to model merging ignore the fundamental roles of neurons, connectivity and activation. |
| Approach: | They propose a framework that relies on neuronal mechanisms to mitigate task interference . they decomposed task-specific representations into two complementary subspaces . their results offer new insights into mitigating task interference and improving knowledge fusion . |
| Outcome: | The proposed framework reduces task interference within neurons and improves knowledge fusion. |
Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing task vector-based model merging methods apply uniform coefficients across all parameters, overlooking varying parameter importance both within and across tasks. |
| Approach: | They propose a sensitivity-guided coefficient adjustment method that optimizes existing model merging techniques by operating at both task-specific and cross-task levels. |
| Outcome: | The proposed method outperforms existing model merging techniques on mistral 7B and LLaMA2 7B/13B models and enables them to outperformed specialized models. |
Dynamic Fisher-weighted Model Merging via Bayesian Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing merging approaches involve scaling the parameters model-wise or integrating parameter importance parameter-wise. |
| Approach: | They propose a method for merging model-based models at the parameter level without training data or joint training. |
| Outcome: | The proposed model merging framework outperforms baseline models on validation sets. |
LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a framework for model merging is proposed without additional training . task vectors from fine-tuned models exhibit a limited number of dominant singular values . |
| Approach: | They propose a framework for model merging based on low-rank estimation of task vectors without access to the base model. |
| Outcome: | The proposed framework improves models without additional training without additional inputs. |
Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models fail due to performance degradation . existing methods fail for models fine- tuned with low-rank adaptation . |
| Approach: | They propose to constrain the LoRA subspace prior to fine-tuning to ensure that updates relevant to one task do not adversely shift outputs for others. |
| Outcome: | The proposed method can integrate with most existing merging algorithms, reducing unintended interference among tasks. |
LM-Cocktail: Resilient Tuning of Language Models via Model Merging (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models are continually fine-tuned to better support downstream applications. however, this operation may result in significant performance degeneration on general perspectives. |
| Approach: | They propose a method which enables pre-trained language models to stay resilient in general perspectives. |
| Outcome: | The proposed model achieves strong empirical performance in the whole scope of general tasks while preserving a superior capacity in its targeted domain. |
Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies suggest that the order of training samples can affect model performance, but this is not the case. |
| Approach: | They propose to merge supervised fine-tuning models with different data orders to mitigate this imbalance by parameter merging. |
| Outcome: | The proposed method outperforms the weighted-average method on five datasets. |
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. |
| Approach: | They propose a framework that can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks. |
| Outcome: | The proposed framework improves performance on the well-known GLUE benchmark while adding only 0.29% parameters per task. |
Practical Guidelines for Model Merging in LLMs Pre-Training (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing studies on model merging have focused on stable learning rate regimes, but its effectiveness during LLM pre-training remains underexplored. |
| Approach: | They systematically investigate model merging across training phases, focusing on the transition from stable to decaying learning rates. |
| Outcome: | The proposed methods improve performance during stable learning rate regimes but diminish under decay, a phe-nomenon that is linked to reduced checkpoint diversity and lower parameter-space variability. |