LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a framework for model merging is proposed without additional training . task vectors from fine-tuned models exhibit a limited number of dominant singular values . |
| Approach: | They propose a framework for model merging based on low-rank estimation of task vectors without access to the base model. |
| Outcome: | The proposed framework improves models without additional training without additional inputs. |
Similar Papers
PRIME: Ultra-Low-Rank Principal–Residual Model Merging (2026.findings-acl)
Copied to clipboard
Seung-Ho Lee, Kyungsu Lee, Bazarvaani Zuchi, Jeongmin Ahn, Insuk Seo, Donghyeon Jeon, Inho Kang, Seung-Hoon Na
| Challenge: | Existing methods for model merging have been limited by task-specific performance and task-related tasks. |
| Approach: | They propose an ultra-low-rank principal-residual model merging framework that decomposes task vector merging into two stages. |
| Outcome: | Experiments on eight natural language processing tasks show that PRIME outperforms existing models while preserving the task-specific capabilities of the original models. |
Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models fail due to performance degradation . existing methods fail for models fine- tuned with low-rank adaptation . |
| Approach: | They propose to constrain the LoRA subspace prior to fine-tuning to ensure that updates relevant to one task do not adversely shift outputs for others. |
| Outcome: | The proposed method can integrate with most existing merging algorithms, reducing unintended interference among tasks. |
Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing task vector-based model merging methods apply uniform coefficients across all parameters, overlooking varying parameter importance both within and across tasks. |
| Approach: | They propose a sensitivity-guided coefficient adjustment method that optimizes existing model merging techniques by operating at both task-specific and cross-task levels. |
| Outcome: | The proposed method outperforms existing model merging techniques on mistral 7B and LLaMA2 7B/13B models and enables them to outperformed specialized models. |
Unlocking the Potential of Model Merging for Low-Resource Languages (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Adapting large language models (LLMs) to new languages requires continual pre-training followed by supervised fine-tuning. |
| Approach: | They propose a model merging solution that integrates LLMs with distinct capabilities into a single model without additional training. |
| Outcome: | The proposed model merging outperforms CT-then-SFT in low-resource languages with scarce data. |
To See a World in a Spark of Neuron: Disentangling Multi-Task Interference for Training-Free Model Merging (2025.emnlp-main)
Copied to clipboard
Zitao Fang, Guodong Du, Shuyang Yu, Yifei Guo, Yiwei Zhang, Yiyao Cao, Jing Li, Ho-Kin Tang, Sim Kuan Goh
| Challenge: | Existing approaches to model merging ignore the fundamental roles of neurons, connectivity and activation. |
| Approach: | They propose a framework that relies on neuronal mechanisms to mitigate task interference . they decomposed task-specific representations into two complementary subspaces . their results offer new insights into mitigating task interference and improving knowledge fusion . |
| Outcome: | The proposed framework reduces task interference within neurons and improves knowledge fusion. |
LM-Cocktail: Resilient Tuning of Language Models via Model Merging (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models are continually fine-tuned to better support downstream applications. however, this operation may result in significant performance degeneration on general perspectives. |
| Approach: | They propose a method which enables pre-trained language models to stay resilient in general perspectives. |
| Outcome: | The proposed model achieves strong empirical performance in the whole scope of general tasks while preserving a superior capacity in its targeted domain. |
STAR: Spectral Truncation and Rescale for Model Merging (2025.naacl-short)
Copied to clipboard
| Challenge: | Model merging is an efficient way of obtaining a multi-task model from several pretrained models without further fine-tuning. |
| Approach: | They propose a model merging technique that aims at mitigating "merging conflicts" by truncating small components in the respective spectral spaces and then an automatic parameter rescaling scheme to retain the nuclear norm of the original matrix. |
| Outcome: | The proposed model outperforms baseline models on flan-T5 by 4.2% and is robust to hyperparamater choice. |
Enhancing Multilingual Reasoning via Steerable Model Merging (2026.findings-acl)
Copied to clipboard
Zhuoran Li, Rui Xu, Jian Yang, Junnan Liu, Zhijun Chen, Qianren Mao, Hongcheng Guo, Jiaheng Liu, Likang Xiao, Ming LI, Xiaojie Wang
| Challenge: | Model merging is an effective technique for composing the capabilities of a multilingual model and a reasoning model. |
| Approach: | They propose a model merging framework that modulates the contribution of each source model. |
| Outcome: | Experiments show that the proposed model merging framework outperforms strong baselines on multilingual reasoning benchmarks across 21 different languages. |
Model Merging and Safety Alignment: One Bad Model Spoils the Bunch (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for merging large language models often overlook safety alignment during merging, leading to misaligned models. |
| Approach: | They propose to combine safety and domain-specific data to optimize model merging techniques . they propose to use this data to maximize model alignment . |
| Outcome: | The proposed method allows for models that excel in both domain expertise and alignment. |
Mergenetic: a Simple Evolutionary Model Merging Library (2025.acl-demo)
Copied to clipboard
| Challenge: | Recent work shows that combining model merging with evolutionary algorithms can boost performance, but there is currently no library for experimenting with different evolutionary algorithms and merging methods. |
| Approach: | They propose an open-source library for evolutionary model merging that enables easy composition of merging methods and evolutionary algorithms while incorporating lightweight fitness estimators to reduce evaluation costs. |
| Outcome: | The proposed library produces competitive results across languages and tasks using modest hardware. |