| Challenge: | Recent work shows that combining model merging with evolutionary algorithms can boost performance, but there is currently no library for experimenting with different evolutionary algorithms and merging methods. |
| Approach: | They propose an open-source library for evolutionary model merging that enables easy composition of merging methods and evolutionary algorithms while incorporating lightweight fitness estimators to reduce evaluation costs. |
| Outcome: | The proposed library produces competitive results across languages and tasks using modest hardware. |
Similar Papers
Arcee’s MergeKit: A Toolkit for Merging Large Language Models (2024.emnlp-industry)
Copied to clipboard
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, Jacob Solawetz
| Challenge: | Open-source language models can merge their parameters to improve performance and versatility without additional training. |
| Approach: | They propose to integrate model checkpoints into powerful multitask models without additional training. |
| Outcome: | the library has facilitated the merging of thousands of models, contributing to some of the world’s most powerful open-source model checkpoints. |
MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs (2025.naacl-long)
Copied to clipboard
Yuhang Zhou, Giannis Karamanolakis, Victor Soto, Anna Rumshisky, Mayank Kulkarni, Furong Huang, Wei Ai, Jianhua Lu
| Challenge: | State-of-the-art methods for merging expert models with different architectures do not address parameter interference and require extensive fine-tuning to restore performance. |
| Approach: | They propose a method for merging experts with different architectures into a unified Mixture-of-Experts model with a goal of enhancing performance in each domain while retaining effectiveness on general tasks. |
| Outcome: | Experiments across multiple domains show that the proposed methods reduce fine-tuning costs and improve performance over state-of-the-art methods. |
Unlocking the Potential of Model Merging for Low-Resource Languages (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Adapting large language models (LLMs) to new languages requires continual pre-training followed by supervised fine-tuning. |
| Approach: | They propose a model merging solution that integrates LLMs with distinct capabilities into a single model without additional training. |
| Outcome: | The proposed model merging outperforms CT-then-SFT in low-resource languages with scarce data. |
Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Adapting general-purpose language models to new skills is currently expensive . Adaptation to new skill sets requires repeated training or models forget older skills . |
| Approach: | They propose a parallel-train-then-merge procedure that adds new skills to preexisting models in isolation and later merges with the general model. |
| Outcome: | The proposed method is cheaper than retraining models on updated datasets . it improves model compliance with safe prompts while preserving model's ability to refuse dangerous or harmful prompts. |
LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a framework for model merging is proposed without additional training . task vectors from fine-tuned models exhibit a limited number of dominant singular values . |
| Approach: | They propose a framework for model merging based on low-rank estimation of task vectors without access to the base model. |
| Outcome: | The proposed framework improves models without additional training without additional inputs. |
Split-Merge: Scalable and Memory-Efficient Merging of Expert LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | a zero-shot merging framework for large language models consolidates specialized domain experts into a single model without any further training. |
| Approach: | They propose a zero-shot merging framework that consolidates specialized domain experts into a single model without further training. |
| Outcome: | Experiments on code generation, mathematical reasoning, medical question answering, and instruction-following benchmarks confirm the versatility and effectiveness of the proposed framework. |
Knowledge Fusion By Evolving Weights of Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Experimental results on mainstream language models show that Evolver outperforms previous state-of-the-art models by large margins due to the high training costs of large language models. |
| Approach: | They propose a method to integrate multiple models from diverse training scenarios into a unified model. |
| Outcome: | The proposed method outperforms state-of-the-art models on mainstream language models by large margins. |
DivMerge: A divergence-based model merging method for multi-tasking (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for multitask learning struggle with interference between tasks, especially as the number of tasks grows. |
| Approach: | They propose a reference-free method that minimizes the divergence between models' outputs and those of the merged model, automatically balancing task importance. |
| Outcome: | The proposed method outperforms existing methods on classification and generative tasks and remains robust when scaling to more tasks. |
Dynamic Fisher-weighted Model Merging via Bayesian Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing merging approaches involve scaling the parameters model-wise or integrating parameter importance parameter-wise. |
| Approach: | They propose a method for merging model-based models at the parameter level without training data or joint training. |
| Outcome: | The proposed model merging framework outperforms baseline models on validation sets. |
To See a World in a Spark of Neuron: Disentangling Multi-Task Interference for Training-Free Model Merging (2025.emnlp-main)
Copied to clipboard
Zitao Fang, Guodong Du, Shuyang Yu, Yifei Guo, Yiwei Zhang, Yiyao Cao, Jing Li, Ho-Kin Tang, Sim Kuan Goh
| Challenge: | Existing approaches to model merging ignore the fundamental roles of neurons, connectivity and activation. |
| Approach: | They propose a framework that relies on neuronal mechanisms to mitigate task interference . they decomposed task-specific representations into two complementary subspaces . their results offer new insights into mitigating task interference and improving knowledge fusion . |
| Outcome: | The proposed framework reduces task interference within neurons and improves knowledge fusion. |