Arcee’s MergeKit: A Toolkit for Merging Large Language Models (2024.emnlp-industry)
Copied to clipboard
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, Jacob Solawetz
| Challenge: | Open-source language models can merge their parameters to improve performance and versatility without additional training. |
| Approach: | They propose to integrate model checkpoints into powerful multitask models without additional training. |
| Outcome: | the library has facilitated the merging of thousands of models, contributing to some of the world’s most powerful open-source model checkpoints. |
Similar Papers
Mergenetic: a Simple Evolutionary Model Merging Library (2025.acl-demo)
Copied to clipboard
| Challenge: | Recent work shows that combining model merging with evolutionary algorithms can boost performance, but there is currently no library for experimenting with different evolutionary algorithms and merging methods. |
| Approach: | They propose an open-source library for evolutionary model merging that enables easy composition of merging methods and evolutionary algorithms while incorporating lightweight fitness estimators to reduce evaluation costs. |
| Outcome: | The proposed library produces competitive results across languages and tasks using modest hardware. |
MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods face the trilemma of performance, data privacy, and computational costs, which hinders their application to LLMs. |
| Approach: | They propose a model-exclusive task arithmetic method for merging GPT-scale models which is data-agnostic and bypasses the heavy search process. |
| Outcome: | The proposed method achieves state-of-the-art performance on multiple tasks while minimizing the average loss difference between the merged model and each individual task model. |
Training-free LLM Merging for Multi-task Learning (2025.acl-long)
Copied to clipboard
Zichuan Fu, Xian Wu, Yejing Wang, Wanyu Wang, Shanshan Ye, Hongzhi Yin, Yi Chang, Yefeng Zheng, Xiangyu Zhao
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional capabilities across diverse natural language processing tasks. |
| Approach: | They propose a training-free method for unifying different specialized LLMs into a single model using model-wise and layer-wise pruning and scaling. |
| Outcome: | The proposed method outperforms existing merging techniques and surpasses models fine-tuned on combined datasets in most scenarios. |
Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Adapting general-purpose language models to new skills is currently expensive . Adaptation to new skill sets requires repeated training or models forget older skills . |
| Approach: | They propose a parallel-train-then-merge procedure that adds new skills to preexisting models in isolation and later merges with the general model. |
| Outcome: | The proposed method is cheaper than retraining models on updated datasets . it improves model compliance with safe prompts while preserving model's ability to refuse dangerous or harmful prompts. |
Unlocking the Potential of Model Merging for Low-Resource Languages (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Adapting large language models (LLMs) to new languages requires continual pre-training followed by supervised fine-tuning. |
| Approach: | They propose a model merging solution that integrates LLMs with distinct capabilities into a single model without additional training. |
| Outcome: | The proposed model merging outperforms CT-then-SFT in low-resource languages with scarce data. |
MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs (2025.naacl-long)
Copied to clipboard
Yuhang Zhou, Giannis Karamanolakis, Victor Soto, Anna Rumshisky, Mayank Kulkarni, Furong Huang, Wei Ai, Jianhua Lu
| Challenge: | State-of-the-art methods for merging expert models with different architectures do not address parameter interference and require extensive fine-tuning to restore performance. |
| Approach: | They propose a method for merging experts with different architectures into a unified Mixture-of-Experts model with a goal of enhancing performance in each domain while retaining effectiveness on general tasks. |
| Outcome: | Experiments across multiple domains show that the proposed methods reduce fine-tuning costs and improve performance over state-of-the-art methods. |
Split-Merge: Scalable and Memory-Efficient Merging of Expert LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | a zero-shot merging framework for large language models consolidates specialized domain experts into a single model without any further training. |
| Approach: | They propose a zero-shot merging framework that consolidates specialized domain experts into a single model without further training. |
| Outcome: | Experiments on code generation, mathematical reasoning, medical question answering, and instruction-following benchmarks confirm the versatility and effectiveness of the proposed framework. |
A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM𝛥 Integration into Upcycled MoE (2026.acl-long)
Copied to clipboard
Hao Zhou, Tianhao Li, Zhijun Wang, Shuaijie She, Linjuan Wu, Hao-Ran Wei, Baosong Yang, Jiajun Chen, Shujian Huang
| Challenge: | Large Language Models (LLMs) are expensive and require extensive Continued Pre-Training and data-intensive alignment to expand. |
| Approach: | They propose a method which upcycles a dense model into a Mixture-of-Experts architecture, allocating different experts to different languages. |
| Outcome: | Experiments show that the proposed model upcycles a dense model into a Mixture-of-Experts(MoE) architecture, allocating different experts to different languages. |
Enhancing Multilingual Reasoning via Steerable Model Merging (2026.findings-acl)
Copied to clipboard
Zhuoran Li, Rui Xu, Jian Yang, Junnan Liu, Zhijun Chen, Qianren Mao, Hongcheng Guo, Jiaheng Liu, Likang Xiao, Ming LI, Xiaojie Wang
| Challenge: | Model merging is an effective technique for composing the capabilities of a multilingual model and a reasoning model. |
| Approach: | They propose a model merging framework that modulates the contribution of each source model. |
| Outcome: | Experiments show that the proposed model merging framework outperforms strong baselines on multilingual reasoning benchmarks across 21 different languages. |
OpenUE: An Open Toolkit of Universal Extraction from Text (2020.emnlp-demos)
Copied to clipboard
Ningyu Zhang, Shumin Deng, Zhen Bi, Haiyang Yu, Jiacheng Yang, Mosha Chen, Fei Huang, Wei Zhang, Huajun Chen
| Challenge: | a large number of natural language processing tasks focus on token-level or sentence-level understandings. |
| Approach: | They propose an open-source and extensible toolkit for various extraction tasks . they deploy an online demo with restful APIs to support real-time extraction . |
| Outcome: | The proposed model can be used to extract information from text without training and deployment. |