MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods face the trilemma of performance, data privacy, and computational costs, which hinders their application to LLMs. |
| Approach: | They propose a model-exclusive task arithmetic method for merging GPT-scale models which is data-agnostic and bypasses the heavy search process. |
| Outcome: | The proposed method achieves state-of-the-art performance on multiple tasks while minimizing the average loss difference between the merged model and each individual task model. |
Similar Papers
Training-free LLM Merging for Multi-task Learning (2025.acl-long)
Copied to clipboard
Zichuan Fu, Xian Wu, Yejing Wang, Wanyu Wang, Shanshan Ye, Hongzhi Yin, Yi Chang, Yefeng Zheng, Xiangyu Zhao
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional capabilities across diverse natural language processing tasks. |
| Approach: | They propose a training-free method for unifying different specialized LLMs into a single model using model-wise and layer-wise pruning and scaling. |
| Outcome: | The proposed method outperforms existing merging techniques and surpasses models fine-tuned on combined datasets in most scenarios. |
Improving Training Efficiency and Reducing Maintenance Costs via Language Specific Model Merging (2026.eacl-industry)
Copied to clipboard
| Challenge: | Recent research on merging multilingual multitask models has shown promise in terms of improved quality, but its computational and maintenance efficiency remains unstudied. |
| Approach: | They propose a method for fine-tuning a multilingual large language model . they compare a "retrain-all" approach and a 'train-once, merge-as-needed' approach . |
| Outcome: | The proposed approach reduces training time by up to 50% while maintaining parity in terms of quality. |
Unlocking the Potential of Model Merging for Low-Resource Languages (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Adapting large language models (LLMs) to new languages requires continual pre-training followed by supervised fine-tuning. |
| Approach: | They propose a model merging solution that integrates LLMs with distinct capabilities into a single model without additional training. |
| Outcome: | The proposed model merging outperforms CT-then-SFT in low-resource languages with scarce data. |
LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a framework for model merging is proposed without additional training . task vectors from fine-tuned models exhibit a limited number of dominant singular values . |
| Approach: | They propose a framework for model merging based on low-rank estimation of task vectors without access to the base model. |
| Outcome: | The proposed framework improves models without additional training without additional inputs. |
Efficient Compositional Multi-tasking for On-device Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Adapter parameters provide a mechanism to modify the behavior of machine learning models and have gained significant popularity in the context of large language models (LLMs). |
| Approach: | They propose a benchmark for text-based compositional multi-tasking where multiple tasks are executed simultaneously. |
| Outcome: | The proposed method is optimized for on-device applications where computational resources are limited. |
Towards Better Multi-task Learning: A Framework for Optimizing Dataset Combinations in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Using a neural network, large language models can be trained on multiple tasks, allowing them to perform tasks efficiently. |
| Approach: | They propose a framework that leverages a neural network to select the best dataset combinations for enhancing multi-task learning (MTL) They propose to iteratively refine the selection, greatly improving efficiency while being model-, dataset-, and domain-independent. |
| Outcome: | The proposed framework iteratively refines the selection, greatly improving efficiency, while being model-, dataset-, and domain-independent. |
Towards Unified Task Embeddings Across Multiple Models: Bridging the Gap for Prompt-Based Large Language Models and Beyond (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing task embedding methods rely on fine-tuned, task-specific language models, which hinders their adaptability to prompt-guided Large Language Models (LLMs). |
| Approach: | They propose a framework for unified task embedding that harmonizes task embeds from various models within a single vector space. |
| Outcome: | The proposed framework harmonizes task embeddings from various models within a single vector space. |
Arcee’s MergeKit: A Toolkit for Merging Large Language Models (2024.emnlp-industry)
Copied to clipboard
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, Jacob Solawetz
| Challenge: | Open-source language models can merge their parameters to improve performance and versatility without additional training. |
| Approach: | They propose to integrate model checkpoints into powerful multitask models without additional training. |
| Outcome: | the library has facilitated the merging of thousands of models, contributing to some of the world’s most powerful open-source model checkpoints. |
Split-Merge: Scalable and Memory-Efficient Merging of Expert LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | a zero-shot merging framework for large language models consolidates specialized domain experts into a single model without any further training. |
| Approach: | They propose a zero-shot merging framework that consolidates specialized domain experts into a single model without further training. |
| Outcome: | Experiments on code generation, mathematical reasoning, medical question answering, and instruction-following benchmarks confirm the versatility and effectiveness of the proposed framework. |
Investigating the Potential of Task Arithmetic for Cross-Lingual Transfer (2024.eacl-short)
Copied to clipboard
| Challenge: | Massively multilingual Transformer-based models (MMTs) can learn representations which have a degree of cross-lingual alignment despite being trained using purely unsupervised objectives. |
| Approach: | They propose a modular approach to cross-lingual transfer using task arithmetic . they show that modularity can be achieved even with full model fine-tuning . |
| Outcome: | The proposed approach shows strong performance on multilingual benchmarks encompassing both high-resource and low-resourced languages. |