Why Do More Experts Fail? A Theoretical Analysis of Model Merging (2026.acl-long)
Copied to clipboard
Zijing Wang, Xingle Xu, YongKang Liu, Yiqun Zhang, Peiqin Lin, Shi Feng, Daling Wang, Xiaocui Yang, Hinrich Schuetze
| Challenge: | Existing methods for model merging struggle to maintain performance gains as the number of merged models increases. |
| Approach: | They propose a Reparameterized Heavy-Tailed method to extend the merged model’s coverage and enhance performance. |
| Outcome: | The proposed method extends the merged model’s coverage and enhances performance on 19 benchmarks, including knowledge-intensive and general-purpose tasks. |
Similar Papers
Will it Merge? On The Causes of Model Mergeability (2026.findings-acl)
Copied to clipboard
| Challenge: | Model merging has emerged as a promising technique for combining fine-tuned models into a single expert model without retraining. |
| Approach: | They propose a model merging technique that preserves weak model knowledge . they define mergeability as a property of model updates that captures how well they retain trained knowledge when merged with other model updates. |
| Outcome: | The proposed method preserves weak knowledge in the base model. |
MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs (2025.naacl-long)
Copied to clipboard
Yuhang Zhou, Giannis Karamanolakis, Victor Soto, Anna Rumshisky, Mayank Kulkarni, Furong Huang, Wei Ai, Jianhua Lu
| Challenge: | State-of-the-art methods for merging expert models with different architectures do not address parameter interference and require extensive fine-tuning to restore performance. |
| Approach: | They propose a method for merging experts with different architectures into a unified Mixture-of-Experts model with a goal of enhancing performance in each domain while retaining effectiveness on general tasks. |
| Outcome: | Experiments across multiple domains show that the proposed methods reduce fine-tuning costs and improve performance over state-of-the-art methods. |
Merging Experts into One: Improving Computational Efficiency of Mixture of Experts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Extensive experiments show that MEO significantly improves computational efficiency . compared to dense networks, sparsely activated networks only employ a few parameters for each input . |
| Approach: | They propose a method that merges multiple experts into one to reduce computation costs . they demonstrate that a sparse Mixture of Experts (MoE) can reduce the cost by activating a small subset of parameters for each input . |
| Outcome: | The proposed approach reduces the computational cost to that of a single expert by 83.3% compared to 82.6% in vanilla MoE. |
Dynamic Fisher-weighted Model Merging via Bayesian Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing merging approaches involve scaling the parameters model-wise or integrating parameter importance parameter-wise. |
| Approach: | They propose a method for merging model-based models at the parameter level without training data or joint training. |
| Outcome: | The proposed model merging framework outperforms baseline models on validation sets. |
A Few More Examples May Be Worth Billions of Parameters (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on few-shot learning for natural language tasks explores the dynamics of scaling up either the number of model parameters or labeled examples while controlling for the other variable by setting it to a constant. |
| Approach: | They explore the dynamics of scaling up the number of model parameters versus the number labeled examples across a wide variety of tasks. |
| Outcome: | The results show that scaling parameters yields performance improvements, while adding examples does not. |
Model Merging and Safety Alignment: One Bad Model Spoils the Bunch (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for merging large language models often overlook safety alignment during merging, leading to misaligned models. |
| Approach: | They propose to combine safety and domain-specific data to optimize model merging techniques . they propose to use this data to maximize model alignment . |
| Outcome: | The proposed method allows for models that excel in both domain expertise and alignment. |
Split-Merge: Scalable and Memory-Efficient Merging of Expert LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | a zero-shot merging framework for large language models consolidates specialized domain experts into a single model without any further training. |
| Approach: | They propose a zero-shot merging framework that consolidates specialized domain experts into a single model without further training. |
| Outcome: | Experiments on code generation, mathematical reasoning, medical question answering, and instruction-following benchmarks confirm the versatility and effectiveness of the proposed framework. |
Practical Guidelines for Model Merging in LLMs Pre-Training (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing studies on model merging have focused on stable learning rate regimes, but its effectiveness during LLM pre-training remains underexplored. |
| Approach: | They systematically investigate model merging across training phases, focusing on the transition from stable to decaying learning rates. |
| Outcome: | The proposed methods improve performance during stable learning rate regimes but diminish under decay, a phe-nomenon that is linked to reduced checkpoint diversity and lower parameter-space variability. |
DivMerge: A divergence-based model merging method for multi-tasking (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for multitask learning struggle with interference between tasks, especially as the number of tasks grows. |
| Approach: | They propose a reference-free method that minimizes the divergence between models' outputs and those of the merged model, automatically balancing task importance. |
| Outcome: | The proposed method outperforms existing methods on classification and generative tasks and remains robust when scaling to more tasks. |
Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing task vector-based model merging methods apply uniform coefficients across all parameters, overlooking varying parameter importance both within and across tasks. |
| Approach: | They propose a sensitivity-guided coefficient adjustment method that optimizes existing model merging techniques by operating at both task-specific and cross-task levels. |
| Outcome: | The proposed method outperforms existing model merging techniques on mistral 7B and LLaMA2 7B/13B models and enables them to outperformed specialized models. |