Papers by Shengzhuang Chen
Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts (2025.acl-long)
Copied to clipboard
| Challenge: | Sparse Interpolated Mixture-of-Experts (SIMoE) instruction-tuning is an end-to-end algorithm designed to fine-tune a dense pre-trained Large Language Model (LLM) into a MoE-style model that possesses capabilities in multiple specialized domains. |
| Approach: | They propose an algorithm to fine-tune a dense pre-trained Large Language Model into a MoE-style model that possesses capabilities in multiple specialized domains. |
| Outcome: | The proposed algorithm achieves state-of-the-art on common instruction-tuning benchmarks while maintaining an optimal performance-compute trade-off compared to baselines. |