Papers by Xingkai Yu
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (2024.acl-long)
Copied to clipboard
Damai Dai, Chengqi Deng, Chenggang Zhao, R.x. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.k. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, Wenfeng Liang
| Challenge: | Mixture-of-Experts (MoE) architectures face challenges in ensuring expert specialization . despite the promising performance, scaling language models to an extremely large scale is associated with exceedingly high computational costs. |
| Approach: | They propose an architecture that allows for ultimate expert specialization by segmenting experts into mN ones and activating mK from them. |
| Outcome: | The proposed architecture achieves comparable performance with GShard with 2B parameters and computation. |
Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models (2026.acl-long)
Copied to clipboard
Xin Cheng, Wangding Zeng, Damai Dai, Qinyu Chen, Bingxuan Wang, Zhenda Xie, Kezhao Huang, Xingkai Yu, Zhewen Hao, Han Zhang, Yu-Kun Li, Huishuai Zhang, Dongyan Zhao, Wenfeng Liang
| Challenge: | Mixture-of-Experts (MoE) scales capacity via conditional computation, but lacks knowledge lookup primitive. |
| Approach: | They propose a conditional memory instantiated via Deep Sparse Embedding (DSE) they propose 'u-shaped scaling law' that identifies optimal balance between MoE experts and DSE memory . |
| Outcome: | The proposed model outperforms an iso-parameter and isoFLOPs MoE baseline across knowledge and reasoning benchmarks and is infrastructure-efficient. |