Papers with parameter-sharing
Sparse Universal Transformer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that use VTs as their backbone model are based on UTs that share parameters across layers and have better compositional generalization. |
| Approach: | They propose to use Sparse Mixture of Experts to reduce UT's computation complexity while retaining its parameter efficiency and generalization ability. |
| Outcome: | The proposed model achieves strong generalization results on formal language tasks and impressive parameter and computation efficiency on standard natural language benchmarks. |
ModularMoE: Fast LLM Customization with Parameter-Sharing Mixture-of-Experts for Low-Resource Settings (2026.findings-acl)
Copied to clipboard
Jiaxing Liu, Qi Qi, Haifeng Sun, Dunjun Li, Zirui Zhuang, Bo He, Xiang Yang, Cong Liu, Jianxin Liao, Jingyu Wang
| Challenge: | Large Language Models impose significant computational and storage burdens on personal devices . existing customization approaches incur excessive computational costs or lead to suboptimal performance . |
| Approach: | They propose a training framework that converts pre-trained LLMs into parameter-sharing MoE models for lightweight deployment. |
| Outcome: | The proposed training framework outperforms state-of-the-art training frameworks at the same sparsity level while delivering up to 2.71 inference speedup. |