Approximating Two-Layer Feedforward Networks for Efficient Transformers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work uses sparse Mixtures of Experts (MoEs) to build resource-efficient large language models. |
| Approach: | They propose a general framework that unifies various methods to build two-layer NNs . they propose methods to improve both MoEs and PKMs based on their results . |
| Outcome: | The proposed framework improves both MoEs and product-key memories (PKMs) it shows that MoE's are competitive with dense Transformer-XL on two different scales while being much more resource efficient. |
Similar Papers
Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large and sparse feed-forward layers (S-FFN) have proven effective in scaling up the model size for pretraining large language models. |
| Approach: | They compare S-FFN architectures for language modeling and compare their performance and efficiency . they found a simpler selection method that selects blocks through their mean aggregated hidden states . |
| Outcome: | The proposed model size and selection method achieve lower perplexity in language model pretraining compared to existing MoE architectures. |
A Closer Look into Mixture-of-Experts in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Mixture-of-experts (MoE) architectures are gaining increasing attention for their unique properties and remarkable performance. |
| Approach: | They propose a mixture-of-experts architecture that allows for model scaling without sacrificing computational efficiency. |
| Outcome: | The proposed model increases model size without sacrificing computational efficiency . the proposed model is modular and can be used by a broad spectrum of practitioners . |
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge. |
| Approach: | They propose to convert a model into its MoE version with the same parameters and build expert routers to decide which experts will be used for each input. |
| Outcome: | The proposed model can use 10% to 30% of FFN parameters while maintaining over 95% original performance. |
AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation (2023.findings-acl)
Copied to clipboard
Ganesh Jawahar, Subhabrata Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks Lakshmanan, V.S., Ahmed Hassan Awadallah, Sebastien Bubeck, Jianfeng Gao
| Challenge: | Existing MoE designs do not consider computational constraints (e.g., FLOPs, latency) Existing works in MoE consider homogeneous design where the same number of experts of the same size are placed uniformly throughout the network. |
| Approach: | They propose a framework for designing heterogeneous MoEs under computational constraints. |
| Outcome: | The proposed framework achieves 4x inference speedup and FLOPs reduction over manual models and within 1 BLEU point of MoE SwitchTransformer over benchmark datasets for NMT. |
Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) LLMs achieve higher performance with fewer active parameters, but are still difficult to deploy due to their immense parameter sizes. |
| Approach: | They propose expert-level sparsification techniques to enhance the deployment efficiency of large language models by introducing plug-and-play expert pruning and skipping techniques. |
| Outcome: | The proposed methods reduce model sizes and increase inference speed while maintaining satisfactory performance across a wide range of tasks. |
Breaking ReLU Barrier: Generalized MoEfication for Dense Pretrained Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to convert pretrained dense models to MoEs are limited to ReLU-based models with natural sparsity. |
| Approach: | They propose a G-MoEfication approach for arbitrary dense models where activation sparsity assumptions no longer hold. |
| Outcome: | The proposed method reduces the inference cost associated with dense models by sparsely activating experts. |
Scaling Vision-Language Models with Sparse Mixture of Experts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a study explores the effectiveness of mixture-of-experts (MoE) techniques in scaling vision-language models . alayrac and colleagues demonstrate the effectiveness and performance of MoE in scaling VLMs . |
| Approach: | They propose to use sparsely-gated mixture-of-experts techniques to scale vision-language models . they show that MoE can achieve state-of the-art performance over dense models a range of benchmarks . |
| Outcome: | The proposed approach achieves state-of-the-art performance over dense models of equivalent computational cost. |
Efficient Large Scale Language Modeling with Mixtures of Experts (2022.emnlp-main)
Copied to clipboard
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, Veselin Stoyanov
| Challenge: | Mixture of Experts layers (MoEs) enable efficient scaling of language models . large autoregressive language models such as GPT-3 can be adapted to a wide range of tasks . |
| Approach: | They propose to use Mixture of Experts layers to enable efficient scaling of language models . they find that MoEs are substantially more compute efficient than dense models compared to MoE models - but only when they are more modestly trained . |
| Outcome: | The proposed model outperforms dense models in a wide range of tasks and domains. |
Bag of Tricks for Sparse Mixture-of-Experts: A Benchmark Across Reasoning, Efficiency, and Safety (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks focus on isolated aspects of MoE, with conflicting conclusions . a lack of consensus on optimal design choices is limiting to specific aspects of the model. |
| Approach: | They propose to evaluate two popular MoE backbones across four dimensions of design choices . they find token-level routing and z-loss regularization improve reasoning performance . |
| Outcome: | The proposed framework evaluates two popular MoE backbones on over eight metrics. |
Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules (2024.emnlp-main)
Copied to clipboard
| Challenge: | Empirical results show that MoMs consistently outperform vanilla transformers . |
| Approach: | They propose an architecture that allows for a mixture-of-modules computation that uses a finite set of modules defined by multi-head attention and feed-forward networks. |
| Outcome: | The proposed architecture outperforms vanilla Transformers and their variants in multiple ways. |