Papers with PKMs
Approximating Two-Layer Feedforward Networks for Efficient Transformers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work uses sparse Mixtures of Experts (MoEs) to build resource-efficient large language models. |
| Approach: | They propose a general framework that unifies various methods to build two-layer NNs . they propose methods to improve both MoEs and PKMs based on their results . |
| Outcome: | The proposed framework improves both MoEs and product-key memories (PKMs) it shows that MoE's are competitive with dense Transformer-XL on two different scales while being much more resource efficient. |