Papers with MNNs
Unlocking Emergent Modularity in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing MNNs are explicit, with predefined modular architectures and individual modules expected to implement distinct functions. |
| Approach: | They propose to unlock emergent modularity in language models by fine-tuning them as Mixture-of-Experts (MoEs) EMoE is robust to various configurations and can scale up to Large Language Models . |
| Outcome: | The proposed models can be fine-tuned as Mixture-of-Expert (MoE) counterparts without introducing any extra parameters. |