Papers by Isaac Noble
Memory Augmented Language Models through Mixture of Word Experts (2024.naacl-long)
Copied to clipboard
| Challenge: | Increasing the parameter count of language models has been a primary driver of improved model quality, but increasing the model size also increases the cost of training and serving the model. |
| Approach: | They propose to decouple learning capacity and FLOPs by using a mixture-of-experts approach with large knowledge-rich vocabulary based routing functions. |
| Outcome: | The proposed model outperforms the T5 family of models with similar number of FLOPs on knowledge intensive tasks and similar performance to memory augmented approaches. |