Papers by Amarda Shehu
Birdie: Advancing State Space Language Modeling with Dynamic Mixtures of Training Objectives (2024.emnlp-main)
Copied to clipboard
| Challenge: | Efficient state space models struggle with tasks requiring in-context retrieval, such as text copying and associative recall, limiting their usefulness in practical settings. |
| Approach: | They propose a training procedure that improves the performance of SSMs on retrieval-intensive tasks such as phone book lookup, long paragraph question-answering, and infilling tasks. |
| Outcome: | The proposed training procedure improves performance on retrieval-intensive tasks that challenge current SSMs, such as phone book lookup, long paragraph question-answering, and infilling tasks. |