Papers with LaMemo
LaMemo: Language Modeling with Look-Ahead Memory (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to model long-term dependencies are limited to long texts with thousands of words. |
| Approach: | They propose a look-ahead memory that augments the recurrence memory by attending to the right-side tokens and interpolating with the old memory states to maintain long-term information in the history. |
| Outcome: | Experiments on widely used language modeling benchmarks show that LaMemo outperforms baseline models with recurrence memory. |