Papers by Nathan Hu
Meta-Learning Online Adaptation of Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models encode impressively broad world knowledge in their parameters, but the knowledge in static models becomes stale. |
| Approach: | They propose to use a small, autoregressive model to reweight the language modeling loss for each token during online fine-tuning to maximize the out-of-date model’s ability to answer questions about a document. |
| Outcome: | The proposed approach improves the model's ability to answer questions on streams of thousands of documents compared with fine-tuning and baseline heuristics. |