Papers by George Polovets
Adaptation Approaches for Nearest Neighbor Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Semi-parametric Nearest Neighbor Language Models (kNN-LMs) have produced impressive gains over purely parametric LMs, however, there has been little investigation into adapting such models for new domains. |
| Approach: | They propose to adapt kNN-LMs to expand neighborhood retrieval over an additional adaptation datastore and adapt the weights of retrieved neighbors using a learned Rescorer module. |
| Outcome: | The proposed approach outperforms purely parametric adaptation and zero-shot models and achieves perplexity improvements of 17.1% and 16% across domains. |
Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation (2022.tacl-1)
Copied to clipboard
| Challenge: | Large pretrained language models are often domain- or task-adapted via finetuning or prompting. |
| Approach: | They propose to use domain-adaptive pretraining to prepare large pretrained language models for domain- or task-adaptation by learning to learn the difference between general and adapted PLMs. |
| Outcome: | Experiments on few-shot dialogue completion, low-resource abstractive summarization, and multi-domain language modeling show improvements in adaptation time and performance over finetuning or preparation via domain-adaptive pretraining. |