| Challenge: | Non-parametric neural language models (NLMs) learn text distributions by memorizing training data points. |
| Approach: | They propose to use an external datastore to learn from a non-parametric language model. |
| Outcome: | The proposed methods achieve up to a 6x speed-up in inference speed while retaining comparable performance. |
Similar Papers
Adaptation Approaches for Nearest Neighbor Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Semi-parametric Nearest Neighbor Language Models (kNN-LMs) have produced impressive gains over purely parametric LMs, however, there has been little investigation into adapting such models for new domains. |
| Approach: | They propose to adapt kNN-LMs to expand neighborhood retrieval over an additional adaptation datastore and adapt the weights of retrieved neighbors using a learned Rescorer module. |
| Outcome: | The proposed approach outperforms purely parametric adaptation and zero-shot models and achieves perplexity improvements of 17.1% and 16% across domains. |
kNN-CM: A Non-parametric Inference-Phase Adaptation of Parametric Text Classifiers (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on text-discriminating properties of semi-parametric models have not been done on non-parameter models. |
| Approach: | They propose an inference-phase approach that incorporates a neighborhood search into a model to enhance the capacity of a pre-trained parametric text classifier. |
| Outcome: | The proposed model improves performance on eight SuperGLUE tasks, three adversarial natural language inference datasets, 11 question-answering (QA) datasets and two sentiment classification datasets. |
Non-Parametric Unsupervised Domain Adaptation for Neural Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | kNN-MT is a non-parametric method that uses nearest neighbor retrieval to translate out-of-domain sentences, rare words, etc. |
| Approach: | They propose a framework that directly uses in-domain monolingual sentences to build an effective datastore for k-nearest-neighbor retrieval. |
| Outcome: | The proposed framework improves translation accuracy with target-side monolingual data while achieving comparable performance with back-translation. |
Nearest Neighbor Zero-Shot Inference (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using non-parametric memory for retrieval-augmented language models yields significant performance boosts over strong zeroshot baselines. |
| Approach: | They propose a retrieval-augmented language model with fuzzy verbalizers that expands the verbalizes that define different end-task class labels. |
| Outcome: | The proposed model outperforms non-retrieval-augmented language models on perplexity-based evaluations but gains transfer marginally . the main challenge is to achieve coverage of the verbalizer tokens that define the different end-task class labels. |
Efficient Cluster-Based k-Nearest-Neighbor Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | k-Nearest-Neighbor Machine Translation (kNN-MT) is a non-parametric solution for domain adaptation . previous studies have shown that kNN retrieval is at the expense of high latency . |
| Approach: | They propose to use clustering to improve retrieval efficiency by combining a non-parametric MT with an in-domain feature-based retrieval module. |
| Outcome: | The proposed method reduces translation latency by 57% while maintaining the most useful information of the original datastore. |
Long-Tail Crisis in Nearest Neighbor Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Prior studies have shown that kNN-LM can retrieve long-tail contexts, leaving the model’s performance underexplored in estimating the probabilities of long-tailed target tokens. |
| Approach: | They investigate the behavior of kNN-LM on low-frequency tokens, examining prediction probability, retrieval accuracy, and token distribution in the datastore. |
| Outcome: | The proposed model improves the perplexity of given text by directly accessing a large datastore built from any text data during inference. |
Neural Natural Language Inference Models Enhanced with External Knowledge (P18-1)
Copied to clipboard
| Challenge: | Existing datasets that allow for complex models to be trained are limited . if data is not available, can machines learn all knowledge needed to perform natural language inference? |
| Approach: | They propose to enrich neural natural language inference models with external knowledge . they propose to use this knowledge to build NLI models to leverage it . |
| Outcome: | The proposed models improve on the SNLI and MultiNLI datasets. |
Great Memory, Shallow Reasoning: Limits of kNN-LMs (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing models trained on poor quality data have shown strong performance in language modeling and some downstream benchmarks. |
| Approach: | They evaluate kNN-LMs on a diverse set of tasks and evaluate their performance. |
| Outcome: | The proposed extension could improve on a variety of tasks, but it fails to perform on reasoning tasks that require integrating multiple pieces of information. |
Efficient Nearest Neighbor based Uncertainty Estimation for Natural Language Processing Tasks (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for uncertainty estimation are inadequate for safety-critical applications. |
| Approach: | They propose a method that uses the distances from neighbors and the ratio of labels in neighbors to estimate uncertainty. |
| Outcome: | The proposed method outperforms baseline and density-based methods in calibration and uncertainty metrics. |
Best Practices for Data-Efficient Modeling in NLG:How to Train Production-Ready Neural Models with Less Data (2020.coling-industry)
Copied to clipboard
Ankit Arun, Soumya Batra, Vikas Bhardwaj, Ashwini Challa, Pinar Donmez, Peyman Heidari, Hakan Inan, Shashank Jain, Anuj Kumar, Shawn Mei, Karthik Mohan, Michael White
| Challenge: | Natural language generation (NLG) is a critical component in conversational systems . Traditionally, NLG components have been deployed using template-based solutions . however, deployment of such model-based systems has been challenging due to high latency and data needs. |
| Approach: | They propose a family of techniques to deploy data-efficient neural solutions for NLG in conversational systems to production. |
| Outcome: | The proposed techniques achieve production quality with light-weight neural network models using fraction of the data needed otherwise. |