Papers with kNN
kNN-BOX: A Unified Framework for Nearest Neighbor Generation (2024.eacl-demo)
Copied to clipboard
| Challenge: | kNN-BOX enables quick development and visualization for novel generation paradigm . Currently, knn-BOx has provided implementation of seven popular kN-MT variants . |
| Approach: | They propose a framework which decomposes the datastore-augmentation approach into three modules . they apply kNN-BOX to machine translation and three other tasks . |
| Outcome: | The proposed framework decomposes the datastore-augmentation approach into three modules . it provides implementation of seven popular kNN-MT variants, covering research from performance enhancement to efficiency optimization. |
Non-Parametric Domain Adaptation for End-to-End Speech Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | End-to-end speech translation (E2E-ST) systems have received increasing attention due to its less error propagation, lower latency and fewer parameters. |
| Approach: | They propose a non-parametric method that leverages in-domain text translation corpus to achieve domain adaptation for E2E-ST systems. |
| Outcome: | The proposed method outperforms the existing in-domain fine-tuning strategies on the Europarl-ST benchmark. |
On Retrieval Augmentation and the Limitations of Language Model Training (2024.naacl-short)
Copied to clipboard
| Challenge: | Recent efforts to improve the performance of language models (LMs) have focused on scaling up model and training data size, though with steep accompanying energy and compute resource costs. |
| Approach: | They propose to augment a language model with k-nearest neighbors retrieval on its training data to reduce its perplexity. |
| Outcome: | The proposed model reduces storage costs by over 25x compared to traditional retrieval methods for GPT-2 and Mistral 7B . |
Adaptive Nearest Neighbor Machine Translation (2021.acl-short)
Copied to clipboard
| Challenge: | kNN-MT uses pre-trained NMT model with token-level k-nearest-neighbor retrieval to improve translation accuracy. |
| Approach: | They propose a method that combines a pre-trained NMT model with token-level k-nearest-neighbor retrieval to improve translation accuracy. |
| Outcome: | The proposed method outperforms the existing model on four benchmark datasets and is open-source. |
Neurocache: Efficient Vector Retrieval for Long-range Language Modeling (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent research shows that retrieval-augmented models with shorter contexts (4K tokens) can match the performance of models with longer contexts (16K/32K token) |
| Approach: | They introduce an approach to extend the effective context size of large language models by using an external vector cache to store past states. |
| Outcome: | The proposed method improves on models trained from scratch and pre-trained models. |
Contrastive Learning-Enhanced Nearest Neighbor Mechanism for Multi-Label Text Classification (2022.acl-short)
Copied to clipboard
| Challenge: | Existing methods for multi-label text classification neglect the knowledge from the existing similar instances when predicting labels of a specific text. |
| Approach: | They propose a k nearest neighbor mechanism which retrieves several neighbor instances and interpolates the model output with their labels. |
| Outcome: | Extensive experiments show that the proposed method can bring significant performance improvements to multiple MLTC models including state-of-the-art pretrained and non-pretrained ones. |
Predicting Numerals in Text Using Nearest Neighbor Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | naive language models treat numerals as string tokens, resulting in difficulty in acquiring commonsense . kNN-LM is an extension of pre-trained neural LMs with the k-nearest neighbor (kNN) search . |
| Approach: | They apply k-nearest neighbor LM to a masked numeral prediction task . they found it is effective for fine-grained predictions of numerals from context . |
| Outcome: | The retrieval-based method is effective for fine-grained numeral prediction from context . it improves accuracy for the OOV numerals, the study shows . |
BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) capture a diverse range of linguistic and factual knowledge without the use of finetuning. |
| Approach: | They combine BERT with a traditional information retrieval step and a kNN search to improve the recall of facts encountered during training. |
| Outcome: | The proposed approach outperforms baselines on cloze-style QA by large margins without further training. |
Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMax (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing kNN-based augmentation techniques blindly incorporate all samples, but MiniMax-kNN uses a subset of augmented samples to maximize KL-divergence between teacher and student models. |
| Approach: | They propose a semi-supervised approach to augmented data augmentation using kNN. |
| Outcome: | The proposed method outperforms existing kNN-based augmentation techniques on several classification tasks and requires fewer augmented examples and less computation to achieve superior performance. |
Contextual Modeling for Document-level ASR Error Correction (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing work on document-level ASR error correction ignores contextual information . however, there are limited studies on incorporating contextual information into AEC . |
| Approach: | They propose a context-aware method that retrieves contextual information from a datastore . they use two English and two Chinese datasets to model document-level AEC . |
| Outcome: | The proposed model can utilize contextual information to improve document-level AEC . the data store containing contextual information provides even better results . |
Non-Parametric Unsupervised Domain Adaptation for Neural Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | kNN-MT is a non-parametric method that uses nearest neighbor retrieval to translate out-of-domain sentences, rare words, etc. |
| Approach: | They propose a framework that directly uses in-domain monolingual sentences to build an effective datastore for k-nearest-neighbor retrieval. |
| Outcome: | The proposed framework improves translation accuracy with target-side monolingual data while achieving comparable performance with back-translation. |
kNN For Whisper And Its Effect On Bias And Speaker Adaptation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Speech recognition performance varies by language, domain, speaker characteristics such as accent . fine-tuning models on any of these categories may lead to catastrophic forgetting . |
| Approach: | They propose a nonparametric method that adapts to any category without training the model. |
| Outcome: | The proposed method improves the performance of a transformer end-to-end speech model on transcribed audio in over 680,000 hours of training data on a wide variety of languages. |
Nearest Neighbor Knowledge Distillation for Neural Machine Translation (2022.naacl-main)
Copied to clipboard
| Challenge: | k-nearest-neighbor machine translation (kNN-MT) is a state-of-the-art machine translation technique . however, it requires conducting kNN searches for each decoding step, which increases the cost of decoding . |
| Approach: | They propose to move the time-consuming kNN search forward to the preprocessing phase and introduce k Nearest Neighbor Knowledge Distillation (kNN-KD) that trains the base NMT model to directly learn the knowledge of kN. |
| Outcome: | The proposed method improves over the state-of-the-art model while maintaining the same training and decoding speed as the standard model. |
Choose Your Transformer: Improved Transferability Estimation of Transformer Models on Classification Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for NLP tasks require fine-tuning, but it is computationally infeasible. |
| Approach: | They propose an approach that inexpensively estimates a ranking of the expected performance of a given set of transformer language models for a specific task. |
| Outcome: | The proposed model improves the Pearson correlation coefficient between the true model ranks and the estimate. |
FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging (2021.emnlp-main)
Copied to clipboard
| Challenge: | despite popularity of influence functions, their computational cost does not scale well with model and training data size. |
| Approach: | They propose a fast parallel variant that approximates the “influences” of training data-points for test predictions. |
| Outcome: | The proposed method achieves about 80X speedup while being highly correlated with the original influence values. |