Nearest Neighbor Knowledge Distillation for Neural Machine Translation (2022.naacl-main)
Copied to clipboard
| Challenge: | k-nearest-neighbor machine translation (kNN-MT) is a state-of-the-art machine translation technique . however, it requires conducting kNN searches for each decoding step, which increases the cost of decoding . |
| Approach: | They propose to move the time-consuming kNN search forward to the preprocessing phase and introduce k Nearest Neighbor Knowledge Distillation (kNN-KD) that trains the base NMT model to directly learn the knowledge of kN. |
| Outcome: | The proposed method improves over the state-of-the-art model while maintaining the same training and decoding speed as the standard model. |
Similar Papers
Domain-Aware k-Nearest-Neighbor Knowledge Distillation for Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to transfer knowledge from kNN datastore into new models are expensive and arbitrarily transfer knowledge. |
| Approach: | They propose a domain-aware method which filters out domain-relevant neighborhood knowledge for learning in the distillation process. |
| Outcome: | The proposed method achieves state-of-the-art on four domain translation tasks. |
Subset Retrieval Nearest Neighbor Machine Translation (2023.acl-long)
Copied to clipboard
| Challenge: | k-nearest-neighbor machine translation (kNN-MT) is a new approach to improve NMT performance without additional training. |
| Approach: | They propose a method that integrates example-search into the decoding algorithm to improve neighbor token retrieval. |
| Outcome: | The proposed method achieves a speed-up of up to 132.2 times and an improvement in BLEU score of up 1.6 compared with kNN-MT in the WMT’19 translation task and the domain adaptation tasks in De-En and En-Ja. |
Fast Nearest Neighbor Machine Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Fast kNN-MT uses the entire corpus as the datastore for the nearest neighbor search . knn-MT is two-orders slower than vanilla MT models . |
| Approach: | They propose a fast kNN-MT model that uses the entire corpus as the datastore for nearest neighbor search. |
| Outcome: | The proposed model is two-orders faster than kNN-MT and is only two times slower than the standard model. |
Adaptive Nearest Neighbor Machine Translation (2021.acl-short)
Copied to clipboard
| Challenge: | kNN-MT uses pre-trained NMT model with token-level k-nearest-neighbor retrieval to improve translation accuracy. |
| Approach: | They propose a method that combines a pre-trained NMT model with token-level k-nearest-neighbor retrieval to improve translation accuracy. |
| Outcome: | The proposed method outperforms the existing model on four benchmark datasets and is open-source. |
Towards Robust k-Nearest-Neighbor Machine Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | k-Nearest-Neighbor Machine Translation (kNN-MT) is a popular research paradigm in machine translation. |
| Approach: | They propose a confidence-enhanced kNN-MT model with robust training to reduce noise . they introduce NMT confidence to refine the modeling of important components of kN-MT . |
| Outcome: | The proposed model improves on four benchmark datasets and is robust to training. |
Efficient Cluster-Based k-Nearest-Neighbor Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | k-Nearest-Neighbor Machine Translation (kNN-MT) is a non-parametric solution for domain adaptation . previous studies have shown that kNN retrieval is at the expense of high latency . |
| Approach: | They propose to use clustering to improve retrieval efficiency by combining a non-parametric MT with an in-domain feature-based retrieval module. |
| Outcome: | The proposed method reduces translation latency by 57% while maintaining the most useful information of the original datastore. |
Nearest Neighbor Machine Translation is Meta-Optimizer on Output Projection Layer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Nearest Neighbor Machine Translation (kNN-MT) is a powerful domain adaptation tool . the reasons for its success have not been thoroughly investigated . |
| Approach: | They propose to integrate pre-trained Neural Machine Translation models with token-level retrieval . they propose to implicitly execute gradient descent on the output projection layer of NMT . |
| Outcome: | The proposed approach outperforms model fine-tuning on in-domain tests while achieving better performance on out-of-domain sets. |
Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation techniques are not suitable for deep learning tasks due to memory constraints. |
| Approach: | They propose to combine knowledge from a large teacher network into a student network (S) they propose to use a combinatorial mechanism to inject layer-level supervision from T to S . |
| Outcome: | The proposed model outperforms existing models in PortugueseEnglish, TurkishEnglish and EnglishGerman directions and students trained using it have 50% fewer parameters and can deliver comparable results to 12-layer teachers. |
INK: Injecting kNN Knowledge in Nearest Neighbor Machine Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Neural machine translation models induce a non-smooth representation space, which harms its generalization results. |
| Approach: | They propose a framework to smooth the representation space by adjusting neighbor representations with a small number of new parameters. |
| Outcome: | The proposed framework outperforms the state-of-the-art kNN-MT system with average gains of 1.99 COMET and 1.0 BLEU on four benchmark datasets. |
Revisiting Source Context in Nearest Neighbor Machine Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing research does not explicitly consider the source context when retrieving similar examples . |
| Approach: | They propose a method to improve neural machine translation via source context enhancement by integrating a source-aware distance calibration module. |
| Outcome: | The proposed approach can be integrated with representative kNN-MT baselines and achieve significant performance improvements. |