Contrastive Token Learning with Similarity Decay for Repetition Suppression in Machine Translation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) is pivotal for crosslingual conversation and trade . traditional solutions that penalize text redundancy or token reoccurrence have shown limited efficacy . |
| Approach: | They propose an algorithm that modulates suppression of tokens dynamically, informed by attention weights and inter-token distances. |
| Outcome: | The proposed algorithm outperforms existing methods in precision and generalizability. |
Similar Papers
Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing language models generate repetitive texts with greedy decoding or beam search. |
| Approach: | They propose a self-contrastive training technique to penalize the output of a premature checkpoint of the same model when it incorrectly predicts repetition. |
| Outcome: | The proposed training mitigates repetition while maintaining fluency while minimizing the overestimation of token-level repetition probabilities. |
Contrastive Decoding Reduces Hallucinations in Large Multilingual Machine Translation Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Hallucinations occur when the target side sentence is detached from the source side sentence, or in other words, when there is a low contribution of the source sentence to the generation of the target sentence. |
| Approach: | They propose to use Contrastive Decoding to maximise the log-likelihood difference between a model and the same model with reduced contribution from the encoder outputs. |
| Outcome: | The proposed algorithm maximises the log-likelihood difference between a model and the same model with reduced contribution from the encoder outputs. |
Fidelity-Enriched Contrastive Search: Reconciling the Faithfulness-Diversity Trade-Off in Text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language models often generate fluent and convincing content but can lack consistency with the provided source, resulting in potential inaccuracies. |
| Approach: | They propose a new decoding method that augments the contrastive search framework with context-aware regularization terms to promote tokens that are semantically similar to the provided source while penalizing repetitiveness in the generated text. |
| Outcome: | The proposed method improves faithfulness across various language models while maintaining output diversity comparable to well-performing decoding algorithms. |
On Long-Tailed Phenomena in Neural Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art Neural Machine Translation models struggle with generating low-frequency tokens, tackling which remains a major challenge. |
| Approach: | They propose a loss function to better adapt model training to structural dependencies of conditional text generation by incorporating inductive biases of beam search into the training process. |
| Outcome: | The proposed method leads to significant gains over cross-entropy across different language pairs, especially on the generation of low-frequency words. |
Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Prior work treats all types of mismatches between source and target as noise . Consequently, it remains unclear how noisy parallel training samples impact NMT training. |
| Approach: | They propose a divergent-aware NMT framework that uses factors to help NMT recover from the degradation caused by naturally occurring divergences. |
| Outcome: | The proposed framework improves translation quality and model calibration on EN-FR tasks. |
Neural Machine Translation with Contrastive Translation Memories (2022.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that retrieval-augmented NMT model obtains substantial improvements over strong baselines in the benchmark dataset. |
| Approach: | They propose a retrieval-augmented NMT model that is holistically similar to the source sentence while individually contrastive to each other. |
| Outcome: | The proposed model improves on baselines in the translation task. |
Decoding Matters: Addressing Amplification Bias and Homogeneity Issue in Recommendations for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to adapt Large Language Models (LLMs) for recommendation encounter significant challenges such as amplification bias and homogeneity. |
| Approach: | They propose a new decoding approach called Debiasing-Diversifying Decoding (D3) that disables length normalization for ghost tokens to alleviate amplification bias and incorporates a text-free assistant model to encourage tokens less frequently generated by LLMs for counteracting recommendation homogeneity. |
| Outcome: | Extensive experiments on real-world datasets demonstrate the proposed approach’s effectiveness in enhancing accuracy and diversity. |
Reducing Word Omission Errors in Neural Machine Translation: A Contrastive Learning Approach (P19-1)
Copied to clipboard
| Challenge: | Existing methods for reducing word omission errors in neural machine translation are prone to omit essential words on the source side. |
| Approach: | They propose a contrastive learning approach to reduce word omission errors in NMT by omitting words. |
| Outcome: | The proposed approach achieves better translation performance than baseline methods on Chinese-to-English, German-to English, and Russian-toEnglish translation tasks. |
A Simple Recipe towards Reducing Hallucination in Neural Surface Realisation (P19-1)
Copied to clipboard
| Challenge: | Recent neural language generation systems often hallucinate contents when trained on loosely corresponding pairs of the input structure and text. |
| Approach: | They propose to integrate a language understanding module for data refinement with self-training iterations to induce strong equivalence between the input data and the paired text. |
| Outcome: | Experiments on the E2E challenge dataset show that the proposed framework reduces relative unaligned noise by 50% compared with the current state-of-the-art ensemble generator. |
Mitigating Tokenization-Induced Distance Distortion in Long-Context Multilingual Machine Translation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing positional encodings rely on fixed token indices and implicitly assume uniform semantic density, which breaks down for long-context inputs. |
| Approach: | They propose a tokenization-aware adaptive positional encoding that conditions relative positional bias on input-level sequence length and fragmentation statistics. |
| Outcome: | The proposed model improves long-context robustness and accuracy over baselines. |