Addressing the Length Bias Challenge in Document-Level Neural Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Document-level neural machine translation (DNMT) models incorporate context information through increased maximum lengths of source and target sentences. |
| Approach: | They propose a sliding decoding strategy that limits the length of target sentences . they propose 'length-normalized attention mechanism' to aid the model in focusing on target information . |
| Outcome: | The proposed method can achieve state-of-the-art results on open datasets. |
Similar Papers
Modeling Context With Linear Attention for Scalable Document-Level Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Document-level machine translation models lack quadratic complexity in the sequence length due to their attention layers. |
| Approach: | They evaluate a recent linear attention model with a sentential gate to promote a recency inductive bias and compare it to open-source document translation. |
| Outcome: | The proposed model significantly improves translation quality on IWSLT 2015 and OpenSubtitles 2018 with similar or better BLEU scores. |
A Simple and Effective Approach to Coverage-Aware Neural Machine Translation (P18-2)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) models are used to solve translation problems using long-term models. |
| Approach: | They propose a method to seek a better balance between model confidence and length preference for Neural Machine Translation. |
| Outcome: | The proposed model improves on Chinese-English and English-German translation tasks. |
Rethinking Document-level Neural Machine Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Neural machine translation models are weak enough for document-level translation . current models only translate sentences individually, resulting in poor document coherence . |
| Approach: | They propose to use the original Transformer model to test document-level neural machine translation . they find that the original transformer models can achieve strong results for document translation if trained properly . |
| Outcome: | The proposed model outperforms sentence-level models on nine datasets and two sentence- level datasets across six languages. |
G-Transformer for Document-Level Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work extends translation unit from single sentence to multiple sentences. |
| Approach: | They propose to introduce locality assumption as an inductive bias into Transformer and reduce the hypothesis space of attention from target to source. |
| Outcome: | The proposed model achieves state-of-the-art BLEU scores on three benchmark datasets. |
Long-Short Term Masking Transformer: A Simple but Effective Baseline for Document-level Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in deep learning have led to significant improvement of document-level neural machine translation (NMT). |
| Approach: | They propose a long-short term masking self-attention on top of the standard transformer to capture the long-range dependence and reduce the propagation of errors. |
| Outcome: | The proposed model captures the long-range dependence and reduces errors on two publicly available document-level datasets. |
Multi-Sentence Resampling: A Simple Approach to Alleviate Dataset Length Bias and Beam-Search Degradation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Neural Machine Translation suffers from a beam-search problem after a certain point, especially for long sentences. |
| Approach: | They propose a data augmentation technique that concatenates several sentences from the original dataset to make a long training example. |
| Outcome: | The proposed technique significantly reduces degradation with growing beam size and improves translation quality on the IWSTL15 En-Vi, IWStl17 En-Fr, and WMT14 En-De datasets. |
On Length Divergence Bias in Textual Matching Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing deep models have been successful in textual matching tasks, but it is unclear whether they understand language or measure semantic similarity of texts. |
| Approach: | They propose an adversarial evaluation scheme which invalidates the length divergence bias in TM datasets. |
| Outcome: | The proposed method improves the robustness and generalization ability of models at the same time. |
Scaling Law for Document Neural Machine Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) methods fail to capture discourse phenomena such as pronominal anaphora, lexical consistency, and document coherence as the input text exceeds a single sentence. |
| Approach: | They examine the effects of model scale, data scale, and sequence length on translation quality when model size is limited. |
| Outcome: | The proposed model scales and data scales are compared with the existing models and show that increasing sequence length improves translation quality when model size is limited. |
In-Context Learning (and Unlearning) of Length Biases (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing work has demonstrated the ability of large language models to learn lexical and label biases in-context negatively impacts performance and robustness of models. |
| Approach: | They investigate the impact of length biases on in-context learning by analyzing model length information in-constext. |
| Outcome: | The proposed model learns length biases in the context window without parameter updates. |
Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translation (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing document-level neural machine translation systems concatenate several consecutive sentences to form a pseudo-document, and then learn inter-sentential dependencies. |
| Approach: | They propose a document flattening technique that integrates Flat-Batch Attention (FBA) and Neural Context Gate (NCG) into Transformer model to utilize information beyond the pseudo-document boundaries. |
| Outcome: | The proposed method outperforms baselines on BLEU, COMET and accuracy on the contrastive test set. |