Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to improve neural machine translation use token-level adaptive training . however, standard models make predictions on condition of previous contexts . |
| Approach: | They propose a target-context-aware metric which can be supplemented by statistical metrics . they propose an adaptive training approach based on token- and sentence-level CBMI . |
| Outcome: | The proposed model outperforms the Transformer baseline and other similar approaches on English-German and Chinese-English tasks. |
Similar Papers
Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to token-level adaptive training only use static word frequency information without considering the source language. |
| Approach: | They propose a bilingual mutual information based adaptive objective that assigns weights to target tokens with higher BMI . they propose to use this approach to improve token-level adaptive training . |
| Outcome: | The proposed method improves token-level adaptive training on two languages. |
Normalizing Mutual Information for Robust Adaptive Training for Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Neural machine translation models have been reported to generate hallucinations . despite the success of the models, there are still challenges to improve fluency . |
| Approach: | They propose a scoring metric for the importance of target sentences and tokens to encourage fluent translations. |
| Outcome: | The proposed metric improves translation fluency and source-faithfulness . the proposed nmi model is not properly normalized, the authors argue . |
Token-level Adaptive Training for Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing token imbalance phenomenon in natural language as different tokens appear with different frequencies, which leads to different learning difficulties for tokens in Neural machine translation (NMT). |
| Approach: | They propose to assign tokens with different frequencies to target tokens during training to encourage the model to pay more attention to low-frequency tokens. |
| Outcome: | The proposed model yields consistent improvements on ZH-EN, EN-RO, and EN-DE translation tasks, especially on sentences that contain more low-frequency tokens. |
Multi-Agent Mutual Learning at Sentence-Level and Token-Level for Neural Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) has achieved significant progress over recent years. |
| Approach: | They extend mutual learning to the machine translation task and operate at both the sentence-level and the token-level. |
| Outcome: | The proposed method improves on the IWSLT’14 German-English task and also on the WMT’14 English-German task. |
Adaptive Token-level Cross-lingual Feature Mixing for Multilingual Neural Machine Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual neural machine translation models can translate multiple language pairs in a single model but lacks ability to capture language-specific features. |
| Approach: | They propose a token-level feature mixing method that captures different features and dynamically determines feature sharing across languages. |
| Outcome: | The proposed method outperforms baselines and can be extended to zero-shot translation. |
Token-Level Self-Evolution Training for Sequence-to-Sequence Learning (2023.acl-short)
Copied to clipboard
| Challenge: | Adaptive training approaches do not consider the variation of learning difficulty in different training steps, making the learning deterministic and sub-optimal. |
| Approach: | They propose a dynamic token-level self-evolution training method that reweighs the training losses of different target tokens based on priors. |
| Outcome: | Empirically, the proposed method yields significant improvements on three translation tasks. |
Self-Vocabularizing Training for Neural Machine Translation (2025.naacl-srw)
Copied to clipboard
| Challenge: | Past vocabulary learning techniques identify relevant vocabulary before training, relying on corpus statistics or frequency counts without considering contextual information or the model's ability to represent it. |
| Approach: | They propose a method that self-vocabularizes a smaller, more optimal vocabulary by pairing source sentences with the model's predictions to define a new vocabulary. |
| Outcome: | The proposed method produces a 1.49 BLEU improvement in the simulated model and an increase in unique token usage and a 6–8% reduction in vocabulary size. |
Integrating Multi-scale Contextualized Information for Byte-based Neural Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Subword tokenization is a common method for vocabulary building in NMT systems . but, it has its disadvantages, such as a vocabulary cannot be modified once it is learned . |
| Approach: | They propose a method that learns contextualized information of varying scales . they propose byte-based tokenization to solve these problems with few embedding parameters . |
| Outcome: | Experiments show that the proposed method outperforms subword-based methods in multilingual and out-of-domain scenarios. |
The Effects of Language Token Prefixing for Multilingual Machine Translation (2022.aacl-short)
Copied to clipboard
| Challenge: | In recent years, the field has moved towards large neural models either translating from or into many languages. |
| Approach: | They propose to prefix language tokens onto a source or target sequence to improve translation performance. |
| Outcome: | The proposed methods improve translation performance and source side prefixes improve translation. |
Token Weighting for Long-Range Language Modeling (2025.findings-naacl)
Copied to clipboard
| Challenge: | Many applications of large language models (LLMs) require long-context understanding, but models still struggle with such tasks. |
| Approach: | They propose token-weighting schemes that assign different weights to each training token in the loss, generalizing existing works. |
| Outcome: | The proposed methods compare confidences of a long-context and short-concept model and show that non-uniform loss weights improve the long-constability of LLMs. |