| Challenge: | Neural machine translation models typically implement encoder and decoder as multiple layers, but only the top layers are leveraged in the subsequent process, which misses the opportunity to exploit useful information embedded in other layers. |
| Approach: | They propose to expose all of these signals with layer aggregation and multi-layer attention mechanisms and introduce an auxiliary regularization term to encourage different layers to capture diverse information. |
| Outcome: | The proposed approach exposes all of these signals with layer aggregation and multi-layer attention mechanisms on widely-used translation datasets. |
Similar Papers
Exploiting Sentential Context for Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to exploit sentential context for machine translation are not well studied. |
| Approach: | They propose a shallow sentential context that exploits top encoder layer, and a deep sentential one that aggregates sentential representations from all internal layers. |
| Outcome: | The proposed model outperforms the strong Transformer model on the English-German and English-French benchmarks. |
Depth Growing for Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Neural machine translation models with tens and even more than a hundred blocks have shown effectiveness in image recognition. |
| Approach: | They propose a two-stage approach with three specially designed components to construct deeper NMT models. |
| Outcome: | The proposed approach improves on WMT14 EnglishGerman and EnglishFrench translation tasks. |
A Simple and Effective Approach to Coverage-Aware Neural Machine Translation (P18-2)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) models are used to solve translation problems using long-term models. |
| Approach: | They propose a method to seek a better balance between model confidence and length preference for Neural Machine Translation. |
| Outcome: | The proposed model improves on Chinese-English and English-German translation tasks. |
Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation (2025.findings-acl)
Copied to clipboard
Yingfeng Luo, Tong Zheng, Yongyu Mu, Bei Li, Qinghong Zhang, Yongqi Gao, Ziqiang Xu, Peinan Feng, Xiaoqian Liu, Tong Xiao, JingBo Zhu
| Challenge: | Recent advances in machine translation have focused on a single pre-trained decoder . encoder-decoder architectures have received relatively little attention in NMT . |
| Approach: | They propose a method that leverages LLMs as MT encoders and pairs them with lightweight decoders to develop universal translation models. |
| Outcome: | The proposed method matches or surpasses baselines in terms of translation quality but achieves 75% reduction in memory footprint of the KV cache. |
Training Deeper Neural Machine Translation Models with Transparent Attention (D18-1)
Copied to clipboard
| Challenge: | Existing NMT models are shallow in comparison to convolutional models used for both text and vision tasks. |
| Approach: | They propose to modify the attention mechanism to ease the optimization of deeper models by a simple modification to the seq2seq with attention paradigm. |
| Outcome: | The proposed model achieves consistent gains of 0.7-1.1 BLEU on the benchmark WMT’14 English-German and WMT'15 Czech-English tasks. |
On the Sub-layer Functionalities of Transformer Decoder (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing efforts to interpret the encoder of Transformer-based encoder-decoder architectures for neural machine translation have focused on assessing the encoded representations or interpreting the multi-head self-attentions. |
| Approach: | They propose to use Transformer-based encoder-decoder architectures to analyze how information is propagated through each module of each decoder layer. |
| Outcome: | The proposed model can be dropped with minimal loss of performance on three translation datasets and can be used to train and inference faster. |
Do Multilingual Neural Machine Translation Models Contain Language Pair Specific Attention Heads? (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent studies on multilingual representations focus on whether there is an emergence of language-independent representations or whether multilingual models partition their weights among different languages. |
| Approach: | They analyze encoder self-attention and encoder-decoder attention heads in a multilingual neural translation model. |
| Outcome: | The proposed model is based on a multilingual neural translation model with a language-independent representation. |
Top-down Tree Structured Decoding with Syntactic Connections for Neural Machine Translation and Parsing (D18-1)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) models are based on sequential decoding or serialisation of structured data into sequence. |
| Approach: | They propose a model that combines sequential encoder with tree-structured decoding augmented with a syntax-aware attention model. |
| Outcome: | The proposed model produces fluent translations with better reordering than previous models. |
Language-aware Interlingua for Multilingual Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing multilingual neural machine translation models fail to capture diversity and specificity of different languages, resulting in inferior performance against individual models that are sufficiently trained. |
| Approach: | They propose to integrate a language-aware interlingua into an Encoder-Decoder architecture to learn a semantic representation from the semantic spaces of different languages while allowing for language-specific specialization of a particular language pair. |
| Outcome: | The proposed model achieves remarkable improvements over state-of-the-art multilingual NMT models and produces comparable performance with strong individual models. |
Neural Machine Translation Incorporating Named Entity (C18-1)
Copied to clipboard
| Challenge: | Conventional NMT models have difficulty translating words with multiple meanings because of the high ambiguity. |
| Approach: | They propose a neural machine translation model that incorporates named entity (NE) tags of source-language sentences to reduce the difficulty in translating multiple meanings. |
| Outcome: | The proposed model achieves 3.11 point improvement in bilingual evaluation understudy (BLEU) on English-to-Japanese translation task with the ASPEC, and English- to-Bulgarian and English to-Romanian translation tasks with the Europarl corpus. |