Multi-Path Transformer is Better: A Case Study on Neural Machine Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Extensive experiments on 12 WMT tasks show that shallower multi-path models can achieve similar or even better performance than the deeper model. |
| Approach: | They propose to use a parameter-efficient multi-path structure to fuse features extracted from different paths to achieve better performance. |
| Outcome: | The proposed model can achieve better performance with the same number of parameters than the deeper model. |
Similar Papers
Learning Deep Transformer Models for Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Neural machine translation models have advanced the previous state-of-the-art by learning mappings between sequences via neural networks and attention mechanisms. |
| Approach: | They propose to use layer normalization to pass the combination of previous layers to the next layer to improve the model. |
| Outcome: | The proposed model outperforms the shallow Transformer-Big/Base baseline model on English-German and Chinese-English tasks by 0.4-2.4 BLEU points. |
The Impact of Depth on Compositional Generalization in Transformer Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | In this paper, we test the hypothesis that deeper transformers generalize more compositionally. |
| Approach: | They propose to add layers to transformers to generalize more compositionally . they propose to fine-tune the models so that the total number of parameters is constant . |
| Outcome: | The proposed model generalizes more compositionally than shallower models, but returns diminish . the proposed model can be made shallower without sacrificing performance . |
Rethinking Document-level Neural Machine Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Neural machine translation models are weak enough for document-level translation . current models only translate sentences individually, resulting in poor document coherence . |
| Approach: | They propose to use the original Transformer model to test document-level neural machine translation . they find that the original transformer models can achieve strong results for document translation if trained properly . |
| Outcome: | The proposed model outperforms sentence-level models on nine datasets and two sentence- level datasets across six languages. |
Auto-Sizing the Transformer Network: Improving Speed, Efficiency, and Performance for Low-Resource Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence models are sensitive to architecture and hyperparameter settings. |
| Approach: | They incorporate architecture search into a single training run through auto-sizing . they show that auto-size can improve BLEU scores by up to 3.9 points . |
| Outcome: | The proposed algorithm improves BLEU scores on low-resource language pairs while removing one-third of the parameters from the model. |
Rethinking the Value of Transformer Components (2020.coling-main)
Copied to clipboard
| Challenge: | Empirical results show that certain components are more important than others . we propose a new training strategy that can improve Transformer models by distinguishing unimportant components . |
| Approach: | They propose a training strategy that distinguishes the unimportant components in training . they compare the impact of individual component (sub-layer) on model performance . |
| Outcome: | The proposed training strategy can improve translation performance by distinguishing unimportant components in training. |
TranSFormer: Slow-Fast Transformer for Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Prior work has focused on treating subwords as basic units in developing such systems. |
| Approach: | They propose a slow-fast two-stream learning model that uses a “slow” branch to deal with subword sequences and a "fast" branch to cope with longer character sequences. |
| Outcome: | The proposed model shows consistent BLEU improvements (larger than 1 BLUE point) on several machine translation benchmarks. |
Improving the Transformer Translation Model with Document-Level Context (D18-1)
Copied to clipboard
| Challenge: | Existing models for document-level context translation ignore documentlevel context. |
| Approach: | They propose a document-level context encoder to represent document- level context and integrate it into the Transformer model. |
| Outcome: | Experiments on NIST Chinese-English and IWSLT French-English datasets show that the proposed translation model outperforms the Transformer model significantly. |
G-Transformer for Document-Level Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work extends translation unit from single sentence to multiple sentences. |
| Approach: | They propose to introduce locality assumption as an inductive bias into Transformer and reduce the hypothesis space of attention from target to source. |
| Outcome: | The proposed model achieves state-of-the-art BLEU scores on three benchmark datasets. |
Revisiting the Markov Property for Machine Translation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Statistical machine translation (SMT) has employed Markov models, but autoregressive models are less effective. |
| Approach: | They propose to use a Markov Autoregressive Transformer to model neural machine translation using four WMT benchmarks. |
| Outcome: | The proposed model performs better than autoregressive models on four WMT benchmarks. |
Optimizing Transformer for Low-Resource Neural Machine Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Language pairs with limited amounts of parallel data remain a challenge for neural machine translation. |
| Approach: | They propose to optimize a Transformer model for low-resource conditions to improve translation quality by 7.3 BLEU points compared to the default settings. |
| Outcome: | The proposed model improves translation quality up to 7.3 BLEU points compared to the default settings on the IWSLT14 training data compared with the Transformer model. |