| Challenge: | Attention-based sequence-to-sequence neural networks learn to jointly align and translate. |
| Approach: | They propose to use a reordering RNN that shares the input encoder with the decoder to decouple re-ordering from translation. |
| Outcome: | The proposed model can achieve superior reordering accuracy without feature engineering and is 2.5x faster in decoding. |
Similar Papers
Neural Machine Translation with Reordering Embeddings (P19-1)
Copied to clipboard
| Challenge: | Existing work exploits the reordering information in neural machine translation . experimental results show that the proposed methods can significantly improve the performance of the transformer translation system. |
| Approach: | They propose a reordering mechanism to learn the re ordering embedding of a word based on contextual information and stack them together with self-attention networks to learn sentence representation for machine translation. |
| Outcome: | The proposed method improves translation performance on English-to-German, NIST Chinese-to English, and WAT Japanese-toEnglish translation tasks. |
Assessing the Ability of Self-Attention Networks to Learn Word Order (P19-1)
Copied to clipboard
| Challenge: | Existing studies have attributed SAN to being weak at learning positional information for sequence modeling due to lack of recurrence structure. |
| Approach: | They propose a word reordering detection task to quantify how well word order information is learned by SAN and RNN. |
| Outcome: | The proposed task quantifies how well word order information learned by SAN and RNN is learned. |
Assessing Non-autoregressive Alignment in Neural Machine Translation via Word Reordering (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing non-autoregressive neural machine translation models that implicitly model dependencies are sub-optimal in handling word order errors. |
| Approach: | They propose to learn a non-autoregressive language model that can be combined with Viterbi decoding to achieve better reordering performance. |
| Outcome: | The proposed model outperforms state-of-the-art reordering mechanisms under different word permutation settings with a 2-27 BLEU improvement, suggesting high potential for word alignment in NAT. |
Neural Syntactic Preordering for Controlled Paraphrase Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to paraphrasing natural language sentences are limited by the complexity of the task. |
| Approach: | They propose a framework for paraphrasing natural language sentences that uses syntactic transformations to softly "reorder" the source sentence and their proposed system is evaluated automatically and by humans . |
| Outcome: | The proposed model retains the quality of the baseline approaches while giving a substantial increase in the diversity of the generated paraphrases. |
RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition (P18-4)
Copied to clipboard
| Challenge: | Using RETURNN, we train and decode attention models for translation and speech recognition. |
| Approach: | They propose a layer-wise pretraining scheme for recurrent attention models and show its significant effect on deep recurrence encoder networks. |
| Outcome: | The proposed training and decoding scheme improves 1% on expected training and improves on WMT 2017 and Switchboard. |
Recursive Neural Network Based Preordering for English-to-Japanese Machine Translation (P18-3)
Copied to clipboard
| Challenge: | Existing methods for preordering require a manual feature design, making language dependent design difficult. |
| Approach: | They propose a preordering method with recursive neural networks that learn features from raw inputs. |
| Outcome: | The proposed method is comparable to the state-of-the-art method but without a manual feature design. |
Exploiting Pre-Ordering for Neural Machine Translation (L18-1)
Copied to clipboard
| Challenge: | Existing studies have shown that Neural Machine Translation suffers from the problems that some source words are mistakenly translated for multiple times . |
| Approach: | They propose a pre-ordering approach to solve the under-translation problem by pre-ordnanced source sentences and position embedding to enhance monotone translation. |
| Outcome: | The proposed method significantly improves translation quality by 2.43 BLEU points on Chinese-to-English translation. |
Accurate Word Alignment Induction from Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Prior work suggests that Transformer captures poor word alignments through its attention mechanism. |
| Approach: | They propose two new word alignment induction methods that use attention weights to capture accurate word alignments. |
| Outcome: | The proposed methods outperform baselines on three publicly available datasets and are significantly better than GIZA++. |
Speeding up Transformer Decoding via an Attention Refinement Network (2022.coling-1)
Copied to clipboard
| Challenge: | Extensive experiments on ten WMT machine translation tasks show that the proposed model yields an average of 1.35x faster (with almost no decrease in BLEU) |
| Approach: | They propose a weighted residual network which reconstructs attention by reusing the features across layers. |
| Outcome: | The proposed model is 1.35x faster than the state-of-the-art inference model on translation tasks compared to AAN and SAN models with fewer parameter numbers . |
Attention Fusion: a light yet efficient late fusion mechanism for task adaptation in NLU (2022.findings-naacl)
Copied to clipboard
| Challenge: | a recent study has shown that fine-tuning pre-trained models is parameter-inefficient and expensive. |
| Approach: | They propose a task-attuned token module which integrates pre-trained network representations into a pre-trainer. |
| Outcome: | The proposed model trains only 0.0009% of the parameters and is efficient during computation and scalable during deployment. |