Papers by Jakob Uszkoreit
Self-Attention with Relative Position Representations (N18-2)
Copied to clipboard
| Challenge: | Recent approaches to sequence to sequence learning leverage recurrence, convolution, attention or combination of recurrent and convolutional neural networks. |
| Approach: | They propose an approach that extends the self-attention mechanism to consider representations of relative positions, or distances between sequence elements. |
| Outcome: | The proposed approach yields 1.3 BLEU and 0.3 BLUE on translation tasks . it is based on a relation-aware self-attention mechanism that can generalize to arbitrary graph-labeled inputs. |
An Empirical Study of Generation Order for Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | a recent study of generation order for machine translation shows it does not affect output quality . Neural sequence models have been successfully applied to a broad range of tasks in recent years . |
| Approach: | They propose a soft order-reward framework that enables models to follow arbitrary oracle generation policies. |
| Outcome: | The proposed framework explores a wide variety of generation orders including uninformed orders, location-based orders, frequency-based or model-based orderings, and model-driven orders. |
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation (P18-1)
Copied to clipboard
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, Macduff Hughes
| Challenge: | In recent years, the emergence of seq2seq models has revolutionized the field of machine translation by replacing traditional phrase-based approaches with neural machine translation (NMT) systems based on the encoder-decoder paradigm. |
| Approach: | They propose to use a convolutional seq2seq model to combine the strengths of the two approaches. |
| Outcome: | The proposed architectures outperform the existing models on the WMT’14 benchmark dataset. |