Papers by Mohammad Norouzi
Non-Autoregressive Machine Translation with Latent Alignments (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing non-autoregressive machine translation methods are lacking in the field of latent alignments. |
| Approach: | They propose two strong methods for non-autoregressive machine translation that model latent alignments with dynamic programming. |
| Outcome: | The proposed models achieve state-of-the-art on the WMT’14 EnDe task, compared with the autoregressive Transformer baseline. |
Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Empirical results on machine translation suggest that DPE is effective for segmenting output sentences. |
| Approach: | They propose a new algorithm for tokenizing sentences into subword units . they propose enabling exact log marginal likelihood estimation and exact MAP inference . |
| Outcome: | The proposed algorithm improves on machine translation datasets and on a large dataset. |
Meta-Learning Fast Weight Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Dynamic evaluation of language models (LMs) adapts model parameters at test time using gradient information from previous tokens. |
| Approach: | They propose a neural component that uses gradient updates as linear attention to improve model performance. |
| Outcome: | The proposed model can be applied at training time and learn to make good use of gradient updates. |
Character-Aware Models Improve Visual Text Rendering (2023.acl-long)
Copied to clipboard
Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, Rj Mical, Mohammad Norouzi, Noah Constant
| Challenge: | Current image generation models struggle to produce well-formed visual text due to lack of character-level input features. |
| Approach: | They conduct a series of experiments to compare character-aware vs. character-blind text encoders to determine their spelling ability. |
| Outcome: | The character-aware models outperform character-blind models on a range of novel text rendering tasks. |
Generate, Annotate, and Learn: NLP with Synthetic Text (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing methods to generate unlabeled text are difficult to find. |
| Approach: | They propose a general framework called "generate, annotate, and learn" to take advantage of synthetic text within knowledge distillation, self-training, and few-shot learning applications. |
| Outcome: | The proposed framework achieves state-of-the-art knowledge distillation results for 6-layer transformers on the GLUE leaderboard. |
The Importance of Generation Order in Language Modeling (D18-1)
Copied to clipboard
| Challenge: | Neural language models are universally autoregressive, generating sentences one token at a time from left to right. |
| Approach: | They propose a two-pass language model that generates partially-filled sentences and fills in missing tokens. |
| Outcome: | The proposed model produces partially-filled sentences and fills in missing tokens. |