Papers by Sergey Edunov
Understanding Back-Translation at Scale (D18-1)
Copied to clipboard
| Challenge: | An effective method to improve neural machine translation with monolingual data is to augment the parallel training corpus with back-translations of target language sentences. |
| Approach: | They propose to augment parallel training corpus with back-translations of target language sentences to improve neural machine translation with monolingual data. |
| Outcome: | The proposed method achieves a state-of-the-art of 35 BLEU on the WMT’14 English-German test set. |
Multilingual Denoising Pre-training for Neural Machine Translation (2020.tacl-1)
Copied to clipboard
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer
| Challenge: | Existing approaches to pre-train models focus on only English corpora, but this is not common in machine translation. |
| Approach: | They propose a sequence-to-sequence denoising auto-encoder pre-trained on monolingual corpora . they show that it produces significant performance gains across MT tasks . |
| Outcome: | The proposed model can achieve significant performance gains across a wide variety of MT tasks. |
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the Web (2021.acl-long)
Copied to clipboard
| Challenge: | Using a curated common crawl corpus, we were able to mine 10.8 billion parallel sentences out of which only 2.9 billions are aligned with English. |
| Approach: | They use 32 snapshots of a curated common crawl corpus totaling 71 billion unique sentences to mine 10.8 billion parallel sentences out of which only 2.9 billions are aligned with English. |
| Outcome: | The proposed system outperforms the best single systems on the WMT’19 test set for English-German/Russian/Chinese and outperformed the best submission at the 2020 WAT workshop. |
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (N19-4)
Copied to clipboard
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli
| Challenge: | OpenNMT is a community-built toolkit written in multiple languages with an emphasis on extensibility. |
| Approach: | They propose to use PyTorch to train custom sequence models for translation, summarization, language modeling, and other tasks. |
| Outcome: | The proposed toolkit is fast, extensible, and useful for both research and production. |
Pre-trained language model representations for language generation (N19-1)
Copied to clipboard
| Challenge: | Pre-trained language model representations have been successful in a wide range of language understanding tasks. |
| Approach: | They propose to use pre-trained language model representations to integrate them into sequence to sequence models and apply it to machine translation and abstractive summarization. |
| Outcome: | The proposed model is able to perform 5.3 BLEU in machine translation and 5.3 on the full text version of CNN/DailyMail. |
Dense Passage Retrieval for Open-Domain Question Answering (2020.emnlp-main)
Copied to clipboard
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih
| Challenge: | Open-domain question answering relies on efficient passage retrieval to select candidate contexts. |
| Approach: | They propose a dual-encoder framework that can be implemented to retrieve passages from a small number of questions and passages. |
| Outcome: | The proposed system outperforms a strong Lucene-BM25 system in top-20 passage retrieval accuracy on multiple open-domain QA benchmarks. |
On The Evaluation of Machine Translation Systems Trained With Back-Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Back-translation is a data augmentation technique that can be used to improve neural machine translation systems. |
| Approach: | They propose to combine back-translation with a language model score to measure fluency. |
| Outcome: | The proposed method improves translation quality of natural text and translationese according to professional translators. |
Classical Structured Prediction Losses for Sequence to Sequence Learning (N18-1)
Copied to clipboard
| Challenge: | Recent work on training neural attention models at the sequence level has focused on a series of objective functions commonly used for structured prediction. |
| Approach: | They propose to use objective functions commonly used to train linear models for structured prediction to train neural attention models at the sequence-level using either reinforcement learning-style methods or beam search optimization. |
| Outcome: | The proposed model outperforms beam search optimization on German-English translation and abstractive summarization tasks. |
Effective Long-Context Scaling of Foundation Models (2024.naacl-long)
Copied to clipboard
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, Hao Ma
| Challenge: | Large language models (LLMs) are rapidly deployed and continue to evolve through scaling. |
| Approach: | They propose a method to train strong long-context LLMs that are capable of utilizing massive context windows of up to 32,000 tokens. |
| Outcome: | The proposed model can surpass gpt-3.5-turbo-16k's overall performance on long-context benchmarks with a cost-effective instruction tuning procedure that is free of expensive annotations. |
Cloze-driven Pretraining of Self-attention Networks (D19-1)
Copied to clipboard
| Challenge: | Existing work on pretraining language models has used unidirectional (left-to-right) or bi-directional (both left-to right and right-to left) LMs with loss function. |
| Approach: | They propose a bi-directional transformer model that pretrains both directions of a large language-model-inspired self-attention cloze model and propose clozing to predict each word in the training data. |
| Outcome: | The proposed model performs well on GLUE and state of the art benchmarks consistent with BERT. |