| Challenge: | Pre-trained language model representations have been successful in a wide range of language understanding tasks. |
| Approach: | They propose to use pre-trained language model representations to integrate them into sequence to sequence models and apply it to machine translation and abstractive summarization. |
| Outcome: | The proposed model is able to perform 5.3 BLEU in machine translation and 5.3 on the full text version of CNN/DailyMail. |
Similar Papers
Improving Language Model Integration for Neural Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to integrate external language models into machine translation systems have been based on the assumption that the external model learns an implicit target-side language model at decoding time. |
| Approach: | They transfer this concept to the task of machine translation and compare it with the most prominent way of including additional monolingual data - namely back-translation. |
| Outcome: | The proposed approach outperforms the most prominent way of including additional monolingual data, namely back-translation. |
Advances in Pre-Training Distributed Word Representations (L18-1)
Copied to clipboard
| Challenge: | Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications. |
| Approach: | They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations. |
| Outcome: | The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data. |
Graph Pre-training for AMR Parsing and Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Abstract meaning representation (AMR) highlights the core semantic information of text in a graph structure. |
| Approach: | They propose two graph auto-encoding strategies for graph-to-graph pre-training and four tasks to integrate text and graph information during pre-tuning to improve structure awareness. |
| Outcome: | The proposed model is superior to pre-trained language models on AMR parsing and AMR-to-text generation tasks. |
Multilingual Denoising Pre-training for Neural Machine Translation (2020.tacl-1)
Copied to clipboard
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer
| Challenge: | Existing approaches to pre-train models focus on only English corpora, but this is not common in machine translation. |
| Approach: | They propose a sequence-to-sequence denoising auto-encoder pre-trained on monolingual corpora . they show that it produces significant performance gains across MT tasks . |
| Outcome: | The proposed model can achieve significant performance gains across a wide variety of MT tasks. |
Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | Pre-trained language models are language models that are pre-taught on large-scaled corpora in a self-supervised fashion. |
| Approach: | This tutorial provides a broad and comprehensive introduction to pre-trained language models . it focuses on emerging methods that enable PLMs to perform diverse downstream tasks . |
| Outcome: | This tutorial focuses on the benefits of pre-trained language models and how to use them in NLP tasks. |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
Synthetic Pre-Training Tasks for Neural Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | toxicity and bias can be addressed by pre-training with synthetic resources . BLEU scores are used to compare methods with real-world data . |
| Approach: | They propose several ways to generate obfuscated data from large parallel corpus and concatenating phrase pairs from small word-aligned corpus with synthetic parallel data without real human language corpora. |
| Outcome: | The proposed methods can be used to generate obfuscated data or synthetic parallel data without real human language corpora even with high levels of oblication. |
Second Language Acquisition of Neural Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | a recent study examined the cross-lingual transferability of neural language models . previous studies focused on their first language acquisition . |
| Approach: | They propose to pretrain bilingual LMs with a scenario similar to human L2 acquisition . they find that pretraining accelerated their linguistic generalization in L2 . |
| Outcome: | The results show that pretraining bilingual LMs accelerates their linguistic generalizations . the results clarify their (non-)human-like L2 acquisition in particular aspects . |
Simple and Effective Noisy Channel Modeling for Neural Machine Translation (D19-1)
Copied to clipboard
| Challenge: | Previous work on noisy channel modeling relied on latent variable models that incrementally process the source and target sentence. |
| Approach: | They propose to use a standard sequence to sequence model which utilizes the entire source and target sentences to estimate posterior probability of a target sequence y given a source sequence x. |
| Outcome: | The proposed model outperforms direct models on German-English translations by up to 3.2 BLEU on four language pairs. |
Pre-training Universal Language Representation (2021.acl-long)
Copied to clipboard
| Challenge: | Despite the cutting-edge representation learning, most language models focus on specific levels of linguistic units. |
| Approach: | They propose a training objective MiSAD that utilizes meaningful n-grams extracted from large unlabeled corpus by an algorithm for pre-trained language models. |
| Outcome: | The proposed model achieves highest accuracy on analogy tasks in different language levels and significantly improves performance on downstream tasks. |