GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator (2023.acl-long)
Copied to clipboard
Jian Yang, Shuming Ma, Li Dong, Shaohan Huang, Haoyang Huang, Yuwei Yin, Dongdong Zhang, Liqun Yang, Furu Wei, Zhoujun Li
| Challenge: | Existing pre-training methods underutilize the benefits of language understanding for generation. |
| Approach: | They propose a GAN-style model for encoder-decoder pre-training with an auxiliary discriminator. |
| Outcome: | The proposed model outperforms existing pre-trained models and achieves state-of-the-art performance. |
Similar Papers
PRINCE: Prefix-Masked Decoding for Knowledge Enhanced Sequence-to-Sequence Pre-Training (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on injecting noises into the input sequence, but feasibility of injecting them into the decoding sequence remains an open question. |
| Approach: | They propose a pre-training paradigm that integrates knowledge-enhanced decoding with noises in the prefix to strengthen the representation learning of entities that span over multiple input tokens. |
| Outcome: | The proposed model achieves state-of-the-art results on two knowledge-driven data-to-text generation tasks with up to 2% BLEU gains. |
Instance Regularization for Discriminative Language Model Pre-training (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have optimized independent strategies of ennoising or denosing . Existing methods treat training instances equally throughout the training process . |
| Approach: | They propose to use ennoising and denoising to train discriminative pre-trained language models . they propose to model the complexity of restoring the original sentences from corrupted ones . |
| Outcome: | Experimental results show that the proposed method improves pre-training efficiency, effectiveness, and robustness. |
Encoder and Decoder, Not One Less for Pre-trained Language Model Sponsored NMT (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for neural machine translation (NMT) use encoder-only enhancement or rely on specific multilingual PLMs. |
| Approach: | They propose a monolingual PLM-sponsored NMT model that lets both encoder and decoder enjoy PLM enhancement. |
| Outcome: | The proposed model improves on IWSLT14 En-De, De-En, WMT14 En -De and En-Fr tasks and achieves new state-of-the-art. |
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models for Natural Languages (NL) like BERT and GPT have been shown to transfer well to Programming Languages. |
| Approach: | They propose a unified pre-trained encoder-decoder Transformer model that leverages the code semantics conveyed from the developer-assigned identifiers. |
| Outcome: | The proposed model outperforms existing models on understanding and generation tasks and can capture semantic information from code. |
Counter-Contrastive Learning for Language GANs (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Generative Adversarial Networks (GANs) have proven to be difficult to generate natural language due to the uninformative learning signals passed from the discriminator. |
| Approach: | They propose to adopt the counter-contrastive learning method to support the generator’s training in language GANs by pulling the language representations of generated and real samples together and pushing apart representations. |
| Outcome: | The proposed method outperforms existing GANs on synthetic and real benchmarks and yields competitive performance compared to previous methods. |
Improving Neural Machine Translation with Conditional Sequence Generative Adversarial Nets (N18-1)
Copied to clipboard
| Challenge: | Experimental results show that the proposed model consistently outperforms the traditional RNNSearch and the newly emerged state-of-the-art Transformer on English-German and Chinese-English translation tasks. |
| Approach: | They propose an approach for applying GANs to NMT by building a conditional sequence generative adversarial net with two adversarials. |
| Outcome: | The proposed model outperforms the existing RNNSearch and Transformer on English-German and Chinese-English translation tasks. |
Making Use of Latent Space in Language GANs for Generating Diverse Text without Pre-training (2021.eacl-srw)
Copied to clipboard
| Challenge: | Existing models for generating diverse texts are not pre-trained . generative adversarial networks suffer from mode-collapsing if they are not trained . |
| Approach: | They propose a GAN model that produces diverse texts conditioned by latent code . they propose to use Gumbel-Softmax distribution for word sampling . |
| Outcome: | The proposed model is competitive with existing models, which requires pre-training. |
BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models of NLP are fading away, but new ones are needed to maintain their dominance. |
| Approach: | They propose a method to pretrain a CNN using Wikipedia data and integrate it with standard TLMs. |
| Outcome: | The proposed method outperforms the original ALBERT on GLUE tasks and achieves similar performance to SOTA on open-domain QA tasks. |
KILM: Knowledge Injection into Encoder-Decoder Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Large pre-trained language models retain implicit knowledge within their parameters, but are susceptible to memorizing the pretraining corpora rather than capturing the knowledge within them. |
| Approach: | They propose to inject entity-related knowledge into encoder-decoder PLMs via a generative knowledge infilling objective through continued pre-training. |
| Outcome: | The proposed approach outperforms state-of-the-art models on general NLU and NLG tasks while maintaining their original performance. |
CDLM: Cross-Document Language Modeling (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing language models (LMs) provide powerful representations for internal text structure, but there are important applications for multi-text tasks. |
| Approach: | They propose a pretraining approach that incorporates two key ideas into the masked language modeling objective. |
| Outcome: | The proposed model improves over existing models and sets of long-range transformers and can be easily applied to multiple multi-text tasks. |