Papers with PrLMs
TSGP: Two-Stage Generative Prompting for Unsupervised Commonsense Question Answering (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies focus on acquiring relevant knowledge by retrieving external knowledge bases and fine-tuning pre-trained models. |
| Approach: | They propose a two-stage prompt-based unsupervised commonsense question answering framework that leverages implicit knowledge stored in PrLMs to generate knowledge for questions with unlimited types and possible candidate answers independent of specified choices. |
| Outcome: | The proposed framework significantly improves the reasoning ability of language models in unsupervised settings. |
Distinguishing Non-natural from Natural Adversarial Samples for More Robust Pre-trained Language Model (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent studies on adversarial attacks achieve high success rates against PrLMs, claiming that they are not robust. |
| Approach: | They propose to use anomaly detector to evaluate PrLMs with more natural adversarial samples to evaluate their robustness. |
| Outcome: | The proposed method can be used to defend all types of attacks and achieve higher accuracy on adversarial samples and compliant samples than other defense frameworks. |
Tracing Origins: Coreference-aware Machine Reading Comprehension (2022.acl-long)
Copied to clipboard
| Challenge: | a recent study has enriched pre-trained language models with syntactic, semantic and other linguistic information to improve their performance. |
| Approach: | They use a pre-trained language model to leverage coreference information to enhance word embeddings . they use additional encoder layers to focus on coreference mentions or a relational graph convolutional network to model the coreference relations. |
| Outcome: | The proposed model imitates the human reading process and leverages coreference information to enhance word embeddings. |
Span Fine-tuning for Pre-trained Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to fine-tune pre-trained language models are time-consuming and lack flexibility. |
| Approach: | They propose a span fine-tuning method which allows for a more efficient and efficient way of incorporating span-level information into pre-training. |
| Outcome: | Experiments on GLUE benchmark show that the proposed method significantly enhances the PrLM and offers more flexibility in an efficient way. |
Back to the Future: Bidirectional Information Decoupling Network for Multi-turn Dialogue Modeling (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on dialogue modeling use pre-trained language models to encode dialogue history as successive tokens, which is insufficient in capturing the temporal characteristics of dialogues. |
| Approach: | They propose a bidirectional information decoupling network as a universal dialogue encoder which explicitly incorporates both the past and future contexts. |
| Outcome: | The proposed model incorporates past and future contexts and can be generalized to a wide range of dialogue-related tasks. |
Dialogue-oriented Pre-training (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models (PrLMs) have shown impressive improvements for various downstream tasks including various dialogue related ones. |
| Approach: | They propose to use pre-trained language models to simulate dialogue features on general plain text with common language model training objectives to improve performance. |
| Outcome: | The proposed method is fine-tuned on three public multi-turn dialogue datasets and achieves significant and consistent improvement over the plain PrLMs. |
Enhancing Language Generation with Effective Checkpoints of Pre-trained Language Model (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to exploit PrLMs for NLG tasks do not get as much performance gain as in the NLU task. |
| Approach: | They propose a method to integrate public checkpoints of PrLMs for the most convenience. |
| Outcome: | The proposed method significantly improves the quality of the language generation tasks on 6 different kinds of PrLMs. |
Semantic-Preserving Adversarial Code Comprehension (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies on improving PrLMs for source code comprehension have not found a way to improve both sides of the trade-off between the two aspects. |
| Approach: | They propose to use semantic-preserving code embeddings to find worst-case attacks while forcing the model to predict the correct labels under these worst cases. |
| Outcome: | The proposed model can stay robust against state-of-the-art attacks while boosting the performance of PrLMs for code. |
Multi2WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialog (2022.naacl-main)
Copied to clipboard
| Challenge: | Task-oriented dialog (TOD) is arguably one of the most popular natural language processing (NLP) application areas. |
| Approach: | They propose a multilingual multi-domain TOD dataset that spans four languages . they use a framework for multilingual conversational specialization of pretrained language models . |
| Outcome: | The proposed datasets show that they perform better than existing datasets in English . the proposed framework allows for sample-efficient few-shot transfer for TOD tasks . |
Defending Pre-trained Language Models from Adversarial Word Substitution Without Performance Sacrifice (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing defense approaches suffer from notable performance loss and complexities. |
| Approach: | They propose a framework for detecting anomalies with frequency-aware randomization to defend adversarial word substitution. |
| Outcome: | The proposed framework outperforms existing defense methods over various tasks with much higher inference speed. |
Structural Pre-training for Dialogue Comprehension (2021.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large-scale pre-training language models (PrLMs) have achieved remarkable successes in a variety of natural language processing tasks. |
| Approach: | They propose to use SPIDER to capture dialogue exclusive features from dialogue texts. |
| Outcome: | The proposed model performs well on widely used dialogue benchmarks. |
Learning Better Masking for Better Language Model Pre-training (2023.acl-long)
Copied to clipboard
| Challenge: | Existing PrLMs adopt a Random-Token Masking strategy with a fixed masking ratio and different contents are masked by an equal probability throughout the training. |
| Approach: | They propose two scheduled masking approaches that adaptively tune masking ratio and masked content in different training stages, which improves pre-training efficiency and effectiveness. |
| Outcome: | The proposed methods improve the pre-training efficiency and effectiveness on the downstream tasks. |
Forging Multiple Training Objectives for Pre-trained Language Models via Meta-Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Empirical studies show that learning multiple training objectives in a single model makes the learned language representation barely converge to the desired optimum. |
| Approach: | They propose a meta-learning-based adaptive sampler which learns latent sampling pattern on arbitrary pre-training objectives. |
| Outcome: | Empirical studies show that learning multiple objectives in a single model makes it difficult to achieve the desired optimum. |
Feature Adaptation of Pre-Trained Language Models across Languages and Domains with Robust Self-Training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Adapting pre-trained language models (PrLMs) to new domains has gained much attention . Adaptation of PrLMs to newdomains is important, but requires fine-tuning . |
| Approach: | They propose to use PrLMs to adapt to new domains without fine-tuning . they use class-aware feature self-distillation to learn discriminative features . |
| Outcome: | The proposed model can learn discriminative features from pre-trained language models without fine-tuning. |
Instance Regularization for Discriminative Language Model Pre-training (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have optimized independent strategies of ennoising or denosing . Existing methods treat training instances equally throughout the training process . |
| Approach: | They propose to use ennoising and denoising to train discriminative pre-trained language models . they propose to model the complexity of restoring the original sentences from corrupted ones . |
| Outcome: | Experimental results show that the proposed method improves pre-training efficiency, effectiveness, and robustness. |