Domain Adversarial Fine-Tuning as an Effective Regularizer (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing fine-tuning techniques can degrade general-domain representations . however, fine-timing can lead to catastrophic forgetting of knowledge . |
| Approach: | They propose a new regularization technique that complements the task-specific loss used during fine-tuning with an adversarial objective. |
| Outcome: | Empirical results show that AFTER improves performance on various natural language understanding tasks compared to standard fine-tuning. |
Similar Papers
G-Tuning: Improving Generalization of Pre-trained Language Models with Generative Adversarial Network (2023.findings-acl)
Copied to clipboard
| Challenge: | Empirical evaluations on the GLUE benchmark demonstrate that fine-tuning can enhance the generalization performance of pre-trained language models (PLMs) in downstream tasks. |
| Approach: | They propose a fine-tuning framework that transforms the latent representation of pre-trained language models from a universal space to a target space and integrates a generative adversarial network into the fine-untun process. |
| Outcome: | Empirical evaluations on the GLUE benchmark and two additional demanding scenarios show that the proposed framework can improve the generalization performance of pre-trained language models (PLMs) in downstream tasks. |
SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models fail to generalize to unseen data. |
| Approach: | They propose a framework for robust and efficient fine-tuning for pre-trained models . proposed framework achieves new state-of-the-art performance on a number of NLP tasks . |
| Outcome: | The proposed framework outperforms the state-of-the-art T5 model on GLUE, SNLI, SciTail and ANLI. |
Improving Pre-trained Language Model Sensitivity via Mask Specific losses: A case study on Biomedical NER (2024.naacl-long)
Copied to clipboard
| Challenge: | Fine-tuning is the prevailing practice for adapting language models (LMs) to new domains. |
| Approach: | They propose a mask specific language model that weights the importance of domain-specific terms during fine-tuning to avoid insensitivity. |
| Outcome: | The proposed approach outperforms advanced masking strategies such as span- and PMI-based masking. |
Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach (2021.naacl-main)
Copied to clipboard
| Challenge: | Fine-tuned pre-trained language models (LMs) have enormous success in many natural language processing tasks, but they still require excessive labeled data in the fine-tuning stage. |
| Approach: | They propose a framework to enable fine-tuning pre-trained language models with weak supervision without any labeled data. |
| Outcome: | The proposed framework outperforms the strongest baseline and achieves competitive performance with fully-supervised fine-tuning methods. |
What’s in a Domain? Learning Domain-Robust Text Representations using Adversarial Training (N18-2)
Copied to clipboard
| Challenge: | a key roadblock is application to new domains, unseen in training. |
| Approach: | They propose a method to optimise in- and out-of-domain accuracy by combing domain-specific and domain-general components with adversarial training for domain. |
| Outcome: | The proposed method improves on domain adaptation and domain-adversarial training. |
In and Out-of-Domain Text Adversarial Robustness via Label Smoothing (2023.acl-short)
Copied to clipboard
| Challenge: | Existing studies show that state-of-the-art NLP models are vulnerable to adversarial attacks . label smoothing has been proven effective in a variety of applications and modalities . |
| Approach: | They propose to use label smoothing to improve adversarial robustness in pre-trained models against various popular attacks. |
| Outcome: | The proposed method significantly improves adversarial robustness in pre-trained models against various popular attacks. |
Robust Semantic Parsing with Adversarial Learning for Domain Generalization (N19-2)
Copied to clipboard
| Challenge: | Using adversarial learning to train models on a higher level of abstraction to increase their robustness to lexical and stylistic variations is crucial for the integration of Semantic Parsing technologies in real applications. |
| Approach: | They propose to perform Semantic Parsing with a domain classification adversarial task and an unsupervised domain discovery approach that yields equivalent improvements. |
| Outcome: | The proposed approach improves on a French corpus of encyclopedic documents annotated with FrameNet and an unsupervised domain discovery approach yields equivalent improvements. |
Semi-supervised Domain Adaptation for Dependency Parsing via Improved Contextualized Word Representations (2020.coling-main)
Copied to clipboard
| Challenge: | Recent advances in deep neural network models have improved parsing performance on in-domain texts . however, the problem is to improve performance on out-of-domain text data when there is only a small-scale out-domain labeled data. |
| Approach: | They propose to use adversarial learning and fine-tuning BERT to improve contextualized word representations on out-of-domain texts. |
| Outcome: | The proposed models achieve consistent improvement and fine-tune BERT processes boost parsing accuracy by a large margin. |
On the Importance of Effectively Adapting Pretrained Language Models for Active Learning (2022.acl-short)
Copied to clipboard
| Challenge: | Recent active learning approaches in NLP use off-the-shelf pretrained language models (LMs) . a poor training strategy can be catastrophic for AL, authors argue . |
| Approach: | They propose to first adapt the pretrained LM to the target task and then use it for AL. |
| Outcome: | The proposed approach provides substantial data efficiency improvements compared to the standard fine-tuning approach. |
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020.acl-main)
Copied to clipboard
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A. Smith
| Challenge: | Language models prerained on text from a wide variety of sources form the foundation of today’s NLP. |
| Approach: | They propose to tailor a pretrained model to the domain of a target task by using domain-adaptive pretraining in-domain. |
| Outcome: | The proposed model can be tailored to the domain of a target task and perform well under both high- and low-resource settings. |