| Challenge: | Existing approaches to solve domain shifts in NLP tasks require additional pre-training . current approaches focus on the downstream corpus when it is small, but are not effective . |
| Approach: | They propose a task-adapted pre-training framework that can be used when the downstream corpus is too small for additional pre-tuning. |
| Outcome: | The proposed framework outperforms baseline methods on biomedical, computer science, news, and movie reviews tasks. |
Similar Papers
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020.acl-main)
Copied to clipboard
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A. Smith
| Challenge: | Language models prerained on text from a wide variety of sources form the foundation of today’s NLP. |
| Approach: | They propose to tailor a pretrained model to the domain of a target task by using domain-adaptive pretraining in-domain. |
| Outcome: | The proposed model can be tailored to the domain of a target task and perform well under both high- and low-resource settings. |
TADA: Efficient Task-Agnostic Domain Adaptation for Transformers (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained transformer-based language models are limited in their expressiveness and domain knowledge. |
| Approach: | They propose a task-agnostic domain adaptation method which is modular, parameter-efficient, and data-efficient. |
| Outcome: | The proposed method is efficient and modular, parameter-efficient, and data-efficient. |
Task-adaptive Pre-training of Language Models with Word Embedding Regularization (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models acquire domain-independent knowledge through pre-training with massive textual resources. |
| Approach: | They propose a task-adaptive pre-training process that makes static embeddings close to the word embedds obtained in the target domain. |
| Outcome: | The proposed process improves on BioASQ and SQuAD when the pre-training corpora were not dominated by indomain data. |
Domain Adaptation with BERT-based Domain Classification and Data Selection (D19-61)
Copied to clipboard
| Challenge: | Modern deep neural models with millions of parameters can easily adapt to a new learning task and dataset when enough supervision is given. |
| Approach: | They propose a domain adaptation framework based on curriculum learning and domain-discriminative data selection. |
| Outcome: | The proposed framework outperforms discrepancy-based methods on transfer tasks while consuming only fraction of training budget. |
Variational Pretraining for Semi-supervised Text Classification (P19-1)
Copied to clipboard
| Challenge: | Empirically, we show the relative strength of VAMPIRE against computationally expensive contextual embeddings and other popular semi-supervised baselines under low resource settings. |
| Approach: | They propose a lightweight framework for effective text classification when data and computing resources are limited. |
| Outcome: | The proposed framework is compared with expensive contextual embeddings and semi-supervised baselines under low resource settings. |
Different Strokes for Different Folks: Investigating Appropriate Further Pre-training Approaches for Diverse Dialogue Tasks (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models can be fine-tuned on domain-specific unlabeled data . however, most further pre-training works just keep running the conventional pre- training task . |
| Approach: | They propose to add a further pre-training phase to the model to improve downstream tasks . they propose to use a domain-adaptive pre-tuning phase to fine-tune the models on unlabeled data . |
| Outcome: | The proposed method improves multiple task-oriented dialogue downstream tasks. |
Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to train pre-trained models require domain-specific data and computational resources. |
| Approach: | They propose a domain-aware N-gram Adaptor to incorporate unseen and domain-specific words into a generic pretrained model. |
| Outcome: | The proposed model can improve on eight low-resource tasks using limited data with lower computational costs. |
Towards Simple and Efficient Task-Adaptive Pre-training for Text Classification (2022.aacl-short)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models are extensively trained on massive heterogeneous datasets, known as pre-training datasets. |
| Approach: | They propose to use Domain Adaptive Pre-training and Task-Adaptive pre-training as intermediate steps before the final finetuning task to cover the target domain vocabulary. |
| Outcome: | The proposed approach is computationally efficient, with 78% fewer parameters trained during TAPT. |
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains (2021.findings-acl)
Copied to clipboard
| Challenge: | Large pre-trained models suffer from domain shift and are not optimal for specific domains. |
| Approach: | They propose a general approach to developing small, fast and effective pretrained models for specific domains by adapting off-the-shelf general pretrained model and performing task-agnostic knowledge distillation in target domains. |
| Outcome: | The proposed approach achieves better performance over the BERT BASE model in domain-specific tasks while 3.3 smaller and 5.1 faster than the BRT BASE. |
Robust Transfer Learning with Pretrained Language Models through Adapters (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to transfer learning with pretrained transformer-based language models are not robust and can be adversarial. |
| Approach: | They propose a simple yet effective adapter-based approach to fine-tune language models on downstream tasks. |
| Outcome: | The proposed approach improves stability and adversarial robustness in transfer learning to various downstream tasks. |