Papers with TAPT
Towards Simple and Efficient Task-Adaptive Pre-training for Text Classification (2022.aacl-short)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models are extensively trained on massive heterogeneous datasets, known as pre-training datasets. |
| Approach: | They propose to use Domain Adaptive Pre-training and Task-Adaptive pre-training as intermediate steps before the final finetuning task to cover the target domain vocabulary. |
| Outcome: | The proposed approach is computationally efficient, with 78% fewer parameters trained during TAPT. |
Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Task-adaptive pre-training (TAPT) and Self-training can be complementary with simple TFS protocol. |
| Approach: | They propose to use task-adaptive pre-training and self-training to combine TAPT and ST with a simple TFS protocol to achieve strong combined gains across six datasets. |
| Outcome: | The proposed method can achieve strong combined gains across six datasets covering sentiment classification, paraphrase identification, natural language inference, named entity recognition and dialogue slot classification. |
Rethinking Semi-supervised Learning with Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Semi-supervised learning (SSL) is a popular setting to make use of unlabelled data . Currently, there are two popular approaches to make effective use of the unlabelled datasets . |
| Approach: | They compare semi-supervised learning (SSL) and task-adaptive pre-training (TAPT) they find TAPT is a stronger and more robust SSL learner, even when using just a few hundred unlabelled samples . |
| Outcome: | The proposed methods improve model performance across different NLP tasks and data sizes. |
AfroXLMR-Social: Adapting Pre-trained Language Models for African Languages Social Media Text (2025.findings-emnlp)
Copied to clipboard
Tadesse Destaw Belay, Israel Abebe Azime, Ibrahim Said Ahmad, David Ifeoluwa Adelani, Idris Abdulmumin, Abinew Ali Ayele, Shamsuddeen Hassan Muhammad, Seid Muhie Yimam
| Challenge: | Domain adaptive pre-training and task-adaptive pre- training (TAPT) are popular methods to reduce this bias for low-resource languages, but they have not been explored for African multilingual encoders. |
| Approach: | They propose a large-scale social media and news domain corpus for continual pre-training on African languages. |
| Outcome: | The proposed methods improve performance on three subjective tasks, including sentiment analysis, multi-label emotion, and hate speech classification, while TAPT improves performance on other related tasks. |
NLoPT: N-gram Enhanced Low-Rank Task Adaptive Pre-training for Efficient Language Model Adaption (2024.lrec-main)
Copied to clipboard
| Challenge: | Pre-trained Language Models (PLMs) have superior performance on downstream tasks . however, conventional TAPT adjusts all parameters of the PLMs, which distorts the learned generic knowledge embedded in the original PLM's weights. |
| Approach: | They propose a two-step n-gram enhanced low-rank task adaptive pre-training method to customize a PLM to the downstream task. |
| Outcome: | The proposed method improves performance on six datasets from four domains. |