Papers by Aamir Miyajiwala
Towards Simple and Efficient Task-Adaptive Pre-training for Text Classification (2022.aacl-short)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models are extensively trained on massive heterogeneous datasets, known as pre-training datasets. |
| Approach: | They propose to use Domain Adaptive Pre-training and Task-Adaptive pre-training as intermediate steps before the final finetuning task to cover the target domain vocabulary. |
| Outcome: | The proposed approach is computationally efficient, with 78% fewer parameters trained during TAPT. |