Papers by Anand Raghunathan
TokenDrop + BucketSampler: Towards Efficient Padding-free Fine-tuning of Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-training of Language Models (LMs) is a challenge due to its huge computational footprint. |
| Approach: | They propose a framework that improves the efficiency and accuracy of LM fine-tuning by removing padding tokens from sequences that are variable-length . |
| Outcome: | The proposed framework accelerates fine-tuning on diverse downstream tasks by 10.61X while producing models that are up to 1.17% more accurate compared to conventional fine-uning. |