Papers by Vivek Madan
TADPOLE: Task ADapted Pre-Training via AnOmaLy DEtection (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to solve domain shifts in NLP tasks require additional pre-training . current approaches focus on the downstream corpus when it is small, but are not effective . |
| Approach: | They propose a task-adapted pre-training framework that can be used when the downstream corpus is too small for additional pre-tuning. |
| Outcome: | The proposed framework outperforms baseline methods on biomedical, computer science, news, and movie reviews tasks. |
Representation Projection Invariance Mitigates Representation Collapse (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Fine-tuning contextualized representations by pre-trained models can lead to representation degradation, which can result in instability, sub-optimal performance, and weak generalization. |
| Approach: | They propose a regularization method to maintain the information content of representations and reduce representation collapse during fine-tuning. |
| Outcome: | The proposed method outperforms baselines on most tasks and improves out-of-distribution performance. |