Papers by Aarohi Srivastava
BERTwich: Extending BERT’s Capabilities to Model Dialectal and Noisy Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models like BERT deteriorate in the face of dialect variation or noise. |
| Approach: | They propose to sandwich BERT's encoder stack between additional encoder layers trained to perform masked language modeling on noisy text. |
| Outcome: | The proposed approach promotes zero-shot transfer to dialectal text and reduces embedding space between words and noisy counterparts. |