Papers by Raj Patel
Enhancing Out-of-Vocabulary Estimation with Subword Attention (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to learn OOV word representations use advanced architectures like attention on the context of the word, but they tend to use simple structures like ngram addition or character based convolutional neural networks (CNN) |
| Approach: | They propose a transformer-based OOV estimation model that uses attention mechanisms on both the context and the subwords to learn OOV representations. |
| Outcome: | The proposed model outperforms current state-of-the-art models on OOV representations based on attention mechanisms on the context and subwords . |
Subword Attention and Post-Processing for Rare and Unknown Contextualized Embeddings (2024.findings-naacl)
Copied to clipboard
| Challenge: | Word embeddings are useful, but struggle on rare and unknown words. |
| Approach: | They propose a rare/unknown embedding architecture that focuses on contextualized representations. |
| Outcome: | The proposed architecture improves performance in most intrinsic and downstream tasks. |
AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has found that few-shot sentence classification based on pre-trained Sentence Encoders (SEs) is efficient, robust, and effective. |
| Approach: | They propose a domain-specialization approach that decouples SEPT from DAPT by training a SEPT adapter on a pre-trained PLM. |
| Outcome: | The proposed approach matches or surpasses the performance of full SEPT on DAPT-ed PLMs while significantly reducing training costs. |