Papers by Ruijia Xu
Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models’ Memories (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models demonstrate excellent abilities to understand texts in the generic domain while struggling in a specific domain. |
| Approach: | They propose to decouple the feed-forward networks of the Transformer architecture into two parts to maintain old-domain knowledge and a mixture-of-adapters gate to inject domain-specific knowledge in parallel. |
| Outcome: | The proposed method achieves superior performance on in-domain, out-of-domain and knowledge-intensive tasks. |
Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to train pre-trained models require domain-specific data and computational resources. |
| Approach: | They propose a domain-aware N-gram Adaptor to incorporate unseen and domain-specific words into a generic pretrained model. |
| Outcome: | The proposed model can improve on eight low-resource tasks using limited data with lower computational costs. |