Papers with VarMAE
VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models have been widely applied to standard benchmarks due to the limited resources available in a domain. |
| Approach: | They propose a Transformer-based language model called VarMAE for domain-adaptive language understanding that encodes the context of a token into a smooth latent distribution. |
| Outcome: | Experiments on science- and finance-domain NLU tasks show that the proposed model can be efficiently adapted to new domains with limited resources. |