Unsupervised Domain Adaptation for Sparse Retrieval by Filling Vocabulary and Word Frequency Gaps (2022.aacl-main)
Copied to clipboard
| Challenge: | IR models with a pretrained language model outperform lexical approaches like BM25 for vocabulary mismatch. |
| Approach: | They propose an unsupervised domain adaptation method by filling vocabulary gaps by expanding queries and documents through an MLM. |
| Outcome: | The proposed method outperforms the current state-of-the-art domain adaptation method on datasets with a large vocabulary gap from a source domain. |
Similar Papers
Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to train pre-trained models require domain-specific data and computational resources. |
| Approach: | They propose a domain-aware N-gram Adaptor to incorporate unseen and domain-specific words into a generic pretrained model. |
| Outcome: | The proposed model can improve on eight low-resource tasks using limited data with lower computational costs. |
Breaking Boundaries in Retrieval Systems: Unsupervised Domain Adaptation with Denoise-Finetuning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing domain adaptation methods for dense retrieval models use unadapted rerank models, leading to imprecise labels. |
| Approach: | They propose to adapt a rerank model to the target domain before using it for label generation. |
| Outcome: | The proposed model achieves better results across three retrieval datasets. |
Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has shown the importance of training contextualised word embedding models on the domain of the target task of interest. |
| Approach: | They propose a masking strategy which adversarially masks out those tokens which are harder to reconstruct by the underlying MLM. |
| Outcome: | The proposed training strategy outperforms random masking on six unsupervised domain adaptation tasks and achieves up to +1.64 F1 score improvements. |
AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to fine-tune a language model with a large corpus in a general domain are suboptimal for downstream data when domain discrepancy exists. |
| Approach: | They propose to consider the pretrained vocabulary as an optimizable parameter . they add domain specific vocabulary based on a tokenization statistic . their method achieved consistent performance improvements on diverse domains . |
| Outcome: | The proposed method achieves consistent performance improvements on diverse domains. |
Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling (D19-1)
Copied to clipboard
| Challenge: | Contextualized word embeddings are becoming a ubiquitous component of natural language processing. |
| Approach: | They propose a domain-adaptive fine-tuning approach to pretrain on unlabeled text . they test this approach on sequence labeling in two challenging domains . |
| Outcome: | The proposed approach improves on sequence labeling in two domains: Early Modern English and Twitter. |
Entity Extraction in Low Resource Domains with Selective Pre-training of Large Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to perform named entity recognition (NER) on unlabeled data are difficult to obtain in low-resource domains. |
| Approach: | They propose ways to use unlabeled data for pretraining to improve performance in downstream tasks. |
| Outcome: | The proposed methods outperform models trained on unlabeled data on seven domains. |
PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models (2020.tacl-1)
Copied to clipboard
| Challenge: | PERL is a representation learning model that uses labeled data from the source domain and unlabeled data not necessarily drawn from the target domain. |
| Approach: | They propose a model that extends contextualized word embedding models with pivot-based fine-tuning to address this bottleneck. |
| Outcome: | The proposed model outperforms strong baselines across 22 sentiment classification domain adaptation setups and improves in-domain model performance. |
GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval (2022.naacl-main)
Copied to clipboard
| Challenge: | Dense retrieval approaches suffer from the lexical gap and require large amounts of training data. |
| Approach: | They propose an unsupervised method for domain adaptation that uses query generator and pseudo labeling from a cross-encoder to improve retrieval performance. |
| Outcome: | The proposed method outperforms state-of-the-art retrieval methods on domain-specialized datasets by 9.3 points nDCG@10 on six tasks. |
Efficient Hierarchical Domain Adaptation for Pretrained Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to allow domain adaptation to diverse domains are expensive and require continuing training in-domain. |
| Approach: | They propose a method to permit domain adaptation to many diverse domains using a computationally efficient adapter approach. |
| Outcome: | The proposed method allows domain adaptation to many diverse domains while avoiding negative interference between unrelated domains. |
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains (2021.findings-acl)
Copied to clipboard
| Challenge: | Large pre-trained models suffer from domain shift and are not optimal for specific domains. |
| Approach: | They propose a general approach to developing small, fast and effective pretrained models for specific domains by adapting off-the-shelf general pretrained model and performing task-agnostic knowledge distillation in target domains. |
| Outcome: | The proposed approach achieves better performance over the BERT BASE model in domain-specific tasks while 3.3 smaller and 5.1 faster than the BRT BASE. |