Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has shown the importance of training contextualised word embedding models on the domain of the target task of interest. |
| Approach: | They propose a masking strategy which adversarially masks out those tokens which are harder to reconstruct by the underlying MLM. |
| Outcome: | The proposed training strategy outperforms random masking on six unsupervised domain adaptation tasks and achieves up to +1.64 F1 score improvements. |
Similar Papers
NB-MLM: Efficient Domain Adaptation of Masked Language Models for Sentiment Analysis (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-training Masked Language Models (MLMs) on massive datasets is expensive, but it is performed for each domain or task individually and is resource-demanding. |
| Approach: | They propose a method for more efficient adaptation that focuses on predicting words with large weights of the Naive Bayes classifier trained for the task at hand. |
| Outcome: | The proposed method improves sentiment analysis by focusing on predicting words with large weights of the Naive Bayes classifier trained for the task at hand. |
Adversarial Domain Adaptation for Machine Reading Comprehension (D19-1)
Copied to clipboard
| Challenge: | Existing models for machine reading comprehension rely on large amounts of human-annotated in-domain data. |
| Approach: | They propose an unsupervised domain adaptation framework for Machine Reading Comprehension where the source domain has a large amount of labeled data, while only unlabeled passages are available in the target domain. |
| Outcome: | The proposed framework can be generalizable to different MRC models and datasets and can be extended to semi-supervised learning. |
Semi-supervised Domain Adaptation for Dependency Parsing via Improved Contextualized Word Representations (2020.coling-main)
Copied to clipboard
| Challenge: | Recent advances in deep neural network models have improved parsing performance on in-domain texts . however, the problem is to improve performance on out-of-domain text data when there is only a small-scale out-domain labeled data. |
| Approach: | They propose to use adversarial learning and fine-tuning BERT to improve contextualized word representations on out-of-domain texts. |
| Outcome: | The proposed models achieve consistent improvement and fine-tune BERT processes boost parsing accuracy by a large margin. |
A Comparative Analysis of Unsupervised Language Adaptation Methods (D19-61)
Copied to clipboard
| Challenge: | Recent proposed approaches to perform unsupervised language adaptation lack annotated resources in less-resourced languages. |
| Approach: | They propose to use Adversarial Training, Sentence Encoder Alignment and Shared-Private Architecture to perform unsupervised language adaptation without using aligned sentences. |
| Outcome: | The proposed approaches are more suitable when the source and target language datasets contain other variations in content besides the language shift. |
Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling (D19-1)
Copied to clipboard
| Challenge: | Contextualized word embeddings are becoming a ubiquitous component of natural language processing. |
| Approach: | They propose a domain-adaptive fine-tuning approach to pretrain on unlabeled text . they test this approach on sequence labeling in two challenging domains . |
| Outcome: | The proposed approach improves on sequence labeling in two domains: Early Modern English and Twitter. |
UDALM: Unsupervised Domain Adaptation through Language Modeling (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing techniques for unsupervised domain adaptation (UDA) are limited by domain shift, which leads to performance degradation. |
| Approach: | They propose a fine-tuning procedure that uses a mixed classification and Masked Language Model loss to adapt to the target domain distribution in a robust and sample efficient manner. |
| Outcome: | The proposed procedure can adapt to the target domain distribution in a robust and sample efficient manner. |
Margin-aware Unsupervised Domain Adaptation for Cross-lingual Text Labeling (2020.findings-emnlp)
Copied to clipboard
Dejiao Zhang, Ramesh Nallapati, Henghui Zhu, Feng Nan, Cicero Nogueira dos Santos, Kathleen McKeown, Bing Xiang
| Challenge: | Existing approaches to learn a model from labeled data are expensive or prohibitive. |
| Approach: | They propose an unsupervised domain adaptation algorithm that leverages labeled data in a source domain to learn a well-performing model in . they use the Margin Disparity Discrepancy algorithm to optimize the margin loss on the source domain. |
| Outcome: | The proposed approach improves on a recent theoretical work on cross-lingual document classification and NER by a large margin. |
Adversarial Domain Adaptation Using Artificial Titles for Abstractive Title Generation (P19-1)
Copied to clipboard
| Challenge: | Obtaining good quality labeled data can be difficult and expensive for abstractive summarization models . authors propose the use of artificial titles for unlabeled target documents . |
| Approach: | They propose to use artificial titles and sequential training to capture grammatical style of unlabeled target domains to adapt to/from news articles and Stack Exchange posts. |
| Outcome: | The proposed techniques can boost performance for unsupervised adaptation and fine-tuning with limited target data. |
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that pretrained Masked Language Models are not effective as universal lexical and sentence encoders off-the-shelf, i.e., without further task-specific fine-tuning on NLI, sentence similarity, or paraphrasing tasks using annotated task data. |
| Approach: | They propose a contrastive learning technique which turns pretrained MLMs into effective universal lexical and sentence encoders without additional data. |
| Outcome: | The proposed technique can turn MLMs into effective universal lexical and sentence encoders even without additional data. |
Neural Unsupervised Domain Adaptation in NLP—A Survey (2020.coling-main)
Copied to clipboard
| Challenge: | Deep neural networks excel at learning from labeled data, but learning from unlabeled data remains a challenge. |
| Approach: | They review neural unsupervised domain adaptation techniques which do not require labeled target domain data. |
| Outcome: | The proposed techniques are more challenging yet widely applicable. |