| Challenge: | Existing domain-specific knowledge of domain-related tasks is lacking in pre-trained language models. |
| Approach: | They propose a domain-adaptation method which can dynamically select domain-specific tokens and guide the discriminator to emphasize them, without introducing new training parameters. |
| Outcome: | The proposed method can capture domain-specific knowledge of domain-related tasks without introducing new training parameters. |
Similar Papers
Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained masked language models perform few-shot learning, but discriminative models like ELECTRA do not fit into the paradigm. |
| Approach: | They propose to use ELECTRA to train pre-trained models to score originality of target options without introducing new parameters. |
| Outcome: | The proposed model outperforms masked language models in a wide range of tasks without adding new parameters. |
Learning to Sample Replacements for ELECTRA Pre-Training (2021.findings-acl)
Copied to clipboard
| Challenge: | Experimental results show that ELECTRA pretrains a discriminator to detect replaced tokens . despite compelling performance, there is no direct feedback loop from discriminator and generator to generator, making replacements biased to correct tokens. |
| Approach: | They propose to augment sampling with a hardness prediction mechanism to encourage the discriminator to learn what it has not acquired. |
| Outcome: | The proposed method improves ELECTRA pre-training on various downstream tasks. |
Training ELECTRA Augmented with Multi-word Selection (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-training methods for NLP tasks require massive computation resources. |
| Approach: | They propose a method that trains a discriminator to detect replaced tokens and select original tokens from candidate sets. |
| Outcome: | The proposed method improves ELECTRA based on multi-task learning on GLUE and SQUAD datasets. |
Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization (2026.acl-long)
Copied to clipboard
| Challenge: | a new approach to adapt generalist models to expert domains is needed to overcome this problem. |
| Approach: | They propose a parameter-efficient domain adaptation approach that combines vocabulary adaptation with pretraining for LLM-based text summarization. |
| Outcome: | The proposed approach reduces training time by 35-55% over continual pretraining and reduces parameter counts up to 37% w.r.t expansion-only methods. |
PEER: Pre-training ELECTRA Extended by Ranking (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing models for pre-training require expensive pre-trainer computation cost . ELECTRA model can perform replaced token detection (RTD) task with reduced pre- training cost compared to current models . |
| Approach: | They propose to extend a discriminator-based replaced token detection task into a ranker-based task . they propose to use a binary classifier to perform a more precise task with negligible additional computation cost. |
| Outcome: | The proposed model outperforms state-of-the-art models with ELECTRA in GLUE tasks given the same cost. |
Pre-trained Token-replaced Detection Model as Few-shot Learner (2022.coling-1)
Copied to clipboard
| Challenge: | Pre-trained masked language models have demonstrated remarkable few-shot learning ability . a novel approach to few- shot learning with pre-tried token-replaced detection models is proposed . |
| Approach: | They propose a method to reformulate a classification or regression task as a token-replaced detection problem by using pre-trained token-based models. |
| Outcome: | The proposed approach outperforms pre-trained masked language models in learning tasks . it can learn models with a few examples and generalize well from limited examples like humans . |
TADA: Efficient Task-Agnostic Domain Adaptation for Transformers (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained transformer-based language models are limited in their expressiveness and domain knowledge. |
| Approach: | They propose a task-agnostic domain adaptation method which is modular, parameter-efficient, and data-efficient. |
| Outcome: | The proposed method is efficient and modular, parameter-efficient, and data-efficient. |
Unsupervised Energy-based Adversarial Domain Adaptation for Cross-domain Text Classification (2021.findings-acl)
Copied to clipboard
| Challenge: | Extensive experiments on multidomain sentiment classification and yes/no question-answering classification are conducted. |
| Approach: | They propose an unsupervised energy-based adversarial domain adaptation framework that maps the text sequences from both source and target domains to a feature space. |
| Outcome: | The proposed framework improves on multidomain sentiment classification and Yes/No question-answering classification. |
Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models’ Memories (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models demonstrate excellent abilities to understand texts in the generic domain while struggling in a specific domain. |
| Approach: | They propose to decouple the feed-forward networks of the Transformer architecture into two parts to maintain old-domain knowledge and a mixture-of-adapters gate to inject domain-specific knowledge in parallel. |
| Outcome: | The proposed method achieves superior performance on in-domain, out-of-domain and knowledge-intensive tasks. |
UDAPTER - Efficient Domain Adaptation Using Adapters (2023.eacl-main)
Copied to clipboard
| Challenge: | Using adapters, unsupervised domain adaptation (UDA) is more parameter efficient and requires large-scale data to be effective. |
| Approach: | They propose to add small bottleneck layers to each layer of a pre-trained language model to make it more parameter efficient by adding adapters. |
| Outcome: | The proposed methods outperform unsupervised domain adaptation methods such as DANN and DSN in natural language inference and sentiment classification tasks. |