An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (N19-1)
Copied to clipboard
| Challenge: | Existing transfer learning methods employ language models pretrained on large generic corpora, but results come at a high computational cost and require task-specific architectures. |
| Approach: | They propose a transfer learning approach that combine a task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. |
| Outcome: | The proposed method surpasses well established transfer learning methods with greater level of complexity on a variety of affective and text classification tasks surpassing well established methods with higher level of difficulty. |
Similar Papers
Robust Transfer Learning with Pretrained Language Models through Adapters (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to transfer learning with pretrained transformer-based language models are not robust and can be adversarial. |
| Approach: | They propose a simple yet effective adapter-based approach to fine-tune language models on downstream tasks. |
| Outcome: | The proposed approach improves stability and adversarial robustness in transfer learning to various downstream tasks. |
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to fine-tune deep pretrained language models face catastrophic forgetting problems. |
| Approach: | They propose a recall and learn mechanism which integrates pretraining and downstream tasks into a single mechanism. |
| Outcome: | The proposed method achieves state-of-the-art performance on the GLUE benchmark and better average performance than directly fine-tuning of BERT-large. |
Transfer Learning in Natural Language Processing (N19-5)
Copied to clipboard
| Challenge: | supervised machine learning is based on learning in isolation, a single predictive model for a task using a dataset. |
| Approach: | They present an overview of modern transfer learning methods in natural language processing . they review examples and case studies on how models can be integrated and adapted . |
| Outcome: | The proposed methods improve upon the state-of-the-art on a wide range of NLP tasks. |
An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-training large language models is a standard practice in the natural language processing community. |
| Approach: | They propose to use elastic weight consolidation to mitigate catastrophic forgetting when pre-trained large language models are evaluated on generic benchmarks. |
| Outcome: | The proposed model achieves state-of-the-art on out-of domain tasks with minimal pre-training . elastic weight consolidation provides best overall scores yielding only a 0.33% drop in performance across seven generic tasks while remaining competitive in bio-medical tasks. |
On the Importance of Effectively Adapting Pretrained Language Models for Active Learning (2022.acl-short)
Copied to clipboard
| Challenge: | Recent active learning approaches in NLP use off-the-shelf pretrained language models (LMs) . a poor training strategy can be catastrophic for AL, authors argue . |
| Approach: | They propose to first adapt the pretrained LM to the target task and then use it for AL. |
| Outcome: | The proposed approach provides substantial data efficiency improvements compared to the standard fine-tuning approach. |
Improving Cross Lingual Transfer by Pretraining with Active Forgetting (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has shown that encoder-only LLMs show impressive cross lingual transfer of their capabilities from English to other languages. |
| Approach: | They propose a pretraining strategy that uses active forgetting to achieve similar cross lingual transfer in decoder-only LLMs. |
| Outcome: | The proposed model improves cross lingual transfer capabilities on non-English languages despite being trained on English data. |
Investigating Transferability in Pretrained Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on deep NLP models has centered on probing, a method that involves training classifiers for different tasks on model representations. |
| Approach: | They propose a method for determining the impact of each pretrained layer on transfer task performance by ablation. |
| Outcome: | The proposed method shows that pretraining models improve performance on downstream tasks . the results highlight the limitations of methods that operate on frozen models or single data samples. |
Incorporating Syntactic Knowledge into Pre-trained Language Model using Optimization for Overcoming Catastrophic Forgetting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models lack syntactic knowledge for many tasks that handle complex or long sentences. |
| Approach: | They propose to use pre-trained language models to incorporate syntactic knowledge into a model by adding additional syntatic knowledge to the model. |
| Outcome: | The proposed model can be easily applied to downstream tasks that require syntactic knowledge. |
Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to pretrain Masked Language Models (MLMs) are expensive and require a full forward and backward pass over the entire model. |
| Approach: | They propose to learn a shallow mini-model from a fraction of a large model's parameters and plug it into a larger model for rapid cross-lingual transfer. |
| Outcome: | Experiments on XNLI, MLQA and PAWS-X show that mini-model adaptation matches the standard approach using up to 2.3x less compute on average. |
Memorisation versus Generalisation in Pre-trained Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art pre-trained language models have been shown to memorise facts and perform well with limited amounts of training data. |
| Approach: | They propose to extend pre-trained language models to generalise and memorise facts in noisy and low-resource scenarios. |
| Outcome: | The proposed extension improves performance in low-resource named entity recognition tasks. |