| Challenge: | Language model pre-training, such as BERT, has achieved strong performance in many NLP tasks. |
| Approach: | They propose to visualize loss landscapes and optimization trajectories of fine-tuning BERT on specific datasets. |
| Outcome: | The proposed model improves performance and generalization capability across tasks. |
Similar Papers
Maximal Multiverse Learning for Promoting Cross-Task Generalization of Fine-Tuned Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent studies suggest the use of general language models for improving natural language processing tasks. |
| Approach: | They propose a method that leverages the second phase to its fullest by applying an extensive number of parallel classifier heads, which are enforced to be orthogonal, while adaptively eliminating the weaker heads during training. |
| Outcome: | The proposed method improves the generalization ability of BERT, sometimes leading to a +9% gain in accuracy. |
Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning (2020.coling-main)
Copied to clipboard
| Challenge: | Recent work has explored the suitability of pre-trained language models in low resource settings with less than 1,000 training data points. |
| Approach: | They propose to use pool-based active learning to speed up training while keeping the cost of labeling new data constant. |
| Outcome: | The proposed model can be fine-tuned to optimize for low-resource settings while keeping the cost of labeling constant. |
BERT Rediscovers the Classical NLP Pipeline (P19-1)
Copied to clipboard
| Challenge: | Pre-trained text encoders have advanced the state of the art on many NLP tasks . Qualitative analysis reveals that the model can and often does adjust this pipeline dynamically . |
| Approach: | They aim to quantify where linguistic information is captured within a network model . they aim to use pre-trained text encoders to displace static word embeddings . |
| Outcome: | The proposed model can adjust the pipeline dynamically, revealing lower-level decisions on the basis of disambiguation from higher-level representations. |
Investigating Learning Dynamics of BERT Fine-Tuning (2020.aacl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that the fine-tuning process improves performance on downstream tasks. |
| Approach: | They propose two new pre-training tasks to improve the model performance on downstream tasks. |
| Outcome: | The proposed model achieves state-of-the-art on a wide array of NLP tasks. |
How Far Is Too Far? Studying the Effects of Domain Discrepancy on Masked Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Pre-trained masked language models perform strongly on a wide variety of NLP tasks. |
| Approach: | They propose a mechanism to quantify the difference in domains between the pre-trained model and the task and partition it using a cloze task. |
| Outcome: | The proposed model performs better on openly available e-commerce datasets than the original model on scientific and biomedical datasets. |
A Closer Look at How Fine-tuning Changes BERT (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained contextualized representations are used to analyze information in NLP . however, how fine-tuning changes the underlying embedding space is less studied . |
| Approach: | They propose to use probing techniques to analyze how fine-tuning changes the embedding space of pre-trained contextualized representations. |
| Outcome: | The proposed model improves classification performance by increasing the distances between examples associated with different labels. |
Unsupervised Fine-tuning for Text Clustering (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to text clustering fine-tune pre-trained models have been limited. |
| Approach: | They propose a method to fine-tune pre-trained models unsupervisedly for text clustering by learning text representations and cluster assignments using a clustering oriented loss. |
| Outcome: | The proposed model outperforms baseline methods and achieves state-of-the-art results on three text clustering datasets. |
How to Train BERT with an Academic Budget (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large language models such as BERT are used in many NLP tasks, but their pretraining phase can be prohibitively expensive for startups and academic research groups. |
| Approach: | They propose a recipe for pretraining a large language model in 24 hours using a low-end deep learning server. |
| Outcome: | The proposed model can be trained on GLUE tasks at fraction of the cost of pretraining. |
PatchBERT: Just-in-Time, Out-of-Vocabulary Patching (2020.emnlp-main)
Copied to clipboard
| Challenge: | a pre-trained language model with low OOV can improve performance for transfer learning . a vocabulary surrogate can provide performance boosts with no additional computation cost . |
| Approach: | They propose multiple methods to mitigate OOV during downstream task fine-tuning . they demonstrate that vocabulary surrogates can provide performance boosts with no additional computation cost . |
| Outcome: | The proposed methods improve performance with the same parameter count when combined with fine-tuning. |
On the Nature of BERT: Correlating Fine-Tuning and Linguistic Competence (2022.coling-1)
Copied to clipboard
| Challenge: | Several studies on the interpretation of Neural Language Models (NLMs) focus on the linguistic generalization abilities of pre-trained models, but little attention is paid to how the linguistic knowledge of the models changes during fine-tuning. |
| Approach: | They propose to examine whether a wide range of linguistic phenomena are forgotten during fine-tuning and whether it is possible to predict the fine- tuned accuracy solely relying on the assessed linguistic competence. |
| Outcome: | The proposed model can predict the evolution of written language competence of native language learners based on the assessed linguistic competence. |