Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations (2023.acl-long)
Copied to clipboard
| Challenge: | Using hidden representations, pretrained language models are prone to overfitting due to the huge amount of parameters. |
| Approach: | They propose a method that inserts random autoencoders between hidden layers of a PLM to transform activations from the previous layers into multi-view compressed representations before feeding them into the upper layers. |
| Outcome: | The proposed method improves performance across sequence- and token-level lowresource tasks. |
Similar Papers
Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuning (2023.findings-acl)
Copied to clipboard
| Challenge: | Efficient finetuning of pretrained language transformers requires a large number of tunable parameters. |
| Approach: | They propose a language transformer finetuning strategy that introduces task-specific parameters in multiple transformer layers. |
| Outcome: | The proposed method outperforms other methods with 4,100 parameters on GLUE tasks with 5% of full finetuning performance. |
Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that powerful pre-trained language models can be fooled by small perturbations or intentional attacks. |
| Approach: | They propose a framework for fine-tuning PLMs using a masked language model and Gaussian noise to augment semantically relevant examples with sufficient diversity. |
| Outcome: | The proposed framework improves the robustness of pre-trained language models and alleviates performance degradation under adversarial attacks. |
HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing techniques to fine-tune pre-trained language models on downstream tasks are inadequate. |
| Approach: | They propose a technique to perturb hidden Transformers representations by enhancing generalization of hidden representations from different layers. |
| Outcome: | The proposed technique outperforms vanilla fine-tuning and enhances generalization of hidden representations from different layers. |
Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained Language Models (PLMs) are advanced but data labels are noisy due to the complex annotation process. |
| Approach: | They propose a framework for fine-tuning PLMs using noisy labels that incorporates guidance from Large Language Models like ChatGPT. |
| Outcome: | Experiments on synthetic and real-world noisy datasets show that the proposed framework outperforms the state-of-the-art framework. |
Small Pre-trained Language Models Can be Fine-tuned as Large Models via Over-Parameterization (2023.acl-long)
Copied to clipboard
| Challenge: | Large pre-trained language models (PLMs) have shown remarkable performance in various natural language processing tasks, outperforming small PLMs by a large margin. |
| Approach: | They propose to scale up parameters of pre-trained language models only during fine-tuning to benefit from over-parameterization. |
| Outcome: | The proposed approach can significantly boost the fine-tuning performance of small PLMs and even help small PDMs outperform 3 parameterized larger ones. |
Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have attributed such instability to the catastrophic forgetting problem in the top layers of PLMs. |
| Approach: | They propose a component-wise gradient norm clipping method to adjust convergence speed for different components to improve generalization performance, convergence speed, and training stability. |
| Outcome: | The proposed method achieves consistent improvements in terms of generalization performance, convergence speed, and training stability. |
DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Conventional fine-tuning works through updating all of the parameters in the pre-trained model, but as the size of pre-train models grows, it can be time-consuming and computationally expensive. |
| Approach: | They propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights. |
| Outcome: | The proposed framework saves 25% inference FLOPs while maintaining competitive downstream performance. |
Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a new algorithm to estimate fine-tuning performance for a target task is proposed . conventional subset selection methods require repeated training on subsets of auxiliary tasks . |
| Approach: | They propose an algorithm to fine-tune a language model for a target task by optimally using auxiliary tasks' information. |
| Outcome: | The proposed method can estimate fine-tuning performance on CPUs in seconds. |
NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better (2022.acl-short)
Copied to clipboard
| Challenge: | Existing methods for finetuning pretrained language models (PLMs) have risks in overfitting the pretraining tasks and data, which may lead to suboptimal performance. |
| Approach: | They propose a method which adds noise to parameters of PLMs before fine-tuning. |
| Outcome: | The proposed method can be used on GLUE English and XTREME multilingual benchmarks. |
Generating Datasets with Pretrained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to obtain high-quality sentence embeddings from pretrained language models require labeled data or finetuned on large set of labeles. |
| Approach: | They propose to use generative abilities of large and high-performing PLMs to generate entire datasets of labeled text pairs from scratch and fine tune much smaller and more efficient models. |
| Outcome: | The proposed approach outperforms baselines on several semantic textual similarity datasets. |