| Challenge: | Several studies have investigated the reasons behind the effectiveness of fine-tuning, usually through the lens of probing. |
| Approach: | They propose to investigate the reasons behind the effectiveness of fine-tuning by examining the impact of data size on the extent of encoded linguistic knowledge. |
| Outcome: | The proposed probes show that the size of the training data affects the recoverability of the changes made to the model’s linguistic knowledge. |
Similar Papers
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (2021.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models can be fine-tuned to produce state-of-the-art results for a wide range of language understanding tasks. |
| Approach: | They propose to analyze fine-tuning through the lens of intrinsic dimension . they show that pre-trained models have a low intrinsic dimension reparameterization . |
| Outcome: | The proposed model can achieve 90% of the full parameter performance levels on MRPC with low data regime. |
Predicting Fine-Tuning Performance with Probing (2022.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale neural models have recently demonstrated impressive performance in language understanding tasks, typically evaluated by their fine-tuned performance. |
| Approach: | They propose to use probing to extract a proxy signal widely used in model development to predict fine-tuning performance. |
| Outcome: | The proposed method predicts fine-tuning performance with errors 40% - 80% smaller than baselines. |
On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | linguistic knowledge encoded in pre-trained contextual embeddings is poorly understood . fine-tuning can be used to investigate the representations of pre-train models . |
| Approach: | They propose to investigate fine-tuning of contextualized embedding models through sentence-level probing. |
| Outcome: | The proposed method improves probing accuracy for three pre-trained models. |
Unveiling the Generalization Power of Fine-Tuned Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional multitasking abilities, but the comprehensive effects of fine-tuning on the LLMs’ generalization ability are not fully understood. |
| Approach: | They conduct extensive experiments across five distinct language tasks on different datasets to investigate whether fine-tuning affects the generalization ability intrinsic to LLMs. |
| Outcome: | The proposed model can generalize to different domains and tasks by integrating the in-context learning strategy during fine-tuning on generation tasks. |
On the data requirements of probing (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to probe neural networks are expensive and require large datasets. |
| Approach: | They propose a method to estimate the required number of data samples in probing datasets . they use a classification task to encode a text with a deep neural network . |
| Outcome: | The proposed method estimates the required number of data samples in two probing configurations and proves it is statistically powerful. |
The Effect of Language Diversity When Fine-Tuning Large Language Models for Translation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Prior research on language diversity in LLM fine-tuning has reported benefits while others find no benefits. |
| Approach: | They find that expanding language diversity during fine-tuning improves translation quality . they also show that increased language diversity creates more language-agnostic representations . |
| Outcome: | The proposed model improves translation quality for unsupervised and supervised pairs . the results plateau or decrease beyond a certain diversity threshold. |
Dynamics of Instruction Fine-Tuning for Chinese Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Instruction tuning is a burgeoning method to elicit the general intelligence of Large Language Models. |
| Approach: | They investigate the effects of data quantity, model size, and data construction methods on instruction tuning for Chinese LLMs. |
| Outcome: | The proposed model includes over 40,000 high-quality instruction instances covering ten underlying abilities. |
Dissecting Fine-Tuning Unlearning in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning-based unlearning are ineffective at completely erasing model-embedded knowledge, but their true effectiveness remains unclear. |
| Approach: | They propose to use activation patching and parameter restoration experiments to examine the limitations of fine-tuning-based unlearning methods for erasing harmful, sensitive, or copyrighted information within large language models. |
| Outcome: | The proposed methods alter the model’s knowledge retrieval process rather than genuinely erasing the problematic knowledge embedded in the model parameters. |
A Closer Look at How Fine-tuning Changes BERT (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained contextualized representations are used to analyze information in NLP . however, how fine-tuning changes the underlying embedding space is less studied . |
| Approach: | They propose to use probing techniques to analyze how fine-tuning changes the embedding space of pre-trained contextualized representations. |
| Outcome: | The proposed model improves classification performance by increasing the distances between examples associated with different labels. |
Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide (2026.tacl-1)
Copied to clipboard
| Challenge: | Pre-trained language models provide strong foundations, but effective adaptation under data scarcity requires efficient and efficient fine-tuning techniques. |
| Approach: | They propose to review parameter-efficient fine-tuning techniques that lower training and deployment costs and domain and cross-lingual adaptation methods for both encoder and decoder models. |
| Outcome: | The proposed techniques lower training and deployment costs, domain and cross-lingual adaptation methods, and model specialization strategies. |