Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in fine-tuning Vision-Language Models have seen the success of prompt tuning and adapter tuning. |
| Approach: | They propose a method to fine-tune CLIP without introducing any overhead of extra parameters. |
| Outcome: | The proposed method improves CLIP by 7.27% average harmonic mean accuracy. |
Similar Papers
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2022.acl-short)
Copied to clipboard
| Challenge: | Large pre-trained models are expensive to train and deploy . large data sets make finetuning expensive to deploy - a new paradigm . |
| Approach: | They propose a sparse-finetuning method where only bias-terms are modified . they show that applying BitFit on pre-trained BERT models is competitive . |
| Outcome: | The proposed method is competitive with sparse-finetuning methods on small-to-medium training data. |
Modular and Parameter-Efficient Fine-Tuning for NLP Models (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | State-of-the-art language models in NLP perform best when fine-tuned even on small datasets. |
| Approach: | They provide an overview of parameter-efficient fine-tuning methods and highlight similarities and differences . they highlight benefits and usage scenarios of a neglected property of parameter efficient models . |
| Outcome: | This paper provides an overview of parameter-efficient fine-tuning methods . it highlights similarities and differences by presenting them in a unified view . |
Parameter-Efficient Fine-Tuning: Is There An Optimal Subset of Parameters to Tune? (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent research has illuminated the possibility of selective parameter-efficient fine-tuning, which retains the inference speed of the original model and comes at no additional computational cost. |
| Approach: | They propose to selectively update only a small subset of parameters during the fine-tuning process, keeping the remaining parameters frozen during training. |
| Outcome: | The proposed methods retain the inference speed of the original model and come at no additional computational cost. |
ViFT: Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Visual instruction tuning is the predominant technology in eliciting multimodal task-solving capabilities of large vision-language models. |
| Approach: | They propose a visual instruction-free fine-tuning framework for large vision-language models . they require only text-only instructions and image caption data during training . |
| Outcome: | The proposed framework is based on visual instruction tuning, but requires images as input . it can achieve state-of-the-art performance on several downstream benchmarks with less training data. |
Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing cross-lingual transfer methods that use labeled data and linguistic resources would consume excessive resources for a large number of languages. |
| Approach: | They propose a parameter-efficient cross-lingual transfer learning framework that utilizes a translation-based alignment method to mitigate multilingual disparities. |
| Outcome: | The proposed framework reduces disparities among languages and improves cross-lingual transfer results in low-resource scenarios while keeping and fine-tuning only a small number of parameters. |
A Parameter-Efficient and Fine-Grained Prompt Learning for Vision-Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Current vision-language models extract semantic information from large-scale cross-modal associations, limiting performance and efficiency. |
| Approach: | They propose a detail-oriented prompt learning method to implement fine-grained multi-modal semantic alignment with merely 0.25M trainable parameters. |
| Outcome: | The proposed method implements fine-grained multi-modal semantic alignment with merely 0.25M trainable parameters. |
Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing fine-tuning approaches for compositional understanding compromise performance in zero-shot multi-modal tasks. |
| Approach: | They propose a method to enhance compositional understanding in pre-trained vision and language models without sacrificing performance in zero-shot multi-modal tasks. |
| Outcome: | The proposed method achieves compositionality on par with state-of-the-art models and retains strong multi-modal capabilities. |
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning. |
| Approach: | They propose to fine tune masked language models with training examples and task descriptions to reduce prompt engineering by using null prompts. |
| Outcome: | The proposed prompts can be used to improve few-shot learning by finetuning only the bias terms while updating only 0.1% of the parameters. |
Evaluating Parameter-Efficient Finetuning Approaches for Pre-trained Models on the Financial Domain (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale language models with millions, billions, or trillions of trainable parameters are becoming increasingly popular. |
| Approach: | They compare performance of financial BERT-like models to their fully fine-tuned counterparts by using parameter-efficient tuning methods. |
| Outcome: | The proposed approaches match full fine-tuning performance on common NLP tasks, but are less studied in finance. |
On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation (2021.acl-long)
Copied to clipboard
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, Luo Si
| Challenge: | Existing studies have shown that adapter-based tuning is more parameter-efficient than fine-tuning. |
| Approach: | They propose to add adapter modules to a pretrained language model and update the parameters of adapter module when learning on a downstream task. |
| Outcome: | The proposed method outperforms fine-tuning on low-resource and cross-lingual tasks and settings. |