Adaptive Weighted Proxy Tuning: Efficient Gray-Box Steering for Image Captioning. (2026.acl-industry)
Copied to clipboard
| Challenge: | Proxy tuning is a decoding-time approach that fails to account for instance-specific variations in model certainty and domain shift. |
| Approach: | They propose a gray-box steering framework that dynamically modulates the logit contributions of a large base model, a fine-tuned expert, and an untune . |
| Outcome: | Adaptive Weighted Proxy Tuning achieves performance parity with fine-tuned models while remaining parameter-free. |
Similar Papers
Black-Box Tuning of Vision-Language Models with Effective Gradient Approximation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large vision-language models are often not open-source due to preventing abuse or commercial factors. |
| Approach: | They propose a method for parameter-efficient fine-tuning to improve model accessibility . large models are often not open-source due to preventing abuse or commercial factors . they propose implementing a lightweight adapter over the output feature of an inaccessible model . |
| Outcome: | The proposed methods improve on 11 benchmarks and are made publicly available. |
CBP-Tuning: Efficient Local Customization for Black-box Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Customized black-box prompt tuning is a new approach to customize large language models . however, as models grow, the resources required for training and deployment become increasingly expensive . |
| Approach: | They propose a framework that facilitates efficient local customization while preserving bidirectional privacy. |
| Outcome: | The proposed framework facilitates efficient local customization while preserving bidirectional privacy. |
Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in fine-tuning Vision-Language Models have seen the success of prompt tuning and adapter tuning. |
| Approach: | They propose a method to fine-tune CLIP without introducing any overhead of extra parameters. |
| Outcome: | The proposed method improves CLIP by 7.27% average harmonic mean accuracy. |
Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work focuses on training vision-language models with long, detailed image captions, but small-scale VLMs struggle to balance the richness of these captions with the risk of hallucinations. |
| Approach: | They propose an evaluation framework that breaks down generated captions into individual propositions, assessing each in isolation. |
| Outcome: | The proposed framework outperforms baselines in both automatic metrics and human evaluations on small-scale vision-language models with long, detailed captions. |
CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Methods for adapting language models to new tasks and domains have traditionally assumed white-box access to the model and work by modifying its parameters. |
| Approach: | They propose a method for adapting large language models to new domains and tasks . they fine-tune a small white-box LM and combine it with a large black-box model at the probability level through a network, learned on a smaller validation set. |
| Outcome: | The proposed method improves performance in all cases, while using a domain expert 23x smaller. |
XtremeCLIP: Extremely Parameter-efficient Tuning for Low-resource Vision Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to fine-tune visual-language understanding (VLU) require tasks-specific designs and sufficient training data. |
| Approach: | They propose a simple yet efficient paradigm for low-resource Visual Language Understanding (VLU) they reformulate a series of VLU tasks as an open-book affinity-matching problem. |
| Outcome: | The proposed framework outperforms baselines in low-resource settings. |
Crafting Adversarial Inputs for Large Vision-Language Models Using Black-Box Optimization (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing white-box jailbreak methods require full model accessibility and require computational costs. |
| Approach: | They propose a black-box jailbreak attack using Zeroth-Order optimization using ZO-SPSA. |
| Outcome: | The proposed method achieves highest jailbreak success rate on three LVLMs, including InstructBLIP, LLaVA and MiniGPT-4. |
Evaluating Parameter-Efficient Finetuning Approaches for Pre-trained Models on the Financial Domain (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale language models with millions, billions, or trillions of trainable parameters are becoming increasingly popular. |
| Approach: | They compare performance of financial BERT-like models to their fully fine-tuned counterparts by using parameter-efficient tuning methods. |
| Outcome: | The proposed approaches match full fine-tuning performance on common NLP tasks, but are less studied in finance. |
Efficient Low-Resource Language Adaptation via Multi-Source Dynamic Logit Fusion (2026.acl-long)
Copied to clipboard
| Challenge: | Proxy Tuning offers a logit-level strategy for introducing scaling effects, but it often fails in LRL settings because the large model’s weak LRL competence might overwhelm the knowledge of specialized smaller models. |
| Approach: | They propose a logit-based framework that balances LRL competence from a continually pretrained small model, task competence from high-resource language instruction tuning, and the scaling benefits of large models. |
| Outcome: | Experiments across four model families and eight LRLs show that TriMix outperforms single-model baselines and Proxy Tuning. |
Selective Self-to-Supervised Fine-Tuning for Generalization in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can be fine-tuned on task-specific data to improve performance on target tasks but can be overfitted resulting in a loss of generalization. |
| Approach: | They propose a method that uses the correct model responses from a training set to fine-tune the model using the correct response and the gold response for the remaining samples. |
| Outcome: | The proposed approach reduces model specialization during the fine-tuning stage while improving generalization. |