Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained large language models can be fine-tuned with instruction tuning to align the model responses with human intentions. |
| Approach: | They investigate the effectiveness of parameter efficient fine-tuning (PEFT) of the Q-Former with visual reasoning benchmarks ScienceQA and IconQA. |
| Outcome: | The proposed model achieves comparable performance to full fine-tuning using under 2% of the trainable parameters. |
Similar Papers
L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Quantization-aware PEFT methods have been developed to reduce memory and computational costs associated with large language models. |
| Approach: | They propose a method that integrates Quantization-Aware Training (QAT) with LoRA to reduce memory overhead and improve model accuracy. |
| Outcome: | The proposed method significantly reduces QAT’s memory overhead while preserving the advantage of QAT in producing fully quantized LLMs with high accuracy. |
A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models are pre-trained on trillions of tokens and instruction-tuned or aligned to specific preferences. |
| Approach: | They propose guidelines to help researchers perform more effective parameter-efficient LLM alignment. |
| Outcome: | The proposed methods outperform preference optimization and outperformed pre-trained models on three key axes. |
PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark (2026.eacl-long)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods reduce the number of trainable parameters while maintaining strong downstream performance. |
| Approach: | They propose a unified benchmark for evaluating diverse PEFT methods on autoregressive LLMs. |
| Outcome: | The proposed methods reduce trainable parameters while maintaining strong downstream performance. |
Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization (2025.emnlp-main)
Copied to clipboard
Shuo Xing, Peiran Li, Yuping Wang, Ruizheng Bai, Yueqi Wang, Chan-Wei Hu, Chengxuan Qian, Huaxiu Yao, Zhengzhong Tu
| Challenge: | emergence of large Vision Language Models (VLMs) has broadened the capabilities of single-modal Large Language Model (LLM) but VLMs are prone to significant hallucinations, especially in the form of cross-modal inconsistencies. |
| Approach: | They propose a new alignment framework that leverages image retrieval to integrate both textual and visual preference signals. |
| Outcome: | The proposed framework mitigates hallucinations more effectively than previous methods . it maintains robustness and scalability across a wide range of VLM sizes and architectures . |
XtremeCLIP: Extremely Parameter-efficient Tuning for Low-resource Vision Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to fine-tune visual-language understanding (VLU) require tasks-specific designs and sufficient training data. |
| Approach: | They propose a simple yet efficient paradigm for low-resource Visual Language Understanding (VLU) they reformulate a series of VLU tasks as an open-book affinity-matching problem. |
| Outcome: | The proposed framework outperforms baselines in low-resource settings. |
Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models are inefficient and redundant . a light-PEFT framework can be used to prune redundant parameters during training . |
| Approach: | They propose a parameter-efficient fine-tuning framework that freezes most parameters of the foundation model and finetuns only a small number of parameters. |
| Outcome: | The proposed framework achieves training and inference speedup, reduces memory usage, and maintains comparable performance and plug-and-play feature of PEFT. |
Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning (PEFT) methods are important in low-resource language (LRL) Neural Machine Translation (NMT) but their practical effectiveness varies significantly across different languages. |
| Approach: | They evaluated the performance of 8 parameters-efficient fine-tuning methods with 15 architectures using the SacreBLEU score. |
| Outcome: | The Houlsby+Inversion adapter outperforms the baseline architectures in both in-domain and out-domain tests and the Houlson+Inverter achieves the best performance overall. |
Token-level Inference-Time Alignment for Vision-Language Models (2026.findings-acl)
Copied to clipboard
Kejia Chen, Junjun Zheng, Jiawen Zhang, Manxi Lin, Xiao Pan, Jiacong Hu, Jian Lou, Zunlei Feng, Mingli Song
| Challenge: | Vision-Language Models (VLMs) often prioritize linguistic fluency over visual fidelity . despite widespread adoption, VLMs often exhibit a critical failure mode: hallucination . |
| Approach: | They propose a framework for Token-level Inference-Time Alignment that steers the decoding process without updating the base model parameters. |
| Outcome: | The proposed framework improves performance on 13 benchmarks across architectures . it boosts LLaVA-1.5-7B by 8.6% on MMVet and achieves a 74.0 MMStar score . |
BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning are limited by computational and storage requirements. |
| Approach: | They propose a budget-guided iterative search strategy to disentangle binary module and rank dimension search spaces and early selection strategies based on parameter budgets. |
| Outcome: | The proposed method significantly improves search efficiency on public benchmarks. |
PEFT-Factory: Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models (2026.eacl-demo)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods address the increasing size of Large Language Models (LLMs). |
| Approach: | They propose a framework for efficient fine-tuning Large Language Models (LLMs) they aim to train only a small percentage of the full model's parameters . |
| Outcome: | Xu et al., 2023; Ding e t al, 2024; Lialin e al. 2023) show that using PEFT methods can improve performance. |