Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work focuses on training vision-language models with long, detailed image captions, but small-scale VLMs struggle to balance the richness of these captions with the risk of hallucinations. |
| Approach: | They propose an evaluation framework that breaks down generated captions into individual propositions, assessing each in isolation. |
| Outcome: | The proposed framework outperforms baselines in both automatic metrics and human evaluations on small-scale vision-language models with long, detailed captions. |
Similar Papers
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era (2025.findings-acl)
Copied to clipboard
Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, Jiajun Chen
| Challenge: | Image captioning has been a challenge for vision-language researchers for decades . current VLMs focus on tasks like visual question answering (YA) but image captioning is not as advanced as expected. |
| Approach: | They evaluate VLMs' performance on image captioning using human annotations . they find that some metrics show high caption-level agreement with humans . |
| Outcome: | The proposed model outperforms open-source models on image captioning . it achieves 93.4% correlation with human rankings at $4 per test . |
VGA: Vision GUI Assistant - Minimizing Hallucinations through Image-Centric Fine-Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing Large Vision-Language Models (VLMs) often overly rely on internal text-based knowledge while neglecting visual inputs. |
| Approach: | They propose a model that balances attention image and text to enhance interpretation and reduce hallucinations by using a visual input. |
| Outcome: | The proposed model improves interpretation and reduces hallucinations by balancing attention image and text to enhance interpretation and reduction of hallucinosity. |
Enhancing Large Vision-Language Models with Ultra-Detailed Image Caption Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing pipelines for generating high-quality, ultra-detailed image captions are limited by the scarcity of image caption data. |
| Approach: | They propose a pipeline for generating high-quality, ultra-detailed image captions that integrates both pre-processing and post-processor stages. |
| Outcome: | The proposed pipeline improves LVLMs' perception and cognitive abilities across multiple vision-language benchmarks. |
Low-Rank Adaptation for Multilingual Summarization: An Empirical Study (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pre-trained Large Language Models have significantly advanced NLP, but their ever-increasing size poses significant challenges for conventional fine-tuning. |
| Approach: | They investigate the potential of Low-Rank Adaptation (LoRA) in multilingual summarization, a task that is challenging and relatively unexplored. |
| Outcome: | The proposed method outperforms full fine-tuning and cross-lingual transfer strategies in multilingual summarization tasks. |
Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in fine-tuning Vision-Language Models have seen the success of prompt tuning and adapter tuning. |
| Approach: | They propose a method to fine-tune CLIP without introducing any overhead of extra parameters. |
| Outcome: | The proposed method improves CLIP by 7.27% average harmonic mean accuracy. |
Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in instruction-tuned Large Vision-Language Models (LVLMs) have imbued the models with the ability to generate high-level, image-grounded explanations with ease. |
| Approach: | They propose to use a multiple granularity attribute-centric benchmark and training mixture to evaluate LVLMs’ fine-grained visual comprehension ability. |
| Outcome: | The proposed model improves on LLaVa-1.5, InstructBLIP and GPT-4V and demonstrates that they struggle to generate descriptive visual attributes based on a concept that appears within an input image despite their prominent zero-shot image captioning ability. |
Less for More: Enhanced Feedback-aligned Mixed LLMs for Molecule Caption Generation and Fine-Grained NLI Evaluation (2025.acl-long)
Copied to clipboard
| Challenge: | Recent trends have led to the use of multimodal models to learn molecular and linguistic representations, either in separate but coordinated spaces or in a common space. |
| Approach: | They propose a novel atomic-level evaluation method leveraging off-the-shelf Natural Language Inference (NLI) models for use in the unseen chemical domain. |
| Outcome: | The proposed method surpasses state-of-the-art models in the unseen chemical domain while relying on a granularity-based evaluation method. |
Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide (2026.tacl-1)
Copied to clipboard
| Challenge: | Pre-trained language models provide strong foundations, but effective adaptation under data scarcity requires efficient and efficient fine-tuning techniques. |
| Approach: | They propose to review parameter-efficient fine-tuning techniques that lower training and deployment costs and domain and cross-lingual adaptation methods for both encoder and decoder models. |
| Outcome: | The proposed techniques lower training and deployment costs, domain and cross-lingual adaptation methods, and model specialization strategies. |
Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-training Large Language Models (LLMs) on textual corpora embeds substantial factual knowledge in their parameters, which is essential for excelling in various downstream applications. |
| Approach: | They propose to use supervised fine-tuning to align large language models to new factual information that is not acquired through pre-training. |
| Outcome: | The proposed model is trained to generate facts that are not grounded in pre-existing knowledge, but hallucinates when examples with new knowledge are learned. |
VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation (2026.acl-long)
Copied to clipboard
Yixia Li, Yaqing Shi, Zhiwen Ruan, Dongdong Zhang, Lingjie Jiang, Shaohan Huang, Yun Chen, Guanhua Chen, Furu Wei
| Challenge: | Multimodal large language models have advanced rapidly, yet most remain English-centric . scaling multilingual multimodal instruction tuning is limited by the scarcity and high cost of non-English image–text supervision. |
| Approach: | They propose a framework that decouples multilingual language enhancement from visual alignment by composing complementary task vectors over a shared LLM backbone. |
| Outcome: | The proposed framework achieves competitive performance with a fully multimodally trained model using less than 2% of the text data. |