VLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a significant drawback of Vision-language Models is their reliance on static training data, leading to outdated information and limited contextual awareness. |
| Approach: | They propose a framework with knowledge-enhanced reranking and noise-injected training to improve the VLM's ranking ability. |
| Outcome: | The proposed framework is based on a simple yet effective instruction template and is able to induce its ranking ability and serve it as a reranker to precisely filter the top-k retrieved images. |
Similar Papers
Progressive Re-ranking for Multimodal Retrieval-Augmented Generation via Curriculum Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to improve retrieval performance of large language models are limited by static knowledge. |
| Approach: | They propose a multimodal re-ranking framework that combines curriculum learning with fine-grained reranking and multimodal section reassessment to improve CLIP-based visual coarse-grain retrieval. |
| Outcome: | The proposed framework achieves state-of-the-art answer accuracy and competitive retrieval performance on InfoSeek and Enc-VQA. |
FastV-RAG: Towards Fast and Fine-Grained Video QA with Retrieval-Augmented Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for retrieval-augmented generation are inefficient and often fail to maintain high answer quality. |
| Approach: | They propose an efficient VLM-based RAG framework built on a speculative decoding pipeline and a similarity-based filtering strategy to mitigate errors. |
| Outcome: | The proposed framework reduces inference latency without sacrificing correctness . it achieves comparable or higher accuracy than standard approaches while speeding up inference by approximately 2x . |
MS-RAG: Simple and Effective Multi-Semantic Retrieval-Augmented Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for large language models suffer from poor indexing and inference speed . graph-based RAGs heavily rely on LLM for retrieval thus inference slow . |
| Approach: | They propose retrieval-augmented generation (RAG) which integrates knowledge with dense vectors to build a multi-semantic RAG. |
| Outcome: | The proposed method achieves state-of-the-art performance with faster inference speed compared to existing methods . |
Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that LLMs face challenges in effectively using retrieved information . authors propose a method that considers LLM as "Information Refiner" |
| Approach: | They propose a method that considers LLMs as "Information Refiners" they propose INFO-RAG, which is low-cost and general across various tasks . |
| Outcome: | The proposed method improves performance of LLaMA2 by 9.39% relative points . it is low-cost and general across various tasks, and is robust and in-context learning is possible . |
OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for Knowledge-Based Visual Question Answering lack multimodal retrieval . large language models (LLMs) have demonstrated remarkable generalization and reasoning capabilities in text-based systems. |
| Approach: | They propose a multimodal vision-language retrieval-augmented generation system that harmonizes multiple modalities and modality to enhance retrieval. |
| Outcome: | The proposed system achieves state-of-the-art retrieval performance and competitive answers on InfoSeek and Encyclopedic-VQA benchmarks. |
Uplift-RAG: Uplift-Driven Knowledge Preference Alignment for Retrieval-Augmented Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing efforts to estimate document utility rely on downstream generation performance, which conflates the influence of external documents with the intrinsic knowledge of the LLM. |
| Approach: | They propose an uplift-based definition of document utility that quantifies each document’s marginal benefit over the LLM’s internal knowledge. |
| Outcome: | The proposed framework improves the performance of the LLM by incorporating external retrieved documents into the model. |
RAG in the Wild: On the (In)effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation (2026.findings-acl)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) enhances large language models by integrating external knowledge retrieved at inference time. |
| Approach: | They evaluate RAG systems using MassiveDS, a large-scale datastore with mixture of knowledge. |
| Outcome: | The proposed approach improves performance on knowledge-intensive NLP tasks. |
DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models focus on identifying relevant documents, but embedding similarity often limits accuracy. |
| Approach: | They propose a method to generate hard negative queries per page instead of negative pages per query . they propose to refine ranking of an initial set of retrieved documents using hard negative mining . |
| Outcome: | The proposed approach outperforms existing models and significantly improves retrieval performance. |
Retrieval-augmented GUI Agents with Generative Guidelines (2025.emnlp-main)
Copied to clipboard
| Challenge: | GUI agents powered by vision-language models struggle with real-world tasks due to their complex nature and limited training data. |
| Approach: | They propose a lightweight vision-language model that leverages web tutorials at inferencetime to synthesize GUI agents. |
| Outcome: | The proposed agent outperforms baseline GUI agents and surpasses other inference baselines by 2.6% to 13.3% across two model sizes. |
RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing medical large vision language models often generate inaccurate and irrelevant answers that do not align with established medical facts. |
| Approach: | They propose a strategy for controlling factuality risk through calibrated selection of the number of retrieved contexts and a preference dataset to fine-tune the model. |
| Outcome: | The proposed model achieves an average improvement of 20.8% on three medical VQA datasets. |