KnowDR-REC: Auditing Knowledge-Conditioned Visual Grounding in Referring Expression Comprehension (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics suggest that Multimodal large language models have acquired fine-grained visual grounding capabilities. |
| Approach: | They propose a benchmark to assess Referring Expression Comprehension (REC) that uses intra-image visual cues to localize target objects and a controllable evaluation mechanism to test sensitivity to fine-grained factual changes. |
| Outcome: | The proposed benchmarks show that multimodal large language models have a high level of performance on the RefCOCO family of benchmarks. |
Similar Papers
FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension (2024.emnlp-main)
Copied to clipboard
| Challenge: | Referring Expression Comprehension (REC) is a cross-modal task that objectively evaluates the capabilities of language understanding, image comprehension, and language-to-image grounding. |
| Approach: | They propose to use a new reference expression comprehension (REC) dataset to evaluate the capabilities of language understanding, image comprehension, and language-to-image grounding. |
| Outcome: | The proposed model is able to reject scenarios where the target object is not visible in the image, a key aspect often overlooked in existing models and approaches. |
ISR: Self-Refining Referring Expressions for Entity Grounding (2025.acl-long)
Copied to clipboard
| Challenge: | Entity grounding is a crucial task in the construction of multimodal knowledge graphs. |
| Approach: | They propose a novel scheme to enhance the multimodal large language model's capability to generate high quality REs for the given entities as explicit contextual clues. |
| Outcome: | The proposed method surpasses other methods in entity grounding, highlighting its effectiveness, robustness and potential for broader applications. |
ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension (2022.acl-long)
Copied to clipboard
| Challenge: | Visual referring expression comprehension (ReC) models can be trained for a domain, but it remains unclear if they can be applied in a zero-shot manner to more complex tasks like ReC. |
| Approach: | They propose a method that repurposes CLIP, a state-of-the-art large-scale model, for training a referring expression comprehension model for a new visual domain. |
| Outcome: | The proposed model reduces the gap between zero-shot baselines from prior work and supervised models by as much as 29% on RefCOCOg, and on ReFGTA (video game imagery), and its relative improvement over supervised ReC models is 8%. |
InteGround: On the Evaluation of Verification and Retrieval Planning in Integrative Grounding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing grounding approaches work well for simple queries, but many real-world information needs require synthesizing multiple pieces of evidence. |
| Approach: | They introduce "integrative grounding" to evaluate the ability to ground large language models in external knowledge sources. |
| Outcome: | The proposed approach is robust to redundant evidence, but rationalizes using internal knowledge when information is incomplete. |
VLURes: Benchmarking Long-Text Grounding and Cross-Lingual Robustness in Vision Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | ***VLURes** provides a practical testbed for long-text grounding and multilingual robustness in web-realistic agent settings. |
| Approach: | They propose a multilingual benchmark for evaluating vision-language models under long-text grounding. |
| Outcome: | ***VLURes** provides a testbed for long-text grounding and multilingual robustness in web-realistic agent settings. |
Ground Then Rank: Revisiting Knowledge-Based VQA with Training-Free Entity Identification (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing multi-modal retrieval augmented generation (MM-RAG) methods tightly couple entity discrimination and section-level evidence ranking into a single re-ranking stage, leading to high cost and limited generalization. |
| Approach: | They propose a framework that decouples entity identification from section-level re-ranking. |
| Outcome: | The proposed framework outperforms fine-tuned multi-modal re-ranking baselines while reducing training and inference complexity. |
Read Before Grounding: Scene Knowledge Visual Grounding via Multi-step Parsing (2025.coling-main)
Copied to clipboard
| Challenge: | Existing VG datasets use simple textual descriptions with limited attribute and spatial information between images and text. |
| Approach: | They propose a method that transforms visual knowledge into concise, information-dense visual descriptions. |
| Outcome: | The proposed method significantly improves performance of multimodal grounding models. |
Words Aren’t Enough, Their Order Matters: On the Robustness of Grounding Visual Referring Expressions (2020.acl-main)
Copied to clipboard
| Challenge: | Visual referring expression recognition is a task that requires natural language understanding in the context of an image. |
| Approach: | They propose to use contrastive learning and multi-task learning to increase the robustness of ViLBERT, the current state-of-the-art model for this task. |
| Outcome: | The proposed methods are 12% to 23% lower in performance than the established progress for this task. |
Simple-VGC: Enhancing Visual Grounding in Multimodal Reasoning via Adaptive Tool Composition (2026.acl-long)
Copied to clipboard
| Challenge: | Existing multimodal large language models suffer from systematic failures in basic visual understanding. |
| Approach: | They propose a tool-augmented reasoning framework with three targeted compensation strategies to address these problems. |
| Outcome: | The proposed framework improves visual grounding by re-injecting the original image to mitigate visual forgetting, the authors show . the proposed framework also improves the accuracy of the visual inputs, the researchers show - and the results are promising . |
The Model Agreed, But Didn’t Learn: Diagnosing Surface Compliance in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models internalize vast world knowledge as parametric memory, yet inherit the staleness and errors of their source corpora. |
| Approach: | They propose a framework that subjects models to discriminative self-assessment under diverse contextual pressures to scrutinize subtle behavioral nuances induced by memory modifications. |
| Outcome: | The proposed framework achieves high benchmarks without overwriting internal beliefs, while recursive modifications accumulate representational residues, triggering cognitive instability and permanently diminishing the reversibility of the model’s memory state. |