| Challenge: | Existing methods for image captioning ignore the long-tailed nature of hallucinations . a new framework is proposed to address hallucines in image captions in the open-vocabulary setting . |
| Approach: | They propose a framework to address hallucinations in image captioning in the open-vocabulary setting. |
| Outcome: | The proposed framework surpasses the CHAIR benchmark in diversity and accuracy in open-vocabulary captioning. |
Similar Papers
Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large vision-language models (LVLMs) often hallucinate and produce captions that mention concepts that cannot be found in the image. |
| Approach: | They propose to add grounding objectives to captions that explicitly align image regions or objects to text spans to reduce hallucination. |
| Outcome: | The proposed evaluation protocol reduces the amount of hallucination in LVLMs by adding grounding objectives. |
Retrieval Visual Contrastive Decoding to Mitigate Object Hallucinations in Large Vision-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Vision Language Models are not free from the issue of Object Hallucination (OH) OH is a phenomenon where LVLMs generate hallucinated objects and descriptions in their outputs. |
| Approach: | They propose a method to suppress OH by referencing images from AI-generated images at the logit level. |
| Outcome: | The proposed method significantly improves existing methods on visual contrast decoding. |
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training (2023.eacl-main)
Copied to clipboard
| Challenge: | Large-scale vision-language pre-trained (VLP) models generate unfaithful or nonsensical texts given the source input, which is called hallucination. |
| Approach: | They propose a VLP loss-based model to mitigate object hallucination by decoupling VLP objectives and a token-level image-text alignment. |
| Outcome: | The proposed model reduces object hallucination by 17.4% on two benchmarks. |
Object Hallucination in Image Captioning (D18-1)
Copied to clipboard
| Challenge: | Existing image captioning metrics do not capture image relevance . current metrics only measure similarity to ground truth captions . |
| Approach: | They propose a new image relevance metric to evaluate captioning models with veridical visual labels and assess their rate of object hallucination. |
| Outcome: | The proposed metrics show that models with veridical visual labels have higher hallucination rates than models with lower hallucinosity. |
CEBC: Conformal Evidence-Bounded Control for Low-Hallucination Vision–Language Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing mitigation approaches reduce hallucinated object mentions at the cost of degraded generation quality or require expensive retraining and task-specific supervision. |
| Approach: | They propose a lightweight framework for low-hallucination vision–language generation . it uses evidence-bounded minimal editing to revise or suppress unsupported referenced entities . |
| Outcome: | The proposed framework reduces hallucinations while maintaining or improving quality metrics. |
Critic-Driven Decoding for Mitigating Hallucinations in Data-to-text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Hallucination of text lacking grounding in input data is a problem in neural data-to-text generation. |
| Approach: | They propose to combine probabilistic output of a generator language model with the output of an “text critic” classifier which guides the generation by assessing the match between the input data and the generated text. |
| Outcome: | The proposed method improves on the WebNLG and OpenDialKG benchmarks. |
ALOHa: A New Measure for Hallucination in Captioning Models (2024.naacl-short)
Copied to clipboard
Suzanne Petryk, David Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph Gonzalez, Trevor Darrell
| Challenge: | Existing metric for object hallucination, CHAIR, is limited to MS COCO objects and synonyms. |
| Approach: | They propose a new open-vocabulary metric, ALOHa, which leverages large language models to measure object hallucinations. |
| Outcome: | The proposed metric correctly identifies 13.6% more hallucinated objects than CHAIR on HAT and 30.8% more on nocaps. |
Bridging by Word: Image Grounded Vocabulary Construction for Visual Captioning (P19-1)
Copied to clipboard
| Challenge: | Existing research on image captioning generates frequent n-grams with irrelevant words. |
| Approach: | They propose to construct an image-grounded vocabulary incorporating visual information and relations among words into the decoding process directly. |
| Outcome: | The proposed framework is compared with state-of-the-art models on MS COCO and Flickr30k and shows that it is more efficient than existing models. |
Hallucination Detection for Grounded Instruction Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for generating instructions for navigation generate references to objects or actions that are inconsistent with what a human follower would perform or encounter along the path. |
| Approach: | They propose a weakly supervised approach that detects hallucinated references by using a pre-trained vision-language model. |
| Outcome: | The proposed model outperforms baseline models and supervised models on generating navigation instructions. |
Low-Hallucination and Efficient Coreference Resolution with LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models have shown promising results in coreference resolution, but they face a critical issue: hallucinations. |
| Approach: | They propose a low-hallucination and efficient solution to the problem of hallucinations . they propose efficient constrained decoding for coreference resolution . |
| Outcome: | The proposed approach achieves better performance on the English OntoNotes development set. |