Papers with WHOOPS-AHA!
When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Vision-language models combine visual and textual information to perform complex tasks. conflicts between internal knowledge and external visual input can lead to hallucinations and unreliable predictions. |
| Approach: | They propose to use a dataset to identify attention heads that deliberately contradict internal commonsense knowledge to resolve cross-modal conflicts. |
| Outcome: | The proposed model can be manipulated to find out which visual inputs are conflicting . the model can then be orientated towards internal parametric knowledge or visual information . |