Multimodal Fact-Checking with Vision Language Models: A Probing Classifier based Solution with Embedding Strategies (2025.coling-main)
Copied to clipboard
| Challenge: | Existing fact-checking systems that use text and image information are susceptible to fake news spread by social media platforms. |
| Approach: | They propose a neural probing classifier based on multimodality and embeddings from text and image encoders to represent multimodal content for fact-checking. |
| Outcome: | The proposed classifier outperforms KNN and SVM baselines in leveraging extracted embeddings, highlighting its effectiveness for multimodal fact-checking. |
Similar Papers
Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Case (2020.coling-main)
Copied to clipboard
| Challenge: | Semantic embeddings have advanced the state of the art for natural language processing tasks . but their inner workings are poorly understood and there is a shortage of analysis tools . |
| Approach: | They propose to extend visual-semantic embeddings to multimodal domains by defining probing tasks for embeddable image-caption pairs and testing them with classifiers. |
| Outcome: | The proposed probing tasks show up to 16% more accurate on visual-semantic embeddings compared to unimodal embedders . the proposed extensions to multimodal domains have been lauded as promising in natural language processing . |
Multimodal Automated Fact-Checking: A Survey (2023.findings-emnlp)
Copied to clipboard
Mubashara Akhtar, Michael Schlichtkrull, Zhijiang Guo, Oana Cocarascu, Elena Simperl, Andreas Vlachos
| Challenge: | Existing studies on automated fact-checking focus on text, but they focus on a single modality, text . multimodal misinformation is perceived as more credible by humans and spreads faster than text-only counterparts. |
| Approach: | They propose a framework for automated fact-checking that includes subtasks unique to multimodal misinformation. |
| Outcome: | The proposed framework includes subtasks unique to multimodal misinformation. |
MetaSumPerceiver: Multimodal Multi-Document Evidence Summarization for Fact-Checking (2024.acl-long)
Copied to clipboard
| Challenge: | Fact-checking claims on social media platforms poses a significant challenge due to the large volume of new claims constantly being posted without sufficient methods for verification. |
| Approach: | They propose a model that generates claim-specific summaries from multimodal multi-document datasets using a perceiver-based model that is able to handle inputs from multiple modalities of arbitrary lengths. |
| Outcome: | The proposed model outperforms the SOTA approach by 4.6% in the claim verification task on the MOCHEG dataset and shows strong performance on the new multi-document claims dataset. |
HintsOfTruth: A Multimodal Checkworthiness Detection Dataset with Real and Synthetic Claims (2025.acl-long)
Copied to clipboard
| Challenge: | Identifying checkworthy claims is the first step, but detection methods struggle with content that is (1) multimodal, (2) from diverse domains, and (3) synthetic. |
| Approach: | They propose a dataset for multimodal checkworthiness detection with 27K real-world and synthetic image/claim pairs. |
| Outcome: | The proposed dataset compares lightweight text-based encoders to multimodal models but only focus on claim-like content. |
How to Train Your Fact Verifier: Knowledge Transfer with Multimodal Open Models (2024.findings-emnlp)
Copied to clipboard
Jaeyoung Lee, Ximing Lu, Jack Hessel, Faeze Brahman, Youngjae Yu, Yonatan Bisk, Yejin Choi, Saadia Gabriel
| Challenge: | a growing influx of misinformation across news and social media is hampered by outdated foundation model training data. |
| Approach: | They propose to use large language models to scale up online policing mechanisms . they evaluate foundation model performance without continual updating . |
| Outcome: | The proposed model can improve performance without continual updating . the proposed model improves on two widely used benchmarks . |
Do Not Guess, Verify: Logic-Guided Adaptive Reasoning for Multimodal Misinformation Detection (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing multimodal misinformation detection paradigms rely on passive aggregation of multimodal features and social signals. |
| Approach: | They propose a verification-oriented framework that integrates large vision–language models into multimodal misinformation detection through explicit rationale-guided reasoning. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on multimodal misinformation detection benchmarks while significantly reducing computational cost. |
Retrieving Multimodal Information for Augmented Generation: A Survey (2023.findings-emnlp)
Copied to clipboard
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, Shafiq Joty
| Challenge: | Large Language Models (LLMs) are increasingly using multimodality to augment their generation ability, but there is no unified perception of at which stage and how to incorporate different modalities. |
| Approach: | They propose to use multimodality to augment Large Language Models (LLMs) this will provide scholars with a deeper understanding of the methods' applications and encourage them to adapt existing techniques to the fast-growing field of LLMs. |
| Outcome: | The proposed methods improve factuality, reasoning, interpretability, and robustness of the generated content. |
Multimodal Misinformation Detection by Learning from Synthetic Data with Multimodal LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Obtaining large-scale, high-quality real-world fact-checking datasets is costly . generalizability of detectors trained on synthetic data to real-life scenarios remains unclear . |
| Approach: | They propose to use synthetic data to learn from real-world data to detect multimodal misinformation . they propose to combine model-agnostic data selection methods with real-life data distributions . |
| Outcome: | The proposed method improves the performance of a small MLLM on real-world fact-checking datasets, surpassing GPT-4V. |
Can VLMs Recall Factual Associations From Visual References? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a systematic deficiency in the multimodal grounding of Vision Language Models is identified . VLMs can recall factual associations when provided a textual reference to an entity . |
| Approach: | They identify a systematic deficiency in the multimodal grounding of Vision Language Models . they show that VLMs struggle to link their internal knowledge of an entity with its image representation . |
| Outcome: | The study shows that VLMs struggle to link internal knowledge with image representations . the findings provide recommendations for future research . |
Watermarking for Factuality: Guiding Vision-Language Models Toward Truth via Tri-layer Contrastive Decoding (2025.findings-emnlp)
Copied to clipboard
Kyungryul Back, Seongbeom Park, Milim Kim, Mincheol Kwon, SangHyeok Lee, Hyunyoung Lee, Junhee Cho, Seunghyun Park, Jinkyu Kim
| Challenge: | Large Vision-Language Models (LVLMs) have shown promising results on multimodal tasks, but remain prone to hallucinations due to their reliance on a single modality or memorizing training data without properly grounding their outputs. |
| Approach: | They propose a training-free, tri-layer contrastive decoding with watermarking that uses a watermark-related question to identify a pivot layer and apply tri-layered contrastive coding to generate the final output. |
| Outcome: | The proposed method reduces hallucinations and generates more visually grounded responses. |