Visual-Textual Alignment for Graph Inference in Visual Dialog (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to visual dialog do not understand semantic dependencies between visual and textual contents. |
| Approach: | They propose a Visual-Textual Alignment for Graph Inference network that makes up the lack of structural inference in visual dialog. |
| Outcome: | The proposed model outperforms existing models on a VisDial dataset. |
Similar Papers
Reasoning Visual Dialog with Sparse Graph Learning and Knowledge Transfer (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Visual dialog is a task of answering questions grounded in an image using dialog history as context. |
| Approach: | They propose a Sparse Graph Learning method to formulate visual dialog as a graph structure learning task. |
| Outcome: | The proposed model outperforms the state-of-the-art models on the VisDial v1.0 dataset. |
Dual Attention Networks for Visual Reference Resolution in Visual Dialog (D19-1)
Copied to clipboard
| Challenge: | Visual dialog (VisDial) requires a dialog agent to answer a series of questions grounded in an image. |
| Approach: | They propose dual attention networks (DAN) for visual reference resolution in VisDial. |
| Outcome: | The proposed model outperforms the previous state-of-the-art model on VisDial datasets. |
GoG: Relation-aware Graph-over-Graph Network for Visual Dialog (2021.findings-acl)
Copied to clipboard
| Challenge: | Experimental results show that our model outperforms the strong baseline in both generative and discriminative settings by a significant margin. |
| Approach: | They propose a relation-aware graph-over-graph network (GoG) for visual dialog . their model outperforms the strong baseline in both generative and discriminative settings . |
| Outcome: | The proposed model outperforms baseline models in both generative and discriminative settings by a significant margin. |
Beyond Embeddings: The Promise of Visual Table in Visual Reasoning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Visual representation learning has been a cornerstone in computer vision for decades. |
| Approach: | They propose a visual representation tailored for visual reasoning that provides instance-level world knowledge and detailed attributes that are essential for visual reason. |
| Outcome: | The proposed visual tables outperform existing models on 11 visual reasoning benchmarks. |
Modeling Coreference Relations in Visual Dialog (2021.eacl-main)
Copied to clipboard
| Challenge: | Visual dialog is a vision-language task where an agent needs to answer a series of questions grounded in an image based on the understanding of the dialog history and the image. |
| Approach: | They propose two soft constraints that can improve the model’s ability of resolving coreferences in dialog in an unsupervised way based on linguistic knowledge and discourse features of human dialog. |
| Outcome: | The proposed model achieves state-of-the-art performance on the VisDial v1.0 dataset without pretraining on other vision language datasets. |
Visual-Linguistic Dependency Encoding for Image-Text Retrieval (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to image-text retrieval ignore semantic discrepancies caused by syntactic structure in natural language expressions and relationships among visual entities. |
| Approach: | They propose a visual-linguistic dependency encoder framework which explicitly models the dependency information among textual words and interaction patterns between image regions. |
| Outcome: | The proposed framework outperforms existing methods on a vision-linguistic compositional structure reasoning dataset. |
See or Say Graphs: Agent-Driven Scalable Graph Understanding with Vision-Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have explored textual graph descriptions and visual modalities for VLMs to understand graphs. |
| Approach: | They propose a unified framework that enhances both scalability and modality coordination in graph understanding by integrating textual and visual modalities. |
| Outcome: | GraphVista scales to large graphs, 200 larger than those used in existing benchmarks, and consistently outperforms existing textual, visual, and fusion-based methods. |
Advancement in Graph Understanding: A Multimodal Benchmark and Fine-Tuning of Vision-Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Graph data organizes complex relationships and interactions between objects . Graph neural networks (GNNs) are becoming more popular in graph learning . |
| Approach: | They propose a new paradigm for interactive and instructional graph data understanding and reasoning . they first evaluate the capabilities of public VLMs in graph learning from multiple aspects . |
| Outcome: | The proposed model achieves an accuracy increase of 5%-15% compared to baseline models . the best-performing model achieve scores comparable to Gemini in GPT-asissted Evaluation . |
Multimodal Logical Inference System for Visual-Textual Entailment (P19-2)
Copied to clipboard
| Challenge: | Recent studies of multimodal inference provide challenging tasks such as visual question answering and visual reasoning. |
| Approach: | They propose an unsupervised multimodal logical inference system that can prove entailment relations between texts and images by combing semantic parsing and theorem proving. |
| Outcome: | The proposed system can handle semantically complex sentences for visual-textual inference. |
MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering (2023.acl-long)
Copied to clipboard
Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, Julian Eisenschlos
| Challenge: | Visual language models that are pretraining on natural images or image-text pairs crawled from the web perform poorly on visual language tasks such as ChartQA and ChartQA. |
| Approach: | They propose to perform several pretraining tasks that cover plot deconstruction and numerical reasoning which are key capabilities in visual language modeling. |
| Outcome: | The proposed model outperforms state-of-the-art methods on benchmarks such as PlotQA and ChartQA by as much as 20%. |