MMCoQA: Conversational Question Answering over Text, Tables, and Images (2022.acl-long)
Copied to clipboard
| Challenge: | Existing conversational QA systems only use a single knowledge source, e.g., paragraphs or knowledge graph, and assume it contains enough evidence to extract answers to users' questions. |
| Approach: | They propose a task to answer users' questions with multimodal knowledge sources via multi-turn conversations using a multimodal dataset. |
| Outcome: | The proposed task brings a series of research challenges, including but not limited to priority, consistency, and complementarity of multimodal knowledge. |
Similar Papers
Unifying Text, Tables, and Images for Multimodal Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to multimodal question answering rely on single-modal or bi-modal models, which limit their ability to integrate information across all modalities. |
| Approach: | They propose a framework that unifies three different input modalities into a text-to-text format by employing position-enhanced table linearization and diversified image captioning techniques. |
| Outcome: | The proposed framework unifies three input modalities into a text-to-text format using position-enhanced table linearization and diversified image captioning techniques. |
M3-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge-based VQA benchmarks focus on coarse-grained categories and simple reasoning over single entities. |
| Approach: | They propose a knowledge-based Visual Question Answering benchmark to enhance multimodality evaluation. |
| Outcome: | The proposed benchmark improves evaluation of multimodal large language models in fine-grained multimodal entity understanding and complex multihop reasoning. |
MIMOQA: Multimodal Input Multimodal Output Question Answering (2021.naacl-main)
Copied to clipboard
| Challenge: | Multimodal research has picked up significantly in the space of question answering with the task being extended to visual question answering, charts question answering as well as multimodal input question answering. |
| Approach: | They propose a multimodal question-answering task that produces a unimodal textual output as the answer through human experiments. |
| Outcome: | The proposed framework outperforms existing frameworks on both automatic and human metrics. |
Knowledge-Aware Reasoning over Multimodal Semi-structured Tables (2024.findings-emnlp)
Copied to clipboard
Suyash Mathur, Jainit Bafna, Kunal Kartik, Harshita Khandelwal, Manish Shrivastava, Vivek Gupta, Mohit Bansal, Dan Roth
| Challenge: | Existing datasets for tabular question answering focus on text within cells, but real-world data is multimodal, often blending images such as symbols, faces, icons, patterns, and charts with textual content. |
| Approach: | They propose a dataset to assess whether current AI models can perform knowledge-aware reasoning on multimodal structured data. |
| Outcome: | The proposed dataset is a robust benchmark for advancing AI’s comprehension and capabilities in analyzing multimodal structured data. |
Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that multimodal machine translation systems exhibit decreased sensitivity to visual information when text inputs are complete. |
| Approach: | They propose to generate parallel VQA style pairs from source text to foster more robust cross-modal interaction. |
| Outcome: | The proposed approach generates parallel VQA style pairs from the source text, fostering more robust cross-modal interaction. |
II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have focused on assessing the model’s overall accuracy without evaluating it on different reasoning cases. |
| Approach: | They propose a novel idea to identify and improve multi-modal multi-hop reasoning in VQA by using two new language prompts to find a reasoning path to reach its answer. |
| Outcome: | The proposed model improves multi-modal multi-hop reasoning in visual question answering (VQA) it finds that the proposed model is easy to answer, simply demanding “single-hop” reasoning, whereas only a few questions require “multi-hop.” |
Data-Centric Improvements for Enhancing Multi-Modal Understanding in Spoken Conversation Modeling (2025.findings-acl)
Copied to clipboard
| Challenge: | Conversational assistants are increasingly popular across diverse real-world applications . speech data constitute high-dimensional signals that are difficult to model even for frontier models . |
| Approach: | They propose a data-centric customization approach for enhancing multimodal understanding in conversational speech modeling. |
| Outcome: | The proposed model achieves state-of-the-art on the Spoken-SQuAD benchmark using 10% of training data with open-weight models. |
Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal large language models are trained on single-turn vision question-answering tasks, which do not accurately reflect real-world human conversations. |
| Approach: | They propose a large-scale multi-turn multimodal dialogue dataset that uses rules and GPT assistance to generate a multi-turned multimodal dialog dataset. |
| Outcome: | The proposed dataset is a strong benchmark for multi-turn multimodal dialogue learning . it features complex dialogues with contextual dependencies that force models to track, ground, and recall information across multiple turns and disparate visual regions. |
MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation (2023.acl-long)
Copied to clipboard
| Challenge: | MMDialog is a dataset of 1.08 million real-world dialogues with 1.53 million unique images across 4,184 topics. |
| Approach: | They propose to use a curated set of 1.08 million dialogues with 1.53 million unique images to generalize the open domain. |
| Outcome: | The proposed system can predict responses to multi-modal content with state-of-the-art techniques and measure their performance. |
FM2DS: Few-Shot Multimodal Multihop Data Synthesis with Knowledge Distillation for Question Answering (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods focus on single-hop, single-modality, or short texts, limiting real-world applications . despite advances in visual question answering, this multihop setting remains underexplored due to a lack of quality datasets. |
| Approach: | They propose a framework for creating a high-quality dataset for multimodal multihop question answering . they use a 5-stage pipeline to acquire relevant multimodal documents from Wikipedia . |
| Outcome: | The proposed framework outperforms existing methods on multimodal multihop question answering datasets. |