| Challenge: | Existing approaches to learn and reason over language and vision data for downstream tasks such as visual question answering (VQA) and natural language for visual reasoning (NLVR) |
| Approach: | They propose a cross-modality relevance module that is used in an end-to-end framework to learn the relevance representation between components of various input modalities under supervision of a target task. |
| Outcome: | The proposed approach shows competitive performance on two different language and vision tasks using public benchmarks and improves the state-of-the-art published results. |
Similar Papers
Visuo-Linguistic Question Answering (VLQA) Challenge (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Understanding images and text together is an important aspect of cognition and building advanced AI systems. |
| Approach: | They propose to derive joint inference about a given image-text modality and compile a question-answering corpus using an image and a reading passage. |
| Outcome: | The proposed method has better baseline performance but is still far behind human performance. |
On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in language and vision assistants have showcased impressive capabilities but suffer from a lack of transparency, limiting broader research and reproducibility. |
| Approach: | They propose to redefine the design of vision-language models by identifying key components and creating efficient models with constrained inference costs. |
| Outcome: | The proposed models achieve significant improvements in inference throughput while maintaining high performance. |
Modality Alignment between Deep Representations for Effective Video-and-Language Learning (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing Video-and-Language models do not take into account the different characteristics of video and text representations. |
| Approach: | They propose a method that exploits Centered Kernel Alignment (CKA) to enhance cross-modality attention by combining multiple modalities. |
| Outcome: | The proposed method outperforms conventional multi-modal methods significantly on video QA tasks with +3.57% accuracy increment compared to the baseline in a popular benchmark dataset. |
Visual–Linguistic Abductive Reasoning with LLMs for Knowledge-based Visual Question Answering (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent efforts to leverage large language models for reasoning focus on visual perception and language reasoning as separate processes. |
| Approach: | They propose a method that integrates visual and linguistic modalities into interpretable abductive reasoning chains. |
| Outcome: | The proposed method improves performance on AOKVQA, OKVQA and GQA by 2.31% . it uses fuzzy scoring to select the most coherent combination, enabling unified reasoning . |
A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for conditional inference on joint textual and visual clues lack multimodal context reasoning capability. |
| Approach: | They propose a multi-modal context reasoning approach that embeds textual semantics and objective image information into the pretrained language model to perform context reasoning. |
| Outcome: | The proposed approach improves on two data sets and shows 4.8% gain on the PMR. |
Object Ordering with Bidirectional Matchings for Visual Reasoning (N18-2)
Copied to clipboard
| Challenge: | et al. (2016) proposed a novel end-to-end neural model for visual reasoning with compositional natural language instructions. |
| Approach: | They propose an end-to-end neural model for a visual reasoning task based on a newly-released Cornell dataset . they use joint bidirectional attention to build a two-way conditioning between visual information and language phrases . then they use an RL-based pointer network to sort and process the varying number of unordered objects in each image and pool over the three decisions . |
| Outcome: | The proposed model achieves 4-6% absolute improvements over the state-of-the-art model on the NLVR dataset. |
Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | a recent study shows that vision-language models have modality gaps that persist even in well-aligned models. |
| Approach: | They propose a modality-dominance score to measure and leverage modality gaps . they propose automatic interpretability metrics to evaluate these features in a scalable manner . |
| Outcome: | The proposed framework allows for training-free probing and editing methods for understanding model perception across genders and generating adversarial examples. |
Delving Deeper into Cross-lingual Visual Question Answering (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual VQA have reported poor zero-shot transfer performance of current multilingual multimodal Transformers . lack of multilingual resources has hindered development and evaluation of VQA methods beyond the English language . |
| Approach: | They analyze cross-lingual VQA across different question types of varying complexity . they show that simple modifications to the standard training setup can substantially reduce the transfer gap to monolingual English performance. |
| Outcome: | The proposed model significantly reduces the transfer gap to monolingual English performance . the proposed model also improves on question types and languages . |
LXMERT: Learning Cross-Modality Encoder Representations from Transformers (D19-1)
Copied to clipboard
| Challenge: | Existing models with better representations of visual content and language have been developed for visual-content understanding. |
| Approach: | They propose a framework to learn vision-and-language connections from Transformers models . they pre-train a large-scale Transformer model with large amounts of image-and sentence pairs . |
| Outcome: | The proposed model improves state-of-the-art on two visual-reasoning tasks by 22% . the proposed model is based on a large-scale Transformer model with three encoders . |
CAST: Cross-modal Alignment Similarity Test for Vision Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Vision Language Models (VLMs) are typically evaluated with Visual Question Answering tasks which assess a model’s understanding of scenes. |
| Approach: | They propose to use visual question answering (VQA) to assess a model's understanding of scenes to probe for self-consistency across modalities. |
| Outcome: | The proposed test does not focus on objective accuracy but rather on whether VLMs are internally consistent in their outputs. |