KM-BART: Knowledge Enhanced Multimodal BART for Visual Commonsense Generation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models for visual and language understanding are not capable of multimodal reasoning. |
| Approach: | They propose a Transformer-based sequence-to-sequence model capable of reasoning about commonsense knowledge from multimodal inputs of images and texts. |
| Outcome: | The proposed model performs state-of-the-art on the Visual Commonsense Generation task. |
Similar Papers
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2020.acl-main)
Copied to clipboard
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer
| Challenge: | Recent work has shown gains by improving the distribution of masked tokens and the order in which mucked tokens are predicted. |
| Approach: | They propose a denoising autoencoder for pretraining sequence-to-sequence models that corrupts text with an arbitrary noising function and learns a model to reconstruct the original text. |
| Outcome: | The proposed model outperforms RoBERTa on GLUE and SQUAD and provides a 1.1 BLEU increase over a back-translation system for machine translation. |
CK-Transformer: Commonsense Knowledge Enhanced Transformers for Referring Expression Comprehension (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing frameworks for referring expression comprehension with commonsense knowledge are lacking in the field of multimodal referring . |
| Approach: | They propose a framework for commonsense knowledge Enhanced Transformers which integrates commonsensible knowledge into representations of objects in an image. |
| Outcome: | The proposed framework improves on the existing state of the art in referring expression comprehension with commonsense knowledge (CK-Transformer) it achieves 3.14% accuracy over the existing framework. |
Learning to Ask: Multi-Decoder Fine-Tuning for Multi-Hop Visual Question Generation with External Knowledge (2026.findings-eacl)
Copied to clipboard
| Challenge: | Traditional supervised QG methods rely on tokenlevel alignment with fixed gold labels struggle to capture diverse valid question formulations. |
| Approach: | They propose a model-agnostic framework that integrates multimodal inputs with a multi-decoder architecture to optimize for multiple labels per sample. |
| Outcome: | The proposed framework improves fluency, reasoning depth, and relevance of visual questions. |
Parallel Refinements for Lexically Constrained Text Generation with BART (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing work injects lexical constraints into the output, which generates generic or ungrammatical sentences and has high computational complexity. |
| Approach: | They propose a model that incorporates pre-specified keywords into the output to control the generated text. |
| Outcome: | The proposed model decomposes the generated text into two sub-tasks and improves the sentence quality. |
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent large-scale vision-language pre-training models are powerful in multimodal classification and retrieval tasks. |
| Approach: | They propose to augment a vision-language pre-training model with a textual pre-trained language model . the model achieves 44.5% zero-shot accuracy on multimodal generation tasks . |
| Outcome: | The proposed model achieves 44.5% zero-shot accuracy on open-ended visual question answering and image captioning tasks. |
Knowledge Image Matters: Improving Knowledge-Based Visual Reasoning with Multi-Image Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Knowledge-based visual reasoning (KB-VR) is a challenging task, as it requires machines not only to understand concepts and relationships of visual scenes, but also to associate them with external world knowledge to perform chain of reasoning on open-world questions. |
| Approach: | They propose a visual knowledge card (VKC) that integrates internal visual knowledge and external world knowledge produced by a knowledge generator into an image. |
| Outcome: | The proposed model achieves new state-of-the-art results compared to previous top-performing models on three popular KB-VR benchmarks. |
MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | Language Models (LLMs) have gained increasing prominence in artificial intelligence, especially Large Language Model (LLm) due to the well-recognized reporting bias, the recording of commonsense information is significantly less than its existence in reality. |
| Approach: | They propose a Multi-mOdal REtrieval framework to leverage both text and images to enhance commonsense ability of language models. |
| Outcome: | The proposed framework can leverage both text and images to enhance commonsense ability of language models. |
KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to Visual Question Answering lack synergistic potential of scene graphs and scene graph. |
| Approach: | They propose a retrieval-and-fusion pipeline that fuses scene graphs and commonsense graphs to enable multi-modal reasoning. |
| Outcome: | Experiments on FVQA 2.0+ and MVQA benchmarks show that KG-ViP outperforms existing methods. |
DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization (2022.acl-short)
Copied to clipboard
Zheng Li, Zijian Wang, Ming Tan, Ramesh Nallapati, Parminder Bhatia, Andrew Arnold, Bing Xiang, Dan Roth
| Challenge: | Empirical analyses show that pre-trained sequence-to-sequence models can achieve a 16.5x model footprint compression ratio with little performance drop relative to full-precision counterparts. |
| Approach: | They propose to distill and quantize pre-trained sequence-to-sequence models to reduce memory and latency requirements. |
| Outcome: | Empirical results show that the proposed model achieves 16.5x model footprint compression ratio with little performance drop relative to full-precision counterparts on multiple summarization and QA datasets. |
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Combining visual modality with pretrained language models has been effective for descriptive tasks such as image captioning. |
| Approach: | They ask: do multimodal models combine visual and visual adapted language models? they find that CLIP image representations and scaling of language models do not consistently improve self-rationalization in multimodal tasks. |
| Outcome: | The proposed model types do not consistently improve self-rationalization in multimodal tasks. |