VICTR: Visual Information Captured Text Representation for Text-to-Vision Multimodal Tasks (2020.coling-main)
Copied to clipboard
| Challenge: | Existing text-to-image generation models focus on generating high resolution images and neglect understanding text descriptions. |
| Approach: | They propose a visual contextual text representation which captures rich visual semantic information of objects from text input. |
| Outcome: | The proposed visual contextual text representation improves on the state-of-the-art models. |
Similar Papers
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Combining visual modality with pretrained language models has been effective for descriptive tasks such as image captioning. |
| Approach: | They ask: do multimodal models combine visual and visual adapted language models? they find that CLIP image representations and scaling of language models do not consistently improve self-rationalization in multimodal tasks. |
| Outcome: | The proposed model types do not consistently improve self-rationalization in multimodal tasks. |
Aligning Images and Text with Semantic Role Labels for Fine-Grained Cross-Modal Understanding (2022.lrec-1)
Copied to clipboard
| Challenge: | Currently, image retrieval systems can retrieve relevant results for diverse inputs, but they do not provide a way to intentionally inject variety into the search results. |
| Approach: | They propose a multimodal dataset that combines semantic annotations with image bounding boxes. |
| Outcome: | The proposed system improves image retrieval performance and flexibility. |
MEVTR: A Multilingual Model Enhanced with Visual Text Representations (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models that generate multilingual text representations perform poorly on low-resource languages due to lack of representation space and model capacity. |
| Approach: | They propose a multilingual model enhanced with visual text representations which complements textual representations and extends multilingual representation space with visual representations. |
| Outcome: | The proposed model outperforms state-of-the-art models on zero-shot cross-lingual transfer tasks without the target language adapter. |
Image Retrieval from Contextual Descriptions (2022.acl-long)
Copied to clipboard
| Challenge: | a new multimodal challenge challenges vision-and-language models to integrate context into their representations. |
| Approach: | They propose a multimodal challenge to integrate context into vision-and-language models . they benchmark several state-of-the-art models using cross-encoders and bi-encodings . |
| Outcome: | The proposed model lags behind human models on imageCoDe, compared with human models. |
Graph Convolution for Multimodal Information Extraction from Visually Rich Documents (N19-2)
Copied to clipboard
| Challenge: | Visually rich documents (VRDs) present information in the form of both text and vision. |
| Approach: | They propose a graph convolution based model to combine textual and visual information presented in VRDs. |
| Outcome: | The proposed model outperforms existing models on two real-world datasets. |
Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions (2025.emnlp-main)
Copied to clipboard
| Challenge: | Contrastively trained Vision-Language Models exhibit shallow language understanding, manifesting bag-of-words behaviour. |
| Approach: | They propose a vision-free, single-encoder retrieval pipeline to replace traditional text-to-image retrieval paradigm with structured image descriptions. |
| Outcome: | The proposed approach reduces the modality gap and improves compositionality and performance on short and long caption queries. |
Exploiting Pseudo Image Captions for Multimodal Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to multimodal summarization with multimodal output (MSMO) lack reference images for training, and exposure of image captions during training is inconsistent with MSMO’s task settings. |
| Approach: | They propose a coarse-to-fine image-text alignment mechanism to identify the most relevant sentence of each image in a document, resembling the role of image captions in capturing visual knowledge. |
| Outcome: | The proposed method sets up state-of-the-art on all intermodality and intramodality metrics and improves on image recommendation precision. |
Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Summarization with Multimodal Output (MSMO) is a new approach to produce a multimodal summary that integrates both text and relevant images. |
| Approach: | They propose an Entity-Guided Multimodal Summarization model that integrates both text and relevant images to produce a multimodal summary. |
| Outcome: | The proposed model integrates text-image and entity-image information and refines image selection through knowledge distillation from a pre-trained vision-language model. |
Towards Text-Image Interleaved Retrieval (2025.acl-long)
Copied to clipboard
Xin Zhang, Ziqi Dai, Yongqi Li, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, Jun Yu, Wenjie Li, Min Zhang
| Challenge: | Existing multimodal information retrieval models rely on single-image inputs . current models use a dense retrieval paradigm, but this approach is not effective . |
| Approach: | They propose a text-image interleaved retrieval task where query and document are interleaves . they adapt off-the-shelf retrievers and build a dense baseline by interleaded multimodal large language model . |
| Outcome: | The proposed model achieves significant improvements over the baseline by substantially fewer visual tokens. |
Visually-Enhanced Phrase Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Large-scale vision-language pre-training models generate high-quality textual representations, which often outperform models that are purely text-based, such as BERT. |
| Approach: | They propose to utilize both textual and visual encoders of multi-modal pre-trained models to enhance language understanding tasks by generating an image associated with a textual prompt. |
| Outcome: | The proposed method outperforms models that are purely text-based on visual and textual understanding tasks and significantly improves the entity clustering task. |