“Let’s not Quote out of Context”: Unified Vision-Language Pretraining for Context Assisted Image Captioning (2023.acl-industry)
Copied to clipboard
| Challenge: | Large enterprises have several teams to create their content for the purpose of marketing, campaigning, or even maintaining a brand presence. |
| Approach: | They propose a new unified Vision-Language (VL) model with a focus on context-assisted image captioning where the caption is generated based on both the image and its context. |
| Outcome: | The proposed model achieves state-of-the-art with an improvement of up to 8.34 CIDEr score on the benchmark news image captioning datasets. |
Similar Papers
Focus! Relevant and Sufficient Context Selection for News Image Captioning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work only coarsely leverages the article to extract the necessary context, which makes it difficult for models to identify relevant events and named entities. |
| Approach: | They propose to use a vision and language retrieval model CLIP to localize the visually grounded entities in the news article and then capture the non-visual entities via an open relation extraction model. |
| Outcome: | The proposed model significantly improves on existing models and achieves state-of-the-art on multiple benchmarks. |
CapOnImage: Context-driven Dense-Captioning on Image (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing image captioning systems generate narrative captions for images, which are spatially detached from the image in presentation. |
| Approach: | They propose a task called captioning on image which generatesense captions at different locations of the image based on contextual information. |
| Outcome: | The proposed model achieves the best results in both captioning accuracy and diversity aspects. |
Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing models require large amounts of image-caption data for pre-training . existing models require expensive data collection and curation . |
| Approach: | They propose to conduct "mask-and-predict" pre-training on text-only and image-only corpora and introduce the object tags detected by an object recognition model as anchor points to bridge two modalities. |
| Outcome: | The proposed approach achieves performance close to a model pre-trained with aligned data, on four English benchmarks. |
Visually-Aware Context Modeling for News Image Captioning (2024.naacl-long)
Copied to clipboard
| Challenge: | a new framework for News Image Captioning emphasizes the connection between textual context and visual elements. |
| Approach: | They propose a face-naming module for learning better name embeddings from news images . they use CLIP to retrieve sentences that are semantically close to the image . |
| Outcome: | The proposed framework outperforms the current state-of-the-art by 7.97/5.80 CIDEr scores on GoodNews/NYTimes800k. |
Learning from Children: Improving Image-Caption Pretraining via Curriculum (2023.findings-acl)
Copied to clipboard
| Challenge: | Image-caption pretraining is a difficult problem as it requires multiple concepts (nouns) from captions to be aligned to multiple objects in images. |
| Approach: | They propose a curriculum learning framework that uses images to align multiple concepts to multiple objects in an image. |
| Outcome: | The proposed learning framework improves over pretraining from scratch, using a pretrained image or/and text encoder, low data regime etc. |
OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to pretrain multilingual models are limited by the number of embedding parameters and the complexity of the model. |
| Approach: | They propose a framework that initializes the embeddings of unseen subwords and can adapt a model to multiple languages. |
| Outcome: | The proposed framework can adapt a pre-trained model to multiple languages efficiently and effectively. |
Vision-Language Pretraining: Current Trends and the Future (2022.acl-tutorials)
Copied to clipboard
| Challenge: | Recent vision-language models are being used for downstream tasks that require large datasets and supervised datasets. |
| Approach: | They focus on recent vision-language pretraining paradigms and their strengths and shortcomings . they compare the different family of models used for vision- language pretraining . |
| Outcome: | This paper provides the background on image–language datasets, benchmarks, and modeling innovations before the multimodal pretraining area. |
Does Vision-and-Language Pretraining Improve Lexical Grounding? (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large pretrained language models (LMs) have been criticized for lack of grounding, i.e., connecting words to their meanings in the physical world. |
| Approach: | They compare vision-and-language (VL) models trained jointly on text and image or video data to find out how they compare to text-only counterparts. |
| Outcome: | The proposed model outperforms the text-only variants on a commonsense question answering task. |
Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages (2023.acl-short)
Copied to clipboard
| Challenge: | Existing studies have shown that the pre-training in English does not transfer well to other languages in a zero-shot setting. |
| Approach: | They propose a simple yet efficient approach to adapt VLP to unseen languages using MPLM. |
| Outcome: | The proposed approach outperforms state-of-the-art models without large parallel corpora across three tasks. |
Retrieval-augmented Image Captioning (2023.eacl-main)
Copied to clipboard
| Challenge: | Experimental results show that image captioning can be effectively formulated from this new perspective. |
| Approach: | They propose a pretrained visual and language decoders for image captioning that generate sentences from the input image and a set of captions retrieved from a datastore. |
| Outcome: | The proposed model generates sentences given the input image and retrieved captions, while the decoder attends to the multimodal encoder representations. |