Removing Word-Level Spurious Alignment between Images and Pseudo-Captions in Unsupervised Image Captioning (2021.eacl-main)
Copied to clipboard
| Challenge: | Unsupervised image captioning is a challenging task that requires manual annotation. |
| Approach: | They propose a simple gating mechanism that is trained to align image features with the most reliable words in pseudo-captions. |
| Outcome: | The proposed method outperforms the previous methods without complex learning objectives. |
Similar Papers
Defoiling Foiled Image Captions (N18-2)
Copied to clipboard
| Challenge: | Existing models for vision-to-language tasks do not understand images sufficiently, despite their impressive performance. |
| Approach: | They propose to use explicit object information to solve foiled caption detection problem . they propose to replace a word in caption with a semantically similar word . |
| Outcome: | The proposed model achieves state-of-the-art on a recently published dataset with scores exceeding those achieved by humans on the task. |
Exploiting Pseudo Image Captions for Multimodal Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to multimodal summarization with multimodal output (MSMO) lack reference images for training, and exposure of image captions during training is inconsistent with MSMO’s task settings. |
| Approach: | They propose a coarse-to-fine image-text alignment mechanism to identify the most relevant sentence of each image in a document, resembling the role of image captions in capturing visual knowledge. |
| Outcome: | The proposed method sets up state-of-the-art on all intermodality and intramodality metrics and improves on image recommendation precision. |
Altogether: Image Captioning via Re-aligning Alt-text (2024.emnlp-main)
Copied to clipboard
Hu Xu, Po-Yao Huang, Xiaoqing Tan, Ching-Feng Yeh, Jacob Kahn, Christine Jou, Gargi Ghosh, Omer Levy, Luke Zettlemoyer, Wen-tau Yih, Shang-Wen Li, Saining Xie, Christoph Feichtenhofer
| Challenge: | Existing captioning models ignore existing alt-text metadata and lack transparency if training data is unknown. |
| Approach: | They propose an approach to edit and re-align alt-texts associated with images using human annotation. |
| Outcome: | The proposed approach improves image captions and improves text-to-image generation and zero-shot image classification tasks. |
Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach (D19-1)
Copied to clipboard
| Challenge: | Recent work on image captioning has made impressive progress . however, the results are limited and the model is difficult to train . |
| Approach: | They propose a semi-supervised framework for training an image captioning model by assigning pseudo-labels to unpaired samples via Generative Adversarial Networks. |
| Outcome: | The proposed framework is compared to baselines when the number of paired samples is scarce. |
Learning from Children: Improving Image-Caption Pretraining via Curriculum (2023.findings-acl)
Copied to clipboard
| Challenge: | Image-caption pretraining is a difficult problem as it requires multiple concepts (nouns) from captions to be aligned to multiple objects in images. |
| Approach: | They propose a curriculum learning framework that uses images to align multiple concepts to multiple objects in an image. |
| Outcome: | The proposed learning framework improves over pretraining from scratch, using a pretrained image or/and text encoder, low data regime etc. |
Cross-Lingual Representation Alignment Through Contrastive Image-Caption Tuning (2025.acl-short)
Copied to clipboard
| Challenge: | Multilingual alignment of sentence representations has mostly required bitexts to bridge the gap between languages. |
| Approach: | They propose to use image captions to implicitly align text representations between languages to make them usable for cross-lingual Natural Language Understanding (NLU) and bitext retrieval. |
| Outcome: | The proposed approach is usable for cross-lingual Natural Language Understanding (NLU) and bitext retrieval. |
Aligners: Decoupling LLMs and Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) need to be aligned with human expectations to ensure their safety and utility in most applications. |
| Approach: | They propose to decouple LLMs and alignment by training *aligner* models that can be used to align any LLM on an as-needed basis. |
| Outcome: | The proposed model can be used to align any LLM for a given criteria on an as-needed basis. |
On the Word Alignment from Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Prior researches suggest that neural machine translation (NMT) captures word alignment through its attention mechanism, however, attention may fail to capture word alignment for some NMT models. |
| Approach: | They propose two methods to induce word alignment which are general and agnostic to specific NMT models. |
| Outcome: | The proposed methods induce much better word alignment than attention. |
When do Contrastive Word Alignments Improve Many-to-many Neural Machine Translation? (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to improve pre-training for many-to-many neural machine translation use manual cleaning of bilingual dictionaries, which are unavailable for most language pairs. |
| Approach: | They propose a word-level contrastive objective to leverage word alignments for many-to-many neural machine translation (NMT) Empirical results show that this leads to 0.8 BLEU gains for several language pairs. |
| Outcome: | Empirical results show that the proposed objective leads to 0.8 BLEU gains for several language pairs. |
Constraining word alignments with posterior regularization for label transfer (2022.naacl-industry)
Copied to clipboard
| Challenge: | Unsupervised word alignments are not always possible in industrial NLP pipelines, where multilingual annotation guidelines are complex and deviate from semantic consistency due to various factors. |
| Approach: | They propose to constrain word alignment models to remain consistent with both source and target annotation guidelines by leveraging posterior regularization and labeled examples. |
| Outcome: | The proposed model improves on the multiATIS++ dataset over AWESoME, and even a small amount of target language annotations can help. |