| Challenge: | Existing methods for classifying images without supervision are limited. |
| Approach: | They propose a top-down attention mechanism to align entities in captions to objects in the image and leverage the syntactic structure of captions for alignment. |
| Outcome: | The proposed model achieves a recall@50 of 15% and recall@100 of 25% on the relationships present in the image and predicts relations that are not present in captions. |
Similar Papers
Improving Image Captioning with Better Use of Caption (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to image captioning focus on visual attention, but many do not. |
| Approach: | They propose a framework that explores semantics available in captions and leverages that to enhance both image representation and caption generation. |
| Outcome: | The proposed framework outperforms baselines on the MSCOCO dataset and is state-of-the-art under a wide range of evaluation metrics. |
Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work in vision-and-language pretraining has investigated supervised signals from object detection data to learn better, fine-grained multimodal representations. |
| Approach: | They propose two approaches to contextualise visual entities in a multimodal setup by using verbalised scene graphs and masked relation prediction. |
| Outcome: | The proposed models can learn better representations from weakly-supervised relations data. |
Improving Image Captioning via Predicting Structured Concepts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on image captioning ignore the relationship between concepts . current methods for image caption generation ignore this relationship . |
| Approach: | They propose a structured concept predictor to predict concepts and their structures . they integrate these predictions into captioning to enhance visual signals . |
| Outcome: | The proposed approach improves image captioning performance by using semantic concepts as a bridge between images and texts. |
Cross-modal Coherence Modeling for Caption Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for image captioning do not guarantee consistent image-text relations . current models do not provide enough data for training robust captioning models . |
| Approach: | They use an annotation protocol specifically devised for capturing image–caption coherence relations to study image captioning. |
| Outcome: | The proposed protocol improves image captioning models with coherence relations . the dataset is large enough to alleviate content hallucinations, the authors show . |
Improving Entity Linking by Modeling Latent Relations between Mentions (P18-1)
Copied to clipboard
| Challenge: | Entity linking systems often exploit relations between textual mentions to decide if the linking decisions are compatible. |
| Approach: | They treat relations as latent variables while optimizing the neural entity-linking model without supervision. |
| Outcome: | The proposed model outperforms its relation-agnostic version and significantly outperformed its relational version. |
Improving Distantly-Supervised Relation Extraction with Joint Label Embedding (D19-1)
Copied to clipboard
| Challenge: | Existing methods for relation extraction treat labels as independent and meaningless one-hot vectors, which cause a loss of potential label information for selecting valid instances. |
| Approach: | They propose a multi-layer attention-based model to improve relation extraction with joint label embedding by gating integration and using the embeddable entities as an atten- tion. |
| Outcome: | The proposed model significantly outperforms state-of-the-art methods in relation extraction with joint label embedding. |
Combining Distant and Direct Supervision for Neural Relation Extraction (N19-1)
Copied to clipboard
| Challenge: | Existing methods to train relation extraction with distant supervision use noisy labels and implicitly assumes that all the KB facts are mentioned in the text. |
| Approach: | They propose to combine distant supervision data with additional directly-supervised data to train relation extraction models by using sigmoidal attention weights with max pooling. |
| Outcome: | The proposed method achieves state-of-the-art on the widely used FB-NYT dataset. |
Bridging by Word: Image Grounded Vocabulary Construction for Visual Captioning (P19-1)
Copied to clipboard
| Challenge: | Existing research on image captioning generates frequent n-grams with irrelevant words. |
| Approach: | They propose to construct an image-grounded vocabulary incorporating visual information and relations among words into the decoding process directly. |
| Outcome: | The proposed framework is compared with state-of-the-art models on MS COCO and Flickr30k and shows that it is more efficient than existing models. |
Contrastive Learning with Expectation-Maximization for Weakly Supervised Phrase Grounding (2022.emnlp-main)
Copied to clipboard
| Challenge: | Weakly supervised phrase grounding aims to learn an alignment between phrases in a caption and objects in an image using only caption-image annotations. |
| Approach: | They propose a novel contrastive learning framework that adaptively refines the target prediction by using only caption-image annotations. |
| Outcome: | The proposed framework outperforms existing methods on two widely used benchmarks, Flickr30K Entities and RefCOCO+. |
R-Bind: Unified Enhancement of Attribute and Relation Binding in Text-to-Image Diffusion Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve semantic binding require costly retraining or focus on only correctly generating attributes of entities. Existing methods focus on correctly generating attributes, ignoring the cruciality of correctly forming relations between entities. |
| Approach: | They propose a training-free method that improves both entity-attribute and entity-relation-entity binding by introducing three inference-time optimization losses that adjust attention maps during generation. |
| Outcome: | The proposed method improves both entity-attribute and entity-relation-entity binding without additional training. |