| Challenge: | Using crowd-sourced annotations, we generate text descriptions of differences between two images . we use a dataset to generate concise and fluent descriptions of visual data . |
| Approach: | They propose a task of automatically generating text to describe the differences between two images . they crowd-sourced the difference descriptions for pairs of images extracted from video-surveillance footage . |
| Outcome: | The proposed model outperforms models that use attention alone for single-sentence generation and multi-sentent generation. |
Similar Papers
L2C: Describing Visual Differences Needs Semantic Understanding of Individuals (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for captioning images without understanding individual's semantics are not effective . a new task, visual comparison, has drawn increasing attention in the field of language and vision . |
| Approach: | They propose a learning-to-compare model which learns to understand semantic structures of two images and compares them while learning to describe each one. |
| Outcome: | The proposed model outperforms the baseline and human evaluation on the Birds-to-Words dataset. |
Neural Naturalist: Generating Fine-Grained Image Comparisons (D19-1)
Copied to clipboard
| Challenge: | a dataset of 41k sentences describes fine-grained differences between photographs of birds . human observers are adept at making fine-grain comparisons, but sometimes require aid in distinguishing visually similar classes. |
| Approach: | They propose a model that generates comparative language from a dataset of 41k sentences describing fine-grained differences between photographs of birds. |
| Outcome: | The proposed model can explain differences in visual embedding space using natural language . it evaluates the results with humans who must use the descriptions to distinguish real images . |
A Probabilistic Model for Joint Learning of Word Embeddings from Texts and Images (D18-1)
Copied to clipboard
| Challenge: | Existing approaches combine language and perception to infer word embeddings . however, the embeddables produced by such models do not reflect the actual word representations. |
| Approach: | They propose a probabilistic model that integrates linguistic and perceptual inputs to explain observed word-context pairs in a text corpus. |
| Outcome: | The proposed model achieves competitive or stronger results on tasks of assessing pairwise word similarity and image/caption retrieval compared to other state-of-the-art models. |
CLIP4IDC: CLIP for Image Difference Captioning (2022.aacl-short)
Copied to clipboard
| Challenge: | Conventional approaches learn an IDC model with a pre-trained and usually frozen visual feature extractor. |
| Approach: | They propose to transfer a CLIP model to the downstream IDC task to address two major issues: (1) a large domain gap exists between the pre-training datasets used for training such a visual feature extractor; (2) the visual feature extraction often does not effectively encode the visual changes between two images. |
| Outcome: | Experiments on three IDC benchmark datasets show the proposed model performs well. |
Contrastive Learning with Narrative Twins for Modeling Story Salience (2026.eacl-long)
Copied to clipboard
| Challenge: | Understanding narratives requires identifying which events are most salient for a story’s progression. |
| Approach: | They propose a contrastive learning framework that learns story embeddings from narrative twins to distinguish a story from its distractor with similar surface features but different plot. |
| Outcome: | The proposed model outperforms a masked-language-model and summarizes sentences with the most reliable operation for identifying salient sentences. |
Uncovering Limitations in Text-to-Image Generation: A Contrastive Approach with Structured Semantic Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a new method for text-to-image generation models is proposed to address these limitations . SSA focuses on learning structured semantic embeddings across different modalities . |
| Approach: | They propose a method to evaluate text-to-image generation models using structured semantic embeddings . they propose to learn mutated prompts by substituting words with equivalent or nonequivalent alternatives . |
| Outcome: | The proposed method improves the measurement of semantic consistency of text-to-image generation models. |
Feature Difference Makes Sense: A medical image captioning model exploiting feature difference and tag information (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing methods for medical image captioning are limited and lack diversity . current methods do not generalize well when applied to unfamiliar images . |
| Approach: | They propose a feature difference and tag information combined long short-term memory model for chest x-ray report generation. |
| Outcome: | The proposed model outperforms existing models in chest x-ray report generation. |
Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have highlighted severe limitations of contrastive learning models in their ability to perform compositional reasoning over objects, attributes, and relations. |
| Approach: | They propose a graph decomposition framework and negative mining techniques to improve attribute binding and relation understanding of scene graphs. |
| Outcome: | The proposed approach improves attribute binding, relation understanding, generalization, and productivity on multiple benchmarks. |
Semantic Relation-aware Difference Representation Learning for Change Captioning (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to describe semantic change in images with distractors are difficult to learn . |
| Approach: | They propose a semantic relation-aware difference representation learning network to explicitly learn the difference representation in the existence of distractors. |
| Outcome: | The proposed network achieves state-of-the-art performance on CLEVR-Change and Spot-the -Diff datasets. |
Precision or Recall? An Analysis of Image Captions for Training Text-to-Image Generation Model (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in text-to-image models have demonstrated remarkable capabilities in image synthesis. |
| Approach: | They analyze the critical role of caption precision and recall in text-to-image model training. |
| Outcome: | The proposed model trains with synthetic captions that show similar behavior to those trained on human-annotated captions. |