| Challenge: | Figure 1 shows representative examples of visual artifacts introduced by diffusion-based inpainting . despite visually plausible reconstructions, localized inpainding artifactors lead to object substitutions, attribute changes, or category-level errors in downstream captions. |
| Approach: | They propose a diagnostic setup in which masked image regions are reconstructed and then provided to captioning models. |
| Outcome: | The proposed diagnostic framework can be used to examine how visual artifacts affect language generation in vision-language models. |
Similar Papers
Seeing Through Words, Speaking Through Pixels: Deep Representational Alignment Between Vision and Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that deep vision-only and language-only models project inputs into a partially aligned representational space. |
| Approach: | They investigate whether a model's representational code is semantically shared . they find that alignment peaks in mid-to-late layers of both model types . |
| Outcome: | a forced-choice "Pick-a-Pic" task shows human preferences for image-caption matches are mirrored in embedding spaces across vision-language model pairs. |
Unraveling the Mystery of Artifacts in Machine Generated Text (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent studies show that human-written text is not distinguishable from synthetic text because of semantic errors or logical contradictions. |
| Approach: | They propose to analyze the forms of artifacts left by neural Text Generation Models by corrupting texts and replacing them with linguistic or statistical features. |
| Outcome: | The proposed method replaces text with linguistic or statistical features and improves the accuracy of the model. |
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Combining visual modality with pretrained language models has been effective for descriptive tasks such as image captioning. |
| Approach: | They ask: do multimodal models combine visual and visual adapted language models? they find that CLIP image representations and scaling of language models do not consistently improve self-rationalization in multimodal tasks. |
| Outcome: | The proposed model types do not consistently improve self-rationalization in multimodal tasks. |
What the DAAM: Interpreting Stable Diffusion Using Cross Attention (2023.acl-long)
Copied to clipboard
Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, Ferhan Ture
| Challenge: | a new text-image attribution analysis model for text-to-image generation is understudied due to ethical constraints . corporators have restricted the general public from using the models and their weights . |
| Approach: | They perform a text-image attribution analysis on Stable Diffusion, a recently open-sourced model. |
| Outcome: | The proposed method achieves a competitive 58.8-64.8 mIoU on noun segmentation and fair to good mean opinion scores on all parts of speech rated by humans . it also achieves good attribution quality on all part of speech, rated in humans - and the first to interpret large diffusion models from a visuolinguistic perspective. |
Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing vision-and-language models perform better on multimodal tasks, but there is little understanding of how multimodal learning can help visual representations. |
| Approach: | They conduct a probing analysis of visual representations in existing vision-and-language models and vision-only models by probing on a broad range of tasks. |
| Outcome: | The proposed model improves vision-and-language models on label and attribute prediction tasks while vision-only models are stronger on dense prediction tasks. |
Precision or Recall? An Analysis of Image Captions for Training Text-to-Image Generation Model (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in text-to-image models have demonstrated remarkable capabilities in image synthesis. |
| Approach: | They analyze the critical role of caption precision and recall in text-to-image model training. |
| Outcome: | The proposed model trains with synthetic captions that show similar behavior to those trained on human-annotated captions. |
Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations (2020.emnlp-main)
Copied to clipboard
| Challenge: | A major challenge in visually grounded language generation is to build robust benchmark datasets and models that can generalize well in real-world settings. |
| Approach: | They propose to use visual attention to build robust benchmark datasets and models that can generalize well in real-world settings. |
| Outcome: | The proposed models show that human-generated references vary drastically in different datasets/tasks, revealing the nature of each task. |
Overestimation of Syntactic Representation in Neural Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Several testing methodologies have been developed to probe models’ syntactic representations. |
| Approach: | They propose a method to determine syntactic structure by training a model on strings generated according to a template and testing its ability to distinguish between similar ones with different syntax. |
| Outcome: | The proposed method reproduces positive results with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs. |
Probing Image-Language Transformers for Verb Understanding (2021.findings-acl)
Copied to clipboard
| Challenge: | Multimodal image-language transformers have achieved impressive results on a variety of tasks that rely on fine-tuning. |
| Approach: | They collect a dataset of image-sentence pairs consisting of 421 verbs . they evaluate pretrained image-language transformers and find they fail more in situations that require verb understanding compared to other parts of speech. |
| Outcome: | The proposed model trains on a manually-annotated and smaller dataset does better on the task. |
Probing Contextual Language Models for Common Ground with Visual Representations (2021.naacl-main)
Copied to clipboard
| Challenge: | Contextual language models have attracted great interest in probing what is encoded in their representations. |
| Approach: | They propose a probing model that evaluates how effective are text-only representations in distinguishing between matching and non-matching visual representations. |
| Outcome: | The proposed model outperforms text-only language models in instance retrieval, but underperform humans. |