Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for vision-language models treat compositionality and long-caption understanding in isolation. |
| Approach: | They analyze when compositional reasoning and long-caption understanding transfer across tasks and when this relationship fails. |
| Outcome: | The proposed model can generalize on poorly grounded captions and with strong visual grounding, while architectural choices can limit compositional learning. |
Similar Papers
Combine to Describe: Evaluating Compositional Generalization in Image Captioning (2022.acl-srw)
Copied to clipboard
| Challenge: | Recent work on compositionality has focused on the ability to combine simpler concepts to understand & generate arbitrarily more complex conceptual structures. |
| Approach: | They propose to use a set of image captioning models to benchmark their compositional generalization properties. |
| Outcome: | The proposed models do not generalize in terms of systematicity and productivity, but are robust to synonym substitutions. |
Text encoders bottleneck compositionality in contrastive vision-language models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal models are often unable to reason about simple spatial relations or attribute attachments. |
| Approach: | They first curate CompPrompts, a set of increasingly compositional image captions that VL models should be able to capture . then train text-only recovery probes that aim to reconstruct captions from single-vector text representations produced by several VL model. |
| Outcome: | The proposed model can reconstruct captions from single-vector text representations produced by several models on a broader range of scenes compared to previous models. |
Can VLMs Actually See and Read? A Survey on Modality Collapse in Vision-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models integrate textual and visual information, enabling them to process visual inputs and generate predictions. |
| Approach: | They review work on modality collapse analysis to provide insights into the reason for this unintended behavior and review probing studies for fine-grained vision-language understanding. |
| Outcome: | The proposed models can achieve competitive performance in vision-language tasks despite relying heavily on textual information and ignoring visual information. |
Quantifying Compositionality of Classic and State-of-the-Art Embeddings (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Static word embeddings make strong claims about compositionality, but the SOTA generative models go too far in the other direction. |
| Approach: | a new study evaluates the compositionality of word embeddings by canonical correlation analysis . strong compositional signals are observed in later training stages across data modalities . |
| Outcome: | a new evaluation of compositional models shows that they exploit access meanings when justified . strong compositional signals are observed in later training stages and in deeper layers of the transformer-based model before a decline at the top layer. |
Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent visuolinguistic pre-trained models fail miserably on the Winoground dataset, which challenges models to match paired images and English captions. |
| Approach: | They propose to annotate a Winoground dataset that challenges visuolinguistic models to match paired images and English captions with items constructed to overlap lexically but differ in meaning. |
| Outcome: | The proposed dataset challenges models to match paired images and English captions with items constructed to overlap lexically but differ in meaning. |
Interpretable Composition Attribution Enhancement for Visio-linguistic Compositional Understanding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Despite promising progress, vision-language models still exhibit significant challenges in understanding visio-linguistic concepts beyond object terms. |
| Approach: | They propose a framework that encourages the model to pay greater attention to composition words denoting relationships and attributes within the text. |
| Outcome: | The proposed framework improves the ability to discern intricate details and construct more sophisticated interpretations of combined visual and linguistic elements. |
Understanding Subword Compositionality of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) take sequences of subwords as input, requiring them to compose subword representations into meaningful word-level representations. |
| Approach: | They propose to probe how large language models compose subword information . they find structural similarity, semantic decomposability, and form retention are key aspects . |
| Outcome: | The proposed models can be classified into three distinct groups, the authors show . they show that they can achieve great performance when probing layer by layer their sensitivity to semantic decompositionality . |
Compositional Networks Enable Systematic Generalization for Grounded Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that deep networks can mimic some human language abilities when presented with novel sentences . a general-purpose mechanism that enables agents to generalize their language understanding to compositional domains is critical to building safe and fair robots, says a new study. |
| Approach: | They build a general-purpose mechanism that enables agents to generalize their language understanding to compositional domains. |
| Outcome: | a new network generalizes its language understanding to compositional domains while generalizing its knowledge when prior work does not. |
The Role of Syntactic Planning in Compositional Image Captioning (2021.eacl-main)
Copied to clipboard
| Challenge: | Image captioning is a core task in multimodal NLP, where the aim is to automatically describe the content of an image in natural language. |
| Approach: | They propose to use syntactic tags and tokens to improve caption generalization . they also propose to model the syntakic structure of a caption to improve generalization. |
| Outcome: | The proposed models improve generalization and performance on standard metrics while requiring syntactic and semantic knowledge of the language. |
Compositional Generalization by Factorizing Alignment and Translation (2020.acl-srw)
Copied to clipboard
| Challenge: | a crucial property underlying the expressive power of human language is its systematicity. |
| Approach: | They propose to make an analogous separation between alignment and translation in neural machine translation to capture compositional structure. |
| Outcome: | The proposed architecture outperforms existing neural networks on a compositional generalization task without supervision. |