| Challenge: | Existing approaches to visual grounding do not explicitly model compositional structures of text expressions. |
| Approach: | They propose a concept-relation Graph and a composition neural network to combine CRGs . they propose to align CRG-based concepts with images to learn visually grounded concepts . |
| Outcome: | The proposed model can model grounded concepts forming at sentence level and word level. |
Similar Papers
Visually Grounded Compound PCFGs (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on visual groundings for language understanding has been drawing much attention. |
| Approach: | They propose to use an extension of probabilistic context-free grammar model to do fully-differentiable end-to-end visually grounded learning. |
| Outcome: | The proposed model outperforms the previous grounded model and significantly outperformed the previous model on the MSCOCO test captions. |
MetaReVision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to learn compositional concepts from previous experience are based on systematic generalization, productivity and substitutivity. |
| Approach: | They propose a retrieval-enhanced meta-learning model to solve the visually grounded compositional concept learning problem by meta-training retrieved primitive concepts from episodes constructed by the retriever. |
| Outcome: | The proposed model outperforms other baselines and the retrieval module plays an important role in this compositional learning process. |
COVR: A Test-Bed for Visually Grounded Compositional Generalization with Real Images (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for visual-grounded models have focused on synthetic images . et al., 2018: compositional generalization is crucial for building models that generalize to new settings. |
| Approach: | They propose a test-bed for visually-grounded compositional generalization with real images. |
| Outcome: | The proposed test-bed enables compositional splits where models need to generalize to new concepts and compositions in a zero- or few-shot setting. |
Compositional Generalization with Grounded Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for combining language models with knowledge graphs struggle with generalization to sequences of unseen lengths and novel combinations of seen base components. |
| Approach: | They propose a procedure for generating natural language questions paired with knowledge graphs that targets different aspects of compositionality and avoids grounding models in information already encoded in their weights. |
| Outcome: | The proposed method fails to generalize to unseen lengths and to novel combinations of seen base components. |
Do Neural Language Models Inferentially Compose Concepts the Way Humans Can? (2024.lrec-main)
Copied to clipboard
| Challenge: | a new study shows that language models and humans may rely on different approaches to represent and compose lexical items across sentence structure. |
| Approach: | They propose to use a dataset to test the performance of neural language models and humans on inferentially driven conceptual compositions. |
| Outcome: | The proposed model elicits probability estimates for a noun in a minimally composed phrase . RoBERTa, BERT-large, and GPT-2 exhibited the closest resemblance to human responses . |
What Does BERT with Vision Look At? (2020.acl-main)
Copied to clipboard
| Challenge: | Pre-trained visual grounded language models have improved performance on vision-and-language tasks but what they learn during pre-training remains unclear. |
| Approach: | They show that attention heads of visual grounded language models actively ground elements of language to image regions. |
| Outcome: | The attention heads of a visual grounded language model can ground elements to image regions, demonstrating their ability to detect syntactic relations between non-entity words and image regions. |
Learning to Represent Image and Text with Denotation Graph (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in learning representations of visual and language information have been a problem with many applications. |
| Approach: | They propose to extract visual expressions from images aligned with linguistic expressions that describe the images to learn representations from implicit expressions. |
| Outcome: | The proposed representations lead to stronger empirical results on downstream tasks of cross-modal image retrieval, referring expression, and compositional attribute-object recognition. |
Incorporating Visual Semantics into Sentence Representations within a Grounded Space (D19-1)
Copied to clipboard
| Challenge: | Language grounding is an active field aiming at enriching textual representations with visual information. |
| Approach: | They propose to transfer visual information to textual representations by learning an intermediate representation space: the grounded space. |
| Outcome: | The proposed model outperforms the previous state-of-the-art on classification and semantic relatedness tasks. |
Read Before Grounding: Scene Knowledge Visual Grounding via Multi-step Parsing (2025.coling-main)
Copied to clipboard
| Challenge: | Existing VG datasets use simple textual descriptions with limited attribute and spatial information between images and text. |
| Approach: | They propose a method that transforms visual knowledge into concise, information-dense visual descriptions. |
| Outcome: | The proposed method significantly improves performance of multimodal grounding models. |
Compositional Networks Enable Systematic Generalization for Grounded Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that deep networks can mimic some human language abilities when presented with novel sentences . a general-purpose mechanism that enables agents to generalize their language understanding to compositional domains is critical to building safe and fair robots, says a new study. |
| Approach: | They build a general-purpose mechanism that enables agents to generalize their language understanding to compositional domains. |
| Outcome: | a new network generalizes its language understanding to compositional domains while generalizing its knowledge when prior work does not. |