Knowledge Supports Visual Language Grounding: A Case Study on Colour Terms (2020.acl-main)
Copied to clipboard
| Challenge: | In human cognition, world knowledge supports the perception of object colours . a lot of recent work in Language & Vision has looked at grounding language in real-world sensory information. |
| Approach: | They propose to integrate visual information and object-specific knowledge via hard-coded or learned fusion to improve visual grounding of colour terms in realistic objects. |
| Outcome: | The proposed models outperform a baseline model that predicts colour terms solely from visual inputs but show interesting differences when predicting atypical colours of so-called colour diagnostic objects. |
Similar Papers
Perceptual Structure in the absence of grounding: the impact of abstractedness and subjectivity in color language for LLMs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that color perception and color language are suitable for empirically studying the problem. |
| Approach: | They propose to quantify alignment between a defined color space and a feature space in a language model by learning a mapping between embedding space and color space. |
| Outcome: | The results show that there is considerable alignment between a defined color space and the feature space defined by a language model. |
Visual Grounding Helps Learn Word Meanings in Low-Data Regimes (2024.naacl-long)
Copied to clipboard
| Challenge: | Modern neural language models (LMs) require distinctly un-human-like ways to achieve these results. |
| Approach: | They train a diverse set of LM architectures with and without auxiliary visual supervision on datasets of varying scales. |
| Outcome: | The proposed models exhibit better learning of syntactic categories, lexical relations, semantic features, word similarity and alignment with human neural representations. |
Grounding learning of modifier dynamics: An application to color naming (D19-1)
Copied to clipboard
| Challenge: | Existing models for grounding are unable to understand modified color expressions, such as “light blue”. |
| Approach: | They propose a model that learns more complex transformations in RGB space and a hard ensemble model that selects a color space depending on the modifier-color pair. |
| Outcome: | The proposed model performs better in the HSV color space than the state-of-the-art model. |
‘Lighter’ Can Still Be Dark: Modeling Comparative Color Descriptions (P18-2)
Copied to clipboard
| Challenge: | Multimodal approaches to object recognition ground adjectives and nouns from text using comparative adjectives. |
| Approach: | They propose a new paradigm of grounding comparative adjectives within the realm of color descriptions by using a vector model. |
| Outcome: | The proposed model generates representations of comparative adjectives with an average accuracy of 0.65 cosine similarity to the desired direction of change. |
Language in a (Search) Box: Grounding Language Learning in Real-World Human-Machine Interaction (2021.naacl-main)
Copied to clipboard
| Challenge: | Scholarly work in this area uses toy worlds and synthetic linguistic data, but grounded language learning offers several practical and scientific advantages. |
| Approach: | They propose to model teacher-learner dynamics through natural interactions occurring between users and search engines. |
| Outcome: | The proposed model is better than non-grounded models on compositionality and zero-shot inference tasks. |
Find Someone Who: Visual Commonsense Understanding in Human-Centric Grounding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Visual scenes often involve multiple people and humans can distinguish between them based on context descriptions about what happened before, their mental/physical states, and intentions. |
| Approach: | They propose a task that tests human-centric commonsense grounding models' ability to distinguish individuals given context descriptions about what happened before and their mental/physical states or intentions. |
| Outcome: | The proposed model outperforms pre-trained and non-pretrained models on 130k commonsense descriptions annotated on 67k images. |
Exploration of the Usage of Color Terms by Color-blind Participants in Online Discussion Platforms (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on the role of sensory vs. linguistic input in the way we acquire and use language have not explored the relative effects of various factors on our linguistic system. |
| Approach: | They propose to use a newly curated dataset to examine the conceptual perception of colors by color-blind speakers. |
| Outcome: | The findings shed some new and interesting light on the role of sensory experience on our linguistic system. |
comp-syn: Perceptually Grounded Word Embeddings with Color (2020.coling-main)
Copied to clipboard
Bhargav Srinivasa Desikan, Tasker Hull, Ethan Nadler, Douglas Guilbeault, Aabir Abubakar Kar, Mark Chu, Donald Ruggiero Lo Sardo
| Challenge: | Existing approaches to natural language processing ignore embodied sensory aspects of language. |
| Approach: | They propose a Python package that provides word embeddings based on Google Image search results. |
| Outcome: | The proposed package provides word embeddings based on the color distributions of Google Image search results. |
Does Visual Grounding Enhance the Understanding of Embodied Knowledge in Large Language Models? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Despite significant progress in multimodal language models, it remains unclear whether visual grounding enhances their understanding of embodied knowledge compared to text-only models. |
| Approach: | They propose to assess vision-language models’ perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. |
| Outcome: | The proposed benchmark assesses the models’ perceptual abilities across different sensory modalities through vector comparison and question-answering tasks with over 1,700 questions. |
Learning Language through Grounding (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | This tutorial provides a historical overview of grounding and discusses its use in computational linguistics and in computational language processing. |
| Approach: | They introduce the concept of grounding and discuss future directions and open challenges . they will delve into recent progress in learning lexical semantics, syntax, and complex meanings through various forms of ground. |
| Outcome: | This course will provide an overview of the field of grounding and discuss future directions and challenges related to large language models and scaling. |