Incorporating Visual Semantics into Sentence Representations within a Grounded Space (D19-1)
Copied to clipboard
| Challenge: | Language grounding is an active field aiming at enriching textual representations with visual information. |
| Approach: | They propose to transfer visual information to textual representations by learning an intermediate representation space: the grounded space. |
| Outcome: | The proposed model outperforms the previous state-of-the-art on classification and semantic relatedness tasks. |
Similar Papers
Learning Visually Grounded Sentence Representations (N18-1)
Copied to clipboard
| Challenge: | Unsupervised sentence representation models suffer from the grounding problem because of lack of association between symbols and external information. |
| Approach: | They train a sentence encoder to predict image features of a caption and use them as sentence representations. |
| Outcome: | The proposed model improves on word embeddings and word representations on standard benchmarks. |
Visual Grounding Helps Learn Word Meanings in Low-Data Regimes (2024.naacl-long)
Copied to clipboard
| Challenge: | Modern neural language models (LMs) require distinctly un-human-like ways to achieve these results. |
| Approach: | They train a diverse set of LM architectures with and without auxiliary visual supervision on datasets of varying scales. |
| Outcome: | The proposed models exhibit better learning of syntactic categories, lexical relations, semantic features, word similarity and alignment with human neural representations. |
Grounding Meaning Representation for Situated Reasoning (2022.aacl-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to build agents that understand language using a simulated environment . situated reasoning is a critical aspect of human language understanding . |
| Approach: | This tutorial combines a synthesis of multimodal grounding and meaning representation techniques with formal and computational models of situated reasoning. |
| Outcome: | This tutorial combines multimodal grounding and meaning representation techniques with formal and computational models of embodied reasoning. |
Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Neural language models (LMs) are trained on orders of magnitude more language data than human language learners receive, but without supervision from other sensory modalities that play a crucial role in human learning. |
| Approach: | They propose a grounded language learning procedure that leverages visual supervision to improve textual representations. |
| Outcome: | The proposed procedure outperforms standard language-only models in terms of learning efficiency in small and developmentally plausible data regimes and improves perplexity by around 5% on multiple language modeling tasks compared to other models trained on the same amount of text data. |
Learning Language through Grounding (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | This tutorial provides a historical overview of grounding and discusses its use in computational linguistics and in computational language processing. |
| Approach: | They introduce the concept of grounding and discuss future directions and open challenges . they will delve into recent progress in learning lexical semantics, syntax, and complex meanings through various forms of ground. |
| Outcome: | This course will provide an overview of the field of grounding and discuss future directions and challenges related to large language models and scaling. |
Language in a (Search) Box: Grounding Language Learning in Real-World Human-Machine Interaction (2021.naacl-main)
Copied to clipboard
| Challenge: | Scholarly work in this area uses toy worlds and synthetic linguistic data, but grounded language learning offers several practical and scientific advantages. |
| Approach: | They propose to model teacher-learner dynamics through natural interactions occurring between users and search engines. |
| Outcome: | The proposed model is better than non-grounded models on compositionality and zero-shot inference tasks. |
Learning grounded word meaning representations on similarity graphs (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn semantic representations from text are limited to words in context and words in isolation. |
| Approach: | They propose a method to learn meaning representations of words as low-dimensional node embeddings on an underlying graph hierarchy. |
| Outcome: | The proposed model outperforms the state-of-the-art in terms of similarity judgments and concept categorization. |
Attention as Grounding: Exploring Textual and Cross-Modal Attention on Entities and Relations in Language-and-Vision Transformer (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing work has focused on what is captured by multi-modal architectures. |
| Approach: | They propose a multi-modal transformer that learns syntactic and semantic representations about entities and relations grounded in objects at the level of masked self-attention and cross-modal attention. |
| Outcome: | The proposed model learns syntactic and semantic representations about objects and relations cross-modally and unimodally. |
Grounding language acquisition by training semantic parsers using captioned videos (D18-1)
Copied to clipboard
| Challenge: | a new method for parsing sentences using captioned videos is being developed . we use video clips to ground the semantics of language, but without annotations . |
| Approach: | They develop a semantic parser that is trained in a grounded setting using captioned videos . they use a corpus of sentences paired with videos without other annotations to train it . |
| Outcome: | The proposed parser recovers the meaning of English sentences despite no annotations . learning a grounded semantic parsers can expand the range of data that parseurs can be trained on . |
Improving Pre-trained Vision-and-Language Embeddings for Phrase Grounding (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the phrase grounding ability of pretrained vision-and-language models, but it is unclear how they can be used for phrase ground. |
| Approach: | They propose to extract phrase-region pairs from pre-trained vision-and-language embeddings and propose four fine-tuning objectives to improve model phrase grounding ability using image-caption data without any supervised grounding signals. |
| Outcome: | The proposed model outperforms baseline models in weakly-supervised and supervised phrase grounding settings on two representative datasets and shows that it is possible to achieve better phrase groundability without sacrificing representation generality. |