| Challenge: | Current approaches to model or simulate the acquisition of spoken language via grounding in perception are not generalizable to real-life situations that humans or adaptive artificial agents experience. |
| Approach: | They propose to use a dataset based on the children’s cartoon Peppa Pig to train a bi-modal architecture that learns aspects of the visual semantics of spoken language. |
| Outcome: | The proposed model learns to represent speech and visual data in a joint vector space. |
Similar Papers
Visual Grounding Helps Learn Word Meanings in Low-Data Regimes (2024.naacl-long)
Copied to clipboard
| Challenge: | Modern neural language models (LMs) require distinctly un-human-like ways to achieve these results. |
| Approach: | They train a diverse set of LM architectures with and without auxiliary visual supervision on datasets of varying scales. |
| Outcome: | The proposed models exhibit better learning of syntactic categories, lexical relations, semantic features, word similarity and alignment with human neural representations. |
Seeing Through Words, Speaking Through Pixels: Deep Representational Alignment Between Vision and Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that deep vision-only and language-only models project inputs into a partially aligned representational space. |
| Approach: | They investigate whether a model's representational code is semantically shared . they find that alignment peaks in mid-to-late layers of both model types . |
| Outcome: | a forced-choice "Pick-a-Pic" task shows human preferences for image-caption matches are mirrored in embedding spaces across vision-language model pairs. |
Weakly-Supervised Spoken Video Grounding via Semantic Interaction Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work on spoken video grounding challenges extracting semantic information from speech . previous studies focused on textual queries, but recent work focuses on spoken queries . |
| Approach: | They propose a framework for weakly-supervised spoken video grounding to represent cross-modal semantics without expensive temporal annotations. |
| Outcome: | The proposed framework is more efficient than existing methods. |
A Probabilistic Model for Joint Learning of Word Embeddings from Texts and Images (D18-1)
Copied to clipboard
| Challenge: | Existing approaches combine language and perception to infer word embeddings . however, the embeddables produced by such models do not reflect the actual word representations. |
| Approach: | They propose a probabilistic model that integrates linguistic and perceptual inputs to explain observed word-context pairs in a text corpus. |
| Outcome: | The proposed model achieves competitive or stronger results on tasks of assessing pairwise word similarity and image/caption retrieval compared to other state-of-the-art models. |
Like a Baby: Visually Situated Neural Language Acquisition (P19-1)
Copied to clipboard
| Challenge: | A multi-modal neural architecture outperforms its equivalent trained on language alone with a 2% decrease in perplexity . |
| Approach: | They propose to use visual context to train neural language models to perform next-word prediction. |
| Outcome: | The proposed model outperforms its equivalent trained on language with 2% decrease in perplexity even when no visual context is available at test. |
Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing language pretraining frameworks only take the language context as selfsupervision . current frameworks do not take grounding information from the external visual world . |
| Approach: | They propose a visually-supervised language model that extrapolates multimodal alignments to language-only data by contextually mapping language tokens to related images. |
| Outcome: | The proposed model improves on multiple pure-language tasks. |
Textual Supervision for Visually Grounded Spoken Language Understanding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | a new approach to spoken language understanding extracts semantic information directly from speech without relying on transcriptions. |
| Approach: | They propose to use textual supervision to train visually-grounded models of spoken language understanding without relying on transcriptions. |
| Outcome: | The proposed model improves when enough text is available, the study shows . compared with pipeline-based models, the pipeline approach performs better when enough data is available . |
Incorporating Visual Semantics into Sentence Representations within a Grounded Space (D19-1)
Copied to clipboard
| Challenge: | Language grounding is an active field aiming at enriching textual representations with visual information. |
| Approach: | They propose to transfer visual information to textual representations by learning an intermediate representation space: the grounded space. |
| Outcome: | The proposed model outperforms the previous state-of-the-art on classification and semantic relatedness tasks. |
Is Child-Directed Speech Effective Training Data for Language Models? (2024.emnlp-main)
Copied to clipboard
| Challenge: | High-performing language models are typically trained on hundreds of billions of words, but human learners use language fluently after far less training data. |
| Approach: | They train GPT-2 and RoBERTa models on 29M words of English child-directed speech and a new matched, synthetic dataset. |
| Outcome: | The proposed models show that child language input is not valuable for training language models. |
Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives (2024.findings-acl)
Copied to clipboard
Thong Nguyen, Yi Bin, Junbin Xiao, Leigang Qu, Yicong Li, Jay Zhangjie Wu, Cong-Duy Nguyen, See-Kiong Ng, Anh Tuan Luu
| Challenge: | Existing video-language understanding systems with human-like senses can mimic both our linguistic medium and visual environment with temporal dynamics. |
| Approach: | They propose to develop video-language understanding systems with human-like senses . they summarize their methods and highlight challenges associated with them . |
| Outcome: | The proposed models perform well in a variety of tasks and domains. |