Quantifying the Visual Concreteness of Words and Topics in Multimodal Datasets (N18-1)
Copied to clipboard
| Challenge: | Existing work suggests that concepts with concrete visual manifestations are easier to learn than abstract ones. |
| Approach: | They propose an algorithm for automatically computing the visual concreteness of words and topics within multimodal datasets. |
| Outcome: | The proposed algorithm predicts the capacity of machine learning algorithms to learn textual/visual relationships. |
Similar Papers
ICC : Quantifying Image Caption Concreteness for Multimodal Dataset Curation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to curation text-image data are noisy and lack the fine-grained ability to isolate the most concrete samples that provide the strongest signal for learning in a noisy dataset. |
| Approach: | They propose a metric that evaluates caption text without an image reference to measure its concreteness and relevancy. |
| Outcome: | The proposed method detects the concreteness of captions without an image reference and correlates with human evaluation of concreteness in both single-word and caption-level texts. |
Grounded Concreteness: Human-Like Concreteness Sensitivity in Vision–Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | a long tradition in cognitive science treats concreteness as a graded dimension of conceptual representation . concrete words benefit from richer sensory codes and exhibit robust behavioral advantages over abstract words . |
| Approach: | They compare vision-language models with text-only large language models to test their concreteness . they find that VLMs show more human-like sensitivity to concreteness than LLMs . |
| Outcome: | The proposed model-based training improves on the Llama text backbones and Llma Vision counterparts. |
What do Models Learn From Training on More Than Text? Measuring Visual Commonsense Knowledge (2022.acl-srw)
Copied to clipboard
| Challenge: | Existing evaluation methods to measure what language models learn from multimodal training are lacking. |
| Approach: | They propose two evaluation tasks to measure commonsense knowledge in language models by using visual data to evaluate multimodal models and unimodal baselines. |
| Outcome: | The proposed evaluation tasks show that training on a visual modality improves on the visual commonsense knowledge in language models. |
MCSE: Multimodal Contrastive Learning of Sentence Embeddings (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to learning semantically meaningful sentence embeddings are limited by the complexity of pre-trained models. |
| Approach: | They propose a sentence embedding learning approach that exploits both visual and textual information via a multimodal contrastive objective. |
| Outcome: | The proposed approach improves the state-of-the-art average Spearman’s correlation by 1.7% on a variety of semantic textual similarity tasks. |
Unveiling the mystery of visual attributes of concrete and abstract concepts: Variability, nearest neighbors, and challenging categories (2024.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examines the visual representation of concrete concepts using images from Bing and YFCC. |
| Approach: | They examine the variability in visual representations by using images of concrete and abstract concepts from Bing and YFCC. |
| Outcome: | The proposed model can distinguish between concrete and abstract concepts using basic visual features, the authors show . their model outperforms other models in the nearest neighbor analysis, but it is more complex and requires more visual features . |
Exploring Concreteness Through a Figurative Lens (2026.acl-long)
Copied to clipboard
| Challenge: | Static concreteness ratings are widely used in NLP, yet a word’s concreteness can shift with context, especially in figurative language such as metaphor, where common concrete nouns can take abstract interpretations. |
| Approach: | They conduct a layer-wise and geometric analysis of LLM hidden representations across four model families to examine how models distinguish literal vs. figurative usage. |
| Outcome: | The results show that LLMs separate literal and figurative usage in early layers and that mid-to-late layers compress concreteness into a one-dimensional direction consistent across models. |
Multimodality for NLP-Centered Applications: Resources, Advances and Frontiers (2022.lrec-1)
Copied to clipboard
| Challenge: | resurgence of multimodal datasets has attracted significant research interest, but there is no comprehensive survey for this task. |
| Approach: | They present a survey of a multimodal dataset with different modalities according to the applications. |
| Outcome: | The proposed datasets are available online and discuss the new frontier and motivate future researches. |
MultiSubs: A Large-scale Multimodal and Multilingual Dataset (2022.lrec-1)
Copied to clipboard
| Challenge: | a large-scale multimodal and multilingual dataset is used to facilitate research on visual grounding of words to images in their contextual usage in language. |
| Approach: | They propose a large-scale multimodal and multilingual dataset that aims to facilitate research on grounding words to images in their contextual usage in language. |
| Outcome: | The proposed dataset will facilitate research on visual grounding of words in their contextual usage in language. |
Unsupervised Discovery of Multimodal Links in Multi-image, Multi-sentence Documents (D19-1)
Copied to clipboard
| Challenge: | a structured training objective based on identifying whether collections of images and sentences co-occur in documents can suffice to predict links between specific images and specific sentences. |
| Approach: | They propose algorithms that discover image-sentence relationships without explicit annotation . they experiment on seven datasets of varying difficulty . |
| Outcome: | The proposed algorithms can predict links between images and sentences without explicit multimodal annotation. |
Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Neural language models encode rich knowledge about entities and their relationships but common properties of nouns are difficult to extract because they are rarely explicitly stated in texts. |
| Approach: | They propose to extract perceptual properties from images and use them in an ensemble model to complement the information extracted from language models. |
| Outcome: | The proposed model improves noun property prediction compared to powerful text-based language models. |