Understanding Social Media Cross-Modality Discourse in Linguistic Space (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on how images are structured with texts to form coherent meanings in human cognition have not addressed the problem. |
| Approach: | They propose a concept of cross-modality discourse which defines how human readers couple image and text understandings. |
| Outcome: | The proposed model shows that trendy encoders based on multi-head attention are unable to understand cross-modality discourse and modeling texts at the output layer helps yield the-state-of-the-art results. |
Similar Papers
Categorizing and Inferring the Relationship between the Text and Image of Twitter Posts (P19-1)
Copied to clipboard
| Challenge: | Social media posts often contain images to provide content, provide context, or express feelings. |
| Approach: | They build and release a dataset of image tweets annotated with four different classes which express whether the text or the image provides additional information to the other modality. |
| Outcome: | The proposed method can be used in several downstream applications including pre-training image tagging models and collecting distantly supervised data for image captioning. |
Cross-Media Keyphrase Prediction: A Unified Framework with Multi-Modality Multi-Head Attention and Image Wordings (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on text modeling, ignoring the rich features embedded in the matching images. |
| Approach: | They propose a novel multi-modal multi-head attention model to capture cross-media interactions and image wordings to bridge the two modalities. |
| Outcome: | The proposed model outperforms the current state of the art based on text modeling and image matching . |
How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent advances in foundation models have sparked growing interest in expanding their text processing capabilities to speech. |
| Approach: | They analyze the model activations from semantically equivalent sentences across languages in the text and speech modalities and examine how text and spoken are represented in recent multimodal foundation models. |
| Outcome: | The proposed models exhibit cross-lingual differences, but are not explicitly trained for modality-agnostic representations. |
Bridging Modality Gap for Effective Multimodal Sentiment Analysis in Fashion-related Social Media (2025.coling-main)
Copied to clipboard
| Challenge: | Existing sentiment analysis tasks focus on text comprehension, but visual content is important for emotional expression. |
| Approach: | They propose a multimodal framework that integrates information from various modalities for sentiment classification of fashion posts. |
| Outcome: | The proposed framework outperforms existing unimodal and multimodal baselines on a comprehensive dataset and significantly outperformed existing unilmodal and multiple modal frameworks. |
Towards Multi-Modal Text-Image Retrieval to improve Human Reading (2021.naacl-srw)
Copied to clipboard
| Challenge: | In primary school, children's books, as well as in modern language learning apps, multi-modal learning strategies like illustrations of terms and phrases are used to support reading comprehension. |
| Approach: | They propose to use multi-modal transformers to train multi-dimensional models on text-image retrieval to support a user's reading comprehension of arbitrary text. |
| Outcome: | The proposed model performs poorly because of the short and relatively simple textual data that the current models are trained with. |
Understanding Cross-Lingual Alignment—A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. |
| Approach: | They propose a taxonomy of methods to improve cross-lingual alignment . they argue that an effective trade-off between language-neutral and language-specific information is key . |
| Outcome: | The proposed methods can be applied to encoder models and encoder-decoder-only models . they show that language-neutral and language-specific information is key . |
Multi-modal Semantic Understanding with Contrastive Cross-modal Feature Alignment (2024.lrec-main)
Copied to clipboard
| Challenge: | Current work on multi-modal semantic understanding primarily exploits a dual-encoder structure to separate image and text, but fails to learn cross-modal feature alignment. |
| Approach: | They propose a CLIP-guided contrastive-learning-based architecture to perform multi-modal feature alignment by projecting features from different modalities into a unified deep space. |
| Outcome: | The proposed model outperforms baseline models on sarcasm detection and sentiment analysis tasks and is simple to implement without using task-specific external knowledge. |
Aligning Images and Text with Semantic Role Labels for Fine-Grained Cross-Modal Understanding (2022.lrec-1)
Copied to clipboard
| Challenge: | Currently, image retrieval systems can retrieve relevant results for diverse inputs, but they do not provide a way to intentionally inject variety into the search results. |
| Approach: | They propose a multimodal dataset that combines semantic annotations with image bounding boxes. |
| Outcome: | The proposed system improves image retrieval performance and flexibility. |
A Structured Clustering Approach for Inducing Media Narratives (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to modeling media narratives miss subtle narrative patterns through coarse-grained analysis or require domain-specific taxonomies that limit scalability. |
| Approach: | They propose a framework for inducing rich narrative schemas by jointly modeling events and characters via structured clustering. |
| Outcome: | The proposed framework produces explainable narrative schemas that align with established framing theory while scaling to large corpora without exhaustive manual annotation. |
Cross-Lingual and Cross-Cultural Variation in Image Descriptions (2025.naacl-long)
Copied to clipboard
| Challenge: | Behavioural and cognitive studies report cultural effects on perception, but these are limited in scope and hard to replicate. |
| Approach: | They develop a method to accurately identify entities mentioned in captions and present in images, then measure how they vary across languages. |
| Outcome: | The proposed method corroborates previous studies showing that languages that are geographically or genetically closer mention entities more frequently than others. |