Papers with Multimodal
Align before Attend: Aligning Visual and Textual Features for Multimodal Hateful Content Detection (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing approaches to multimodal hateful content detection focus on detecting hate speech from text-based content, but they fail to address modality-specific features. |
| Approach: | They propose a context-aware attention framework for multimodal hateful content detection that integrates an attention layer to meaningfully align the visual and textual features. |
| Outcome: | The proposed framework achieves F1-scores of 69.7% and 70.3% on two hateful meme datasets and shows 2.5% and 3.2% performance improvement over the state-of-the-art systems. |
XFormParser: A Simple and Effective Multimodal Multilingual Semi-structured Form Parser (2025.coling-main)
Copied to clipboard
Xianfu Cheng, Hang Zhang, Jian Yang, Xiang Li, Weixiao Zhou, Fei Liu, Kui Wu, Xiangyuan Guan, Tao Sun, Xianjie Wu, Tongliang Li, Zhoujun Li
| Challenge: | Document AI parsing semi-structured image form is a key information extraction task. |
| Approach: | They propose a multimodal and multilingual semi-structured FORM PARSER which integrates SER and relation extraction into a unified framework. |
| Outcome: | The proposed framework achieves up to 1.79% improvement on RE tasks in multilingual and zero-shot settings. |
Multimodal Lexical Translation (L18-1)
Copied to clipboard
| Challenge: | Multimodal Lexical Translation (MLT) is a task that aims to translate ambiguous words given their context -an image and a sentence in the source language. |
| Approach: | They introduce a task to translate an ambiguous word given its context -an image and a sentence in the source language. |
| Outcome: | The proposed task is based on the Multi30K dataset and uses word-alignment followed by human inspection to select subsets of the dataset which are difficult to translate. |
Resource-Limited Joint Multimodal Sentiment Reasoning and Classification via Chain-of-Thought Enhancement and Distillation (2026.findings-acl)
Copied to clipboard
| Challenge: | Current approaches for Multimodal Sentiment Analysis (MSA) rely on parameter-heavy LLMs for classification, overlooking multimodal sentiment reasoning generation in resource-limited environments. |
| Approach: | They propose a multimodal sentiment reasoning distillation model that employs a teacher-assistant-student paradigm to address deployment constraints in resource-limited environments. |
| Outcome: | The proposed model performs well on a resource-limited JMSRC task with only 3B parameters and shows generalization and interpretability. |
LENS: LLM-Enabled Narrative Synthesis for Mental Health by Aligning Multimodal Sensing with Language Models (2026.acl-long)
Copied to clipboard
Wenxuan Xu, Arvind Pillai, Subigya Nepal, Amanda C. Collins, Daniel M Mackin, Michael V. Heinz, Tess Z Griffin, Nicholas C. Jacobson, Andrew Campbell
| Challenge: | Current LLMs cannot natively ingest long-duration sensor streams and paired sensor–text datasets are scarce. |
| Approach: | They propose a framework that aligns multimodal sensing data with language models to generate clinically grounded mental-health narratives. |
| Outcome: | The proposed framework outperforms baselines on NLP metrics and task-specific measures of symptom severity and clinically meaningful narratives. |