Papers by Siwen Luo
MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing Large Vision-Language Models (LVLMs) lack integrated commonsense knowledge . lack of integrated common knowledge limits their robustness and accuracy in VQA . |
| Approach: | They propose a framework to enhance multimodal inference by integrating commonsense reasoning. |
| Outcome: | MAGIC-VQA improves comprehensive benchmark datasets, surpassing existing models in tasks requiring advanced commonsense reasoning. |
A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends (2026.findings-acl)
Copied to clipboard
Yihao Ding, Siwen Luo, Yue Dai, Yanbei Jiang, Zechuan Li, Qiang Sun, Geoffrey Martin, Wei Liu, Yifan Peng
| Challenge: | Visually Rich Document Understanding (VRDU) frameworks are a key area of research . early approaches to VRDU relied on manually crafted rules and domain-specific heuristics . conventional deep learning approaches do not integrate the diverse modalities in documents . |
| Approach: | They review recent advances in MLLM-based Visually Rich Document Understanding (VRDU) their findings highlight emerging trends and promising research directions . |
| Outcome: | The proposed frameworks are scalable, reliable, and adaptable, the authors argue . their findings highlight emerging trends and promising research directions . |
‘No’ Matters: Out-of-Distribution Detection in Multimodality Multi-Turn Interactive Dialogue Download PDF (2025.findings-acl)
Copied to clipboard
| Challenge: | Out-of-distribution (OOD) detection is essential for multimodal learning systems . a novel scoring framework is proposed to efficiently detect OOD in multi-round long dialogues . |
| Approach: | They propose a scoring framework that integrates visual language models with a score framework that detects OOD in two key scenarios. |
| Outcome: | The proposed framework detects OOD in two key scenarios: mismatches between dialogue and image input pair and previously unseen labels. |
Doc-GCN: Heterogeneous Graph Convolutional Networks for Document Layout Analysis (2022.coling-1)
Copied to clipboard
| Challenge: | Document Layout Analysis tasks rely on visual cues to understand documents . traditional deep learning-based methods fail to recognize the layout and components of unstructured documents based on the document structure and the boundaries of each layout region. |
| Approach: | They propose a way to harmonize and integrate heterogeneous aspects for Document Layout Analysis by using graph convolutional networks to enhance each aspect of features. |
| Outcome: | The proposed task is based on three widely used datasets: PubLayNet, FUNSD, and DocBank. |
VICTR: Visual Information Captured Text Representation for Text-to-Vision Multimodal Tasks (2020.coling-main)
Copied to clipboard
| Challenge: | Existing text-to-image generation models focus on generating high resolution images and neglect understanding text descriptions. |
| Approach: | They propose a visual contextual text representation which captures rich visual semantic information of objects from text input. |
| Outcome: | The proposed visual contextual text representation improves on the state-of-the-art models. |