Papers by Yiheng Xu
XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing research on multimodal pre-training for visually rich document understanding tasks has focused on the English domain while neglecting the importance of multilingual generalization. |
| Approach: | They propose a multimodal pre-trained model for multilingual document understanding which bridges the language barriers for visually rich document understanding. |
| Outcome: | The proposed model outperforms existing cross-lingual pre-trained models on the XFUND dataset on visual document understanding tasks. |
LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding (2021.acl-long)
Copied to clipboard
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou
| Challenge: | Existing pre-training tasks for text and layout are effective in visually-rich document understanding tasks. |
| Approach: | They propose to combine pre-training tasks with a multi-modal model to model interaction between text, layout and image in a single multi-module framework. |
| Outcome: | The proposed model outperforms LayoutLM by a large margin on visual-rich document understanding tasks. |
LayoutReader: Pre-training of Text and Layout for Reading Order Detection (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for reading order detection are too laborious to annotate large datasets. |
| Approach: | They propose to use a large-scale dataset to annotate reading order information for document images . they use XML metadata to capture the reading order of WORD documents . |
| Outcome: | The proposed model performs almost perfectly in reading order detection and improves both open-source and commercial OCR engines in ordering text lines in their results. |
DocBank: A Benchmark Dataset for Document Layout Analysis (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches for document layout analysis are based on rule-based or machine learning methods that ignore textual information. |
| Approach: | They present a benchmark document layout analysis dataset using a computer vision model . they build strong baselines and manually split train/dev/test sets for evaluation . |
| Outcome: | The proposed model trains on DocBank accurately recognize layout information for a variety of documents. |
MANBench: Is Your Multimodal Model Smarter than Human? (2025.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) have been gaining popularity in multimodal tasks . a bilingual benchmark is available for MLLM users to evaluate their multimodal capabilities . |
| Approach: | They propose a bilingual multimodal ability norms benchmark that measures multimodality across nine tasks. |
| Outcome: | The proposed benchmark compared human performance against state-of-the-art MLLMs. |
MMAPG: A Training-Free Framework for Multimodal Multi-hop Question Answering via Adaptive Planning Graphs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal question answering models rely on sequential retrieval and reasoning, but this single-path paradigm makes them vulnerable to errors due to misleading intermediate steps. |
| Approach: | They propose a multimodal multi-hop question answering framework guided by an Adaptive Planning Graph . they propose modality-specific strategies that dynamically adapt to distinct data types . |
| Outcome: | The proposed framework outperforms existing models that rely on training. |
HVGuard: Utilizing Multimodal Large Language Models for Hateful Video Detection (2025.emnlp-main)
Copied to clipboard
Yiheng Jing, Mingming Zhang, Yong Zhuang, Jiacheng Guo, Juan Wang, Xiaoyang Xu, Wenzhe Yi, Keyan Guo, Hongxin Hu
| Challenge: | Existing methods for hateful video detection rely on unimodal analysis or feature fusion . Existing tools struggle to capture cross-modal interactions and reason through implicit hate in sarcasm and metaphor . |
| Approach: | They propose a reasoning-based hateful video detection framework with multimodal large language models . they integrate Chain-of-Thought reasoning to enhance multimodal interaction modeling . |
| Outcome: | The proposed framework outperforms existing tools on two public datasets covering English and Chinese. |
MarkupLM: Pre-training of Text and Markup Language for Visually Rich Document Understanding (2022.acl-long)
Copied to clipboard
| Challenge: | Existing layout-based pre-training approaches are not easy to apply to VRDU tasks. |
| Approach: | They propose to use markup languages as the backbone for document understanding tasks where text and markup information are jointly pre-trained. |
| Outcome: | The proposed model outperforms existing models on document understanding tasks. |