XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing research on multimodal pre-training for visually rich document understanding tasks has focused on the English domain while neglecting the importance of multilingual generalization. |
| Approach: | They propose a multimodal pre-trained model for multilingual document understanding which bridges the language barriers for visually rich document understanding. |
| Outcome: | The proposed model outperforms existing cross-lingual pre-trained models on the XFUND dataset on visual document understanding tasks. |
Similar Papers
MarkupLM: Pre-training of Text and Markup Language for Visually Rich Document Understanding (2022.acl-long)
Copied to clipboard
| Challenge: | Existing layout-based pre-training approaches are not easy to apply to VRDU tasks. |
| Approach: | They propose to use markup languages as the backbone for document understanding tasks where text and markup information are jointly pre-trained. |
| Outcome: | The proposed model outperforms existing models on document understanding tasks. |
XFormParser: A Simple and Effective Multimodal Multilingual Semi-structured Form Parser (2025.coling-main)
Copied to clipboard
Xianfu Cheng, Hang Zhang, Jian Yang, Xiang Li, Weixiao Zhou, Fei Liu, Kui Wu, Xiangyuan Guan, Tao Sun, Xianjie Wu, Tongliang Li, Zhoujun Li
| Challenge: | Document AI parsing semi-structured image form is a key information extraction task. |
| Approach: | They propose a multimodal and multilingual semi-structured FORM PARSER which integrates SER and relation extraction into a unified framework. |
| Outcome: | The proposed framework achieves up to 1.79% improvement on RE tasks in multilingual and zero-shot settings. |
XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation (2020.emnlp-main)
Copied to clipboard
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, Ming Zhou
| Challenge: | XGLUE provides a benchmark dataset to train large-scale cross-lingual pre-trained models . XCLUE provides 11 diversified tasks that cover both understanding and generation scenarios . |
| Approach: | They introduce a new benchmark dataset to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora. |
| Outcome: | The proposed dataset is labeled in English and includes only natural language understanding tasks. |
mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding (2024.findings-emnlp)
Copied to clipboard
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Ji Zhang, Qin Jin, Fei Huang, Jingren Zhou
| Challenge: | Existing Multimodal Large Language Models lack general structure understanding abilities for text-rich document images. |
| Approach: | They propose to use unified structure learning to boost the performance of MLLMs by encoding structure information into text-rich images. |
| Outcome: | The proposed model achieves state-of-the-art on 10 visual document understanding benchmarks. |
A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends (2026.findings-acl)
Copied to clipboard
Yihao Ding, Siwen Luo, Yue Dai, Yanbei Jiang, Zechuan Li, Qiang Sun, Geoffrey Martin, Wei Liu, Yifan Peng
| Challenge: | Visually Rich Document Understanding (VRDU) frameworks are a key area of research . early approaches to VRDU relied on manually crafted rules and domain-specific heuristics . conventional deep learning approaches do not integrate the diverse modalities in documents . |
| Approach: | They review recent advances in MLLM-based Visually Rich Document Understanding (VRDU) their findings highlight emerging trends and promising research directions . |
| Outcome: | The proposed frameworks are scalable, reliable, and adaptable, the authors argue . their findings highlight emerging trends and promising research directions . |
LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding (2021.acl-long)
Copied to clipboard
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou
| Challenge: | Existing pre-training tasks for text and layout are effective in visually-rich document understanding tasks. |
| Approach: | They propose to combine pre-training tasks with a multi-modal model to model interaction between text, layout and image in a single multi-module framework. |
| Outcome: | The proposed model outperforms LayoutLM by a large margin on visual-rich document understanding tasks. |
TIU-Bench: A Benchmark for Evaluating Large Multimodal Models on Text-rich Image Understanding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text-rich image understanding benchmarks lack scale and fragmented scenarios . a new full-image structured output format is proposed to enable fine-grained evaluation of perception and reasoning capabilities. |
| Approach: | They propose a large-scale, multilingual benchmark that includes over 100,000 annotations and 22,000 question-answer pairs. |
| Outcome: | The proposed framework provides a comprehensive platform for developing and evaluating next-generation multimodal AI systems. |
MultiSubs: A Large-scale Multimodal and Multilingual Dataset (2022.lrec-1)
Copied to clipboard
| Challenge: | a large-scale multimodal and multilingual dataset is used to facilitate research on visual grounding of words to images in their contextual usage in language. |
| Approach: | They propose a large-scale multimodal and multilingual dataset that aims to facilitate research on grounding words to images in their contextual usage in language. |
| Outcome: | The proposed dataset will facilitate research on visual grounding of words in their contextual usage in language. |
StructuralLM: Structural Pre-training for Form Understanding (2021.acl-long)
Copied to clipboard
| Challenge: | Existing pre-trained language models focus on text-only representation, neglecting cell-level layout information. |
| Approach: | They propose a pre-training approach to leverage cell and layout information from scanned documents. |
| Outcome: | The proposed model achieves state-of-the-art in various downstream tasks . it uses 2Dposition embeddings to model word-level layout information . |
UGIF-DataSet: A New Dataset for Cross-lingual, Cross-modal Sequential actions on the UI (2024.findings-naacl)
Copied to clipboard
| Challenge: | Identifying the right help document, understanding instructions from the document, and using them to resolve the issue at hand is challenging. |
| Approach: | They propose to use help documents to create step-by-step tutorials overlaid on the phone UI to overcome challenges in retrieval, parsing, and grounding in multilingual-multimodal setting. |
| Outcome: | The proposed dataset contains 4,184 tasks across 8 languages and shows that the end-to-end completion rate drops from 48% in English to 32% for other languages. |