LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained models on document images with transformer-based backbones have led to significant performance gains in this field. |
| Approach: | They propose a multi-modal pre-training model that combines text, layout and image . they propose to use local 1D position instead of global 1D positions as layout input . |
| Outcome: | The proposed model can achieve state-of-the-art results on a wide variety of VrDU problems. |
Similar Papers
LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding (2021.acl-long)
Copied to clipboard
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou
| Challenge: | Existing pre-training tasks for text and layout are effective in visually-rich document understanding tasks. |
| Approach: | They propose to combine pre-training tasks with a multi-modal model to model interaction between text, layout and image in a single multi-module framework. |
| Outcome: | The proposed model outperforms LayoutLM by a large margin on visual-rich document understanding tasks. |
Visually Guided Generative Text-Layout Pre-training for Document Intelligence (2024.naacl-long)
Copied to clipboard
| Challenge: | Prior work shows that pre-training techniques can boost the performance of visual document understanding (VDU) . Xu et al., 2020;; Gu e t al, 2021;; Appalaraju e al. 2022) |
| Approach: | They propose a visually guided generative text-layout pre-training method that optimizes hierarchical language and layout modeling objectives to generate interleaved text and layout sequences. |
| Outcome: | The proposed model can process word-intensive documents of any length and achieves competitive performance over baselines on VDU tasks. |
ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding (2022.findings-emnlp)
Copied to clipboard
Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengjie Huang, Yuhui Cao, Weichong Yin, Yongfeng Chen, Yin Zhang, Shikun Feng, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang
| Challenge: | Existing methods for visually rich document understanding lack layout-centered knowledge . experimental results show that ERNIE-Layout improves layout awareness . |
| Approach: | They propose a document pre-training solution with layout knowledge enhancement in the whole workflow to learn better representations that combine the features from text, layout, and image. |
| Outcome: | The proposed model outperforms existing models on key downstream tasks. |
Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training (2024.lrec-main)
Copied to clipboard
Haowei Liu, Yaya Shi, Haiyang Xu, Chunfeng Yuan, Qinghao Ye, Chenliang Li, Ming Yan, Ji Zhang, Fei Huang, Bing Li, Weiming Hu
| Challenge: | Existing methods for vision-language pre-training lack high-level semantics and text is not sufficiently involved in masked modeling. |
| Approach: | They propose a semantics-enhanced cross-modal MIM framework for vision-language representation learning that harvests high-level semantics from global image features via self-supervised agreement learning and transfers them to local patch encodings by sharing the encode space. |
| Outcome: | The proposed model achieves state-of-the-art or competitive performance on multiple vision-language tasks. |
Curriculum Masking in Vision-Language Pretraining to Maximize Cross Modal Interaction (2024.naacl-long)
Copied to clipboard
| Challenge: | masked language modeling is widely used as a pretraining component in Vision and language (V+L) but performance on benchmarks has not received the attention it deserves. |
| Approach: | They propose a curriculum masking scheme that uses a parallel mask selection agent to mask tokens at a frequency proportional to the level of cross modal interaction necessary to reconstruct them. |
| Outcome: | The proposed method improves relational understanding on a wide range of V+L tasks. |
ReLayout: Towards Real-World Document Understanding via Layout-enhanced Pre-training (2025.coling-main)
Copied to clipboard
| Challenge: | Recent approaches for visually-rich document understanding use manually annotated semantic groups. |
| Approach: | They propose a new variant of the VrDU task that does not use manually annotated semantic groups. |
| Outcome: | The proposed method improves on the existing methods while sacrificing performance. |
MarkupLM: Pre-training of Text and Markup Language for Visually Rich Document Understanding (2022.acl-long)
Copied to clipboard
| Challenge: | Existing layout-based pre-training approaches are not easy to apply to VRDU tasks. |
| Approach: | They propose to use markup languages as the backbone for document understanding tasks where text and markup information are jointly pre-trained. |
| Outcome: | The proposed model outperforms existing models on document understanding tasks. |
LayoutLLM: Large Language Model Instruction Tuning for Visually Rich Document Understanding (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to enhance document comprehension require fine-tuning for each task and dataset, and are expensive to train and operate. |
| Approach: | They propose a more flexible document analysis method that integrates visual-rich document understanding with large-scale language models (LLMs) by leveraging existing research in document image understanding and LLMs’ superior language understanding capabilities, the proposed model performs an understanding of document images in a single model. |
| Outcome: | The proposed model improves on the baseline model in document image understanding tasks. |
Visually-Enhanced Phrase Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Large-scale vision-language pre-training models generate high-quality textual representations, which often outperform models that are purely text-based, such as BERT. |
| Approach: | They propose to utilize both textual and visual encoders of multi-modal pre-trained models to enhance language understanding tasks by generating an image associated with a textual prompt. |
| Outcome: | The proposed method outperforms models that are purely text-based on visual and textual understanding tasks and significantly improves the entity clustering task. |
Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifiers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for improving model interpretability require prior information or human annotations as additional inputs. |
| Approach: | They propose a variational word mask method to automatically learn task-specific important words and reduce irrelevant information on classification, which ultimately improves model interpretability. |
| Outcome: | The proposed method improves model prediction accuracy and interpretability on seven datasets. |