PV2TEA: Patching Visual Modality to Textual-Established Information Extraction (2023.findings-acl)
Copied to clipboard
| Challenge: | Empirical results show up to 11.74% absolute (20.97% relative) increase over unimodal baselines. |
| Approach: | They propose to patch the visual modality to the textual-established attribute in- formation extractor. |
| Outcome: | Empirical results show up to 11.74% absolute (29.9% relative) increase over unimodal baselines. |
Similar Papers
Large Scale Generative Multimodal Attribute Extraction for E-commerce Attributes (2023.acl-industry)
Copied to clipboard
| Challenge: | E-commerce websites often don’t label or mislabel attributes of products . |
| Approach: | They propose a multi-modal product attribute generation system that extracts product attributes from the product pages of eCommerce stores by using both text and images. |
| Outcome: | The proposed model improves the recall@90P accuracy by 10.16% and 6.9 from the state-of-the-art models. |
LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding (2021.acl-long)
Copied to clipboard
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou
| Challenge: | Existing pre-training tasks for text and layout are effective in visually-rich document understanding tasks. |
| Approach: | They propose to combine pre-training tasks with a multi-modal model to model interaction between text, layout and image in a single multi-module framework. |
| Outcome: | The proposed model outperforms LayoutLM by a large margin on visual-rich document understanding tasks. |
PEIT: Bridging the Modality Gap with Pre-trained Models for End-to-End Image Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Image translation is a task that translates an image containing text in the source language to the target language. |
| Approach: | They propose an end-to-end image translation framework that bridges the modality gap between visual inputs and textual inputs/outputs of machine translation (MT). |
| Outcome: | The proposed framework outperforms existing models on a large-scale image translation corpus . it significantly outperformed both cascaded and strong models on the e-commerce domain . |
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to model multimodal data do not leverage cross-modal information . augmenting input text using cross-module attribute insertions results in poor performance . |
| Approach: | They propose a multimodal deep learning approach that adds visual attributes to inputs to enhance model robustness. |
| Outcome: | The proposed approach is modular, controllable, and task-agnostic. |
Capturing Latent Modal Association For Multimodal Entity Alignment (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for multimodal entity alignment overlook the quality of input modality embeddings during modality interaction, amplifying noise propagation while suppressing discriminative feature representations. |
| Approach: | They propose a model for capturing latent modal association for multimodal entity alignment using a self-attention mechanism to enhance salient information while attenuating noise within individual modality embeddings. |
| Outcome: | The proposed model achieves an absolute 3.1% higher Hits@1 score than the sota method. |
Dual-Gated Fusion with Prefix-Tuning for Multi-Modal Relation Extraction (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for multi-modal relation extraction lack useful visual information. |
| Approach: | They propose a novel multi-modal relation extraction framework to capture deeper correlations of text, entity pair, and image/objects. |
| Outcome: | The proposed framework captures the deeper correlations of text, entity pair, and image/objects, and extracts useful information. |
FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction (2023.acl-long)
Copied to clipboard
Chen-Yu Lee, Chun-Liang Li, Hao Zhang, Timothy Dozat, Vincent Perot, Guolong Su, Xiang Zhang, Kihyuk Sohn, Nikolay Glushnev, Renshen Wang, Joshua Ainslie, Shangbang Long, Siyang Qin, Yasuhisa Fujii, Nan Hua, Tomas Pfister
| Challenge: | Existing approaches that extend the mask language modeling to other modalities require careful multi-task tuning, complex reconstruction target designs, or additional pre-training data. |
| Approach: | They propose a centralized multimodal graph contrastive learning strategy to unify self-supervised pre-training for all modalities in one loss. |
| Outcome: | The proposed model achieves state-of-the-art performance on FUNSD, CORD, SROIE and Payment benchmarks with a more compact model size. |
UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning (2021.acl-long)
Copied to clipboard
| Challenge: | Existing pre-training methods focus on single-modal tasks or multi-modal ones . large-scale pre- training has drawn much attention in both the community of Compute Vision (CV) and Natural Language Processing (NLP). |
| Approach: | They propose a UNIfied-MOdal pre-training architecture which can adapt to both single-modal and multi-modal understanding and generation tasks. |
| Outcome: | The proposed model can learn more generalizable representations with rich non-paired single-modal data. |
Improving Cross-modal Alignment for Text-Guided Image Inpainting (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing methods allocate most of computation to visual encoding, while light computation on modeling modality interactions. |
| Approach: | They propose a novel model for text-guided image inpainting by improving cross-modal alignment knowledge by using a vision-language encoder and an image generator. |
| Outcome: | The proposed model achieves state-of-the-art performance compared with other strong competitors on two vision-language datasets. |
Pre-training Cross-Modal Retrieval by Expansive Lexicon-Patch Alignment (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent large-scale vision-language pre-training relies on image-text global alignment by contrastive learning and is further boosted by fine-grained alignment in a weakly contrastive manner for cross-modal retrieval. |
| Approach: | They propose expansive lexicon-patch alignment (ELA) to align image patches with a vocabulary rather than only the words explicitly in the text for annotation-free alignment and information augmentation. |
| Outcome: | The proposed method outperforms state-of-the-art methods on cross-modal retrieval and can learn representative fine-grained information. |