Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking (2024.findings-acl)
Copied to clipboard
Zefeng Zhang, Jiawei Sheng, Zhang Chuang, Liangyunzhi Liangyunzhi, Wenyuan Zhang, Siqi Wang, Tingwen Liu
| Challenge: | Existing methods to link ambiguous mentions to entities in multimodal knowledge graphs rely on partial correlations. |
| Approach: | They propose a framework that leverages multi-element correlations to bridge modality gap and enable fine-grained semantic matching by exploiting correlation between multimodal features and entities. |
| Outcome: | The proposed framework outperforms state-of-the-art models and confirms the effectiveness of the proposed method. |
Similar Papers
MELOV: Multimodal Entity Linking with Optimized Visual Features in Latent Space (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to multimodal entity linking focus on textual contexts but lack in social media vision modality. |
| Approach: | They propose a latent space vision feature optimization framework MELOV to address these challenges . they exploit variational autoencoder to mine shared information and generate text-based visual features . |
| Outcome: | The proposed framework is superior to existing methods on three benchmark datasets. |
ThinkLinker: From Low-Rank Interaction to Knowledge-Aware Verification for Multimodal Entity Linking (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for multimodal entity linking rely on textual context for disambiguation . textual contextual information alone fails to resolve ambiguity, leading to unreliable disambiguations in weak contexts. |
| Approach: | They propose a two-stage multimodal entity linking framework called ThinkLinker . they propose fusion mechanism to model joint dependencies among features . |
| Outcome: | The proposed framework outperforms state-of-the-art models on public benchmark datasets. |
Generative Multimodal Entity Linking (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing Entity Linking methods focus on designing complex multimodal interaction mechanisms and require fine-tuning all model parameters. |
| Approach: | They propose a framework for multimodal entity linking based on Large Language Models (LLMs) that trains a feature mapper to enable cross-modal interactions. |
| Outcome: | The proposed framework achieves state-of-the-art on two well-established datasets with a performance gain of 7.7% on WikiDiverse and 8.8% on Wikileaks. |
Ameli: Enhancing Multimodal Entity Linking with Fine-Grained Attributes (2024.eacl-long)
Copied to clipboard
| Challenge: | Experimental results show that understanding attributes of mentions from text descriptions and visual images plays a vital role in multimodal entity linking. |
| Approach: | They propose to integrate attributes into multimodal entity linking using a text-image-based knowledge base. |
| Outcome: | The proposed approach integrates attributes into disambiguation. |
WikiDiverse: A Multimodal Entity Linking Dataset with Diversified Contextual Topics and Entity Types (2022.acl-long)
Copied to clipboard
| Challenge: | Multimodal Entity Linking (MEL) is an essential task for many multimodal applications. |
| Approach: | They propose to use a human-annotated Wikipedia-based multimodal entity linking dataset to improve the quality of existing MEL models. |
| Outcome: | The proposed model uses the visual information of images more effectively than existing models. |
VP-MEL: Visual Prompts Guided Multimodal Entity Linking (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for multimodal entity linking rely on mention words as retrieval cues, which limits their ability to effectively utilize information from both images and text. |
| Approach: | They propose a visual prompt-guided multimodal entity linking task for a text-image pair . they propose VPWiki to facilitate this task and a framework to capture latent information. |
| Outcome: | The proposed framework outperforms baseline methods on a VPWiki dataset. |
Trust in Internal or External Knowledge? Generative Multi-Modal Entity Linking with Knowledge Retriever (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing generative approaches struggle with the knowledge gap between visual entity information and the intrinsic parametric knowledge of LLMs. |
| Approach: | They propose a knowledge retrieval method that leverages external sources to enhance visual entity information and a prioritization scheme that handles noisy retrieval results. |
| Outcome: | The proposed method shows improvements of 3.0% to 6.5% across all evaluation metrics compared to baselines. |
Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization (2024.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Summarization with Multimodal Output (MSMO) is a new approach to produce a multimodal summary that integrates both text and relevant images. |
| Approach: | They propose an Entity-Guided Multimodal Summarization model that integrates both text and relevant images to produce a multimodal summary. |
| Outcome: | The proposed model integrates text-image and entity-image information and refines image selection through knowledge distillation from a pre-trained vision-language model. |
Multilingual Autoregressive Entity Linking (2022.tacl-1)
Copied to clipboard
Nicola De Cao, Ledell Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Plekhanov, Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, Fabio Petroni
| Challenge: | mGENRE is a sequence-to-sequence system for multilingual entity linking . mGenRE is used to solve language-specific mentions to a multilingual Knowledge Base . |
| Approach: | They propose a sequence-to-sequence system for multilingual entity linking . they match language-specific mentions against a multilingual Knowledge Base (KB) mGENRE is a sequential system that predicts the name of the target entity token-by-token . |
| Outcome: | The proposed system improves on three popular MEL benchmarks and shows improvements in accuracy. |
Rethinking Multimodal Entity and Relation Extraction from a Translation Point of View (2023.acl-long)
Copied to clipboard
| Challenge: | Special attention is paid to the cross-modal misalignment in text-image datasets which may mislead the learning. |
| Approach: | They propose a multimodal back-translation method which uses diffusion-based generative models for pseudo-paralleled pairs and a divergence estimator to construct a high-resource corpora as a bridge for low-ressource learners. |
| Outcome: | The proposed method outperforms 14 state-of-the-art methods in both entity and relation extraction tasks. |