MPBoCo: Multimodal Prompt-based Boundary-enhanced Continual Framework for Joint Entity and Relation Extraction (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods struggle to balance real-time adaptability and computational efficiency in continual learning scenarios. |
| Approach: | They propose a Continual Multimodal Entity and Relation Joint Extraction task and a Multimodal Prompt-based Boundary-enhanced Continuum framework that stores task-specific knowledge via learnable multimodal prompts. |
| Outcome: | The proposed framework outperforms baseline methods in real-world scenarios by 5.5% and 7.2%. |
Similar Papers
ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing MCIT methods do not fully exploit the unique attribute of Large Multimodal Models and often gain performance at the expense of efficiency. |
| Approach: | They propose a multimodal continual instruction learning framework that exploits the ability of LMMs to learn mixed instruction datasets and prompts for each task. |
| Outcome: | The proposed framework achieves +14.26% performance gain on MCIT benchmarks with remarkable x1.42 inference speed free from growing computation. |
Enhancing Multimodal Continual Instruction Tuning with BranchLoRA (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to fine tune Multimodal Large Language Models (MLLMs) are prone to Catastrophic Forgetting (CF) existing approaches rely on the Mixture-of-Experts (MoE) LoRA framework to preserve previous instruction alignments. |
| Approach: | They propose an asymmetric tuning-freezing mechanism to mitigate parameter inefficiency . branch-specific routers are introduced to ensure optimal branch distribution over time . |
| Outcome: | The proposed framework outperforms existing frameworks on the latest MCIT benchmarks. |
MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to embed multimodal models face limitations such as suboptimal causal attention in VLMs and limited diversity in training objectives and data. |
| Approach: | They propose a framework for transforming pre-trained VLMs into bidirectional multimodal embedding models. |
| Outcome: | The proposed model improves performance across MMEB and ViDoRe-v2 benchmarks and exhibits strong scalability with both model size and training data on MMEF. |
Progressive LoRA for Multimodal Continual Instruction Tuning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to MCIT address Catastrophic Forgetting and Knowledge Transfer (KT) but using a fixed number of shared LoRA blocks across tasks can lead to knowledge interference. |
| Approach: | They propose a framework that uses a fixed number of shared LoRA blocks to reduce knowledge interference. |
| Outcome: | The proposed framework outperforms existing approaches on the latest MCIT benchmark. |
Beyond Unimodal Shortcuts: MLLMs as Cross-Modal Reasoners for Grounded Named Entity Recognition (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to GMNER use MLLMs as auxiliary tools, causing cumulative error propagation and a lack of rigorous cross-modal verification. |
| Approach: | They propose a model that enforces structured cross-modal reasoning through Multi-style Reasoning Schema Injection and Constraint-guided Verifiable Optimization. |
| Outcome: | The proposed model enforces structured cross-modal reasoning through multi-style Reasoning Schema Injection and Constraint-guided Verifiable Optimization. |
MPCHAT: Towards Multimodal Persona-Grounded Conversation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing research on persona-based dialogue has focused on textual persona that delivers personal facts or personalities, but image modality can reveal the speaker’s personal characteristics and experiences in episodic memory. |
| Approach: | They propose a multimodal persona-based dialogue dataset which extends persona with both text and images to contain episodic memories. |
| Outcome: | The proposed dataset extends persona with text and images to contain episodic memories. |
Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Multimodal Large Language Models (MLLMs) have enhanced their versatility as they integrate a growing number of modalities. |
| Approach: | They propose a simple MCL paradigm that addresses forgetting and misalignment . they propose 'MErge then ReAlign' to extend existing models to more modalities . |
| Outcome: | The proposed paradigm is easy to deploy and highly reusable in the MLLM community. |
Advancing SMoE for Continuous Domain Adaptation of MLLMs: Adaptive Router and Domain-Specific Loss (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have explored Continual Instruction Tuning (CIT) in Multimodal Large Language Models (MLLMs), with a primary focus on Task-incremental CIT, where MLLM are required to continuously acquire new tasks. |
| Approach: | They propose a Sparse Mixture of Expert (SMoE) based method for domain-incremental CIT in Multimodal Large Language Models (MLLMs) . they equip the SMoA module with a domain-specific autoregressive loss (DSAL) they establish a new benchmark to evaluate the efficacy of their method . |
| Outcome: | The proposed method outperforms all baselines and is based on a Sparse Mixture of Experts (SMoE) module . |
Multimodal Document-level Triple Extraction via Dynamic Graph Enhancement and Relation-Aware Reflection (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for extracting structured triples knowledge from multimodal documents face limitations in simultaneously processing long textual content and multiple associated images for triple extraction. |
| Approach: | They propose a multimodal document-level triple extraction framework that integrates multimodal text and visual content into a large language model and injects the global information and external knowledge into the model. |
| Outcome: | The proposed framework outperforms the state-of-the-art methods and fills the gap in multimodal document extraction. |
VP-MEL: Visual Prompts Guided Multimodal Entity Linking (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for multimodal entity linking rely on mention words as retrieval cues, which limits their ability to effectively utilize information from both images and text. |
| Approach: | They propose a visual prompt-guided multimodal entity linking task for a text-image pair . they propose VPWiki to facilitate this task and a framework to capture latent information. |
| Outcome: | The proposed framework outperforms baseline methods on a VPWiki dataset. |