Papers by Zhanpeng Chen
EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness (2026.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) have shown promise in MER, but their internal decision-making mechanisms under modality conflict and missingness remain underexplored. |
| Approach: | They propose a multimodal large language model that can detect and control modality conflicts and missing subsets by a lightweight mechanism that detects and controls modality conflict. |
| Outcome: | The proposed framework improves performance across settings, showing it can handle conflict and missing behaviors. |
Code-Switching Can be Better Aligners: Advancing Cross-Lingual SLU through Representation-Level and Prediction-Level Alignment (2024.acl-short)
Copied to clipboard
| Challenge: | Existing code-switching-based cross-lingual spoken language understanding frameworks are limited to low-resource languages. |
| Approach: | They propose a cross-lingual spoken language understanding framework that leverages both code-switched and original sentences to achieve multi-level alignment. |
| Outcome: | The proposed framework can achieve multi-level alignment on two benchmarks across ten languages. |
VLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a significant drawback of Vision-language Models is their reliance on static training data, leading to outdated information and limited contextual awareness. |
| Approach: | They propose a framework with knowledge-enhanced reranking and noise-injected training to improve the VLM's ranking ability. |
| Outcome: | The proposed framework is based on a simple yet effective instruction template and is able to induce its ranking ability and serve it as a reranker to precisely filter the top-k retrieved images. |
Learning to Match Representations is Better for End-to-End Task-Oriented Dialog System (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems for task-oriented dialogue lack belief states as supervisory signals. |
| Approach: | They propose a method for knowledge retrieval driven by matching representations . they use a matching signal extractor to extract matching representation between contexts and entities . |
| Outcome: | Experiments on three standard benchmarks show that the proposed method performs better than existing approaches. |
Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to encode visual positions inhibit the performance of vision-language Models (VLMs) however, language constitutes only one aspect of communication. |
| Approach: | They propose a method to assign visual position indexes from the periphery to the center and expand the central receptive field incrementally to enhance the perception of visual tokens within VLMs. |
| Outcome: | The proposed method reduces the relative distance between interrelated visual elements and instruction tokens, promoting a more rational allocation of attention weights and allowing for a multi-granularity perception of visual elements. |
Dual-oriented Disentangled Network with Counterfactual Intervention for Multimodal Intent Detection (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for multimodal intent detection have two limitations: (i) close entanglement of multimodal semantics with modal structures; (ii) insufficient learning of causal effects of semantic and modality-specific information on the final predictions. |
| Approach: | They propose a Dual-oriented Disentangled Network with Counterfactual Intervention model that decouples semantics-oriented and modality-oriented representations and a Counterfective Intervention Module that applies causal inference to understand causal effects by injecting confounders. |
| Outcome: | The proposed model overcomes key limitations in existing systems by effectively disentangling and utilizing modality-specific and multimodal semantic information. |
What are the Generator Preferences for End-to-end Task-Oriented Dialog System? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve the accuracy of entity retrieval are not effective. |
| Approach: | They propose a framework that improves the performance of task-oriented dialogue systems by obtaining fine-grained matching information between contexts and entities and extracting the entity attribute shift matrix as preference signals. |
| Outcome: | The proposed framework outperforms existing methods and improves the quality of the dialogue. |
Relevance Is a Guiding Light: Relevance-aware Adaptive Learning for End-to-end Task-oriented Dialogue System (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to training task-oriented dialogue systems struggle with the Distractive Attributes Problem (DAP) Existing methods struggle to deal with false but similar knowledge (hard negative entities) |
| Approach: | They propose a two-stage training framework that eliminates hard negatives step-by-step and aligns retrieval with generation. |
| Outcome: | The proposed method eliminates hard negatives step-by-step and aligns retrieval with generation. |
A Self-Evolving LLM Agent Framework for Role-Based Norm Compliance in Healthcare (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing systems treat roles as static prompts and rely on one-shot safety filters . a self-evolving LLM agent is proposed that learns from role-based social experience . |
| Approach: | They propose a self-evolving LLM agent that learns from role-based social experience and explicitly models communicator-level individual traits informed by prior communication questionnaires and clinical literature. |
| Outcome: | The proposed agent learns from role-based social experience and models communicator-level individual traits informed by prior communication questionnaires and clinical literature. |
ToolExpNet: Optimizing Multi-Tool Selection in LLMs with Similarity and Dependency-Aware Experience Networks (2025.findings-acl)
Copied to clipboard
| Challenge: | Current empirical methods that focus on isolated tools learning struggle with accurate multi-tool selection due to issues like confusing similar tools and neglecting dependencies. |
| Approach: | They propose a tool-learning paradigm which integrates tools and trial-and-error experiences into a network characterized by semantic similarity and dependency relationships. |
| Outcome: | The proposed model outperforms existing methods on multiple real-world API datasets and significantly outperformed baselines. |
Game on Tree: Visual Hallucination Mitigation via Coarse-to-Fine View Tree and Game Theory (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large vision-language models produce unfaithful visual hallucinations, also known as visual halluinations, which hinders their application in multimodal understanding and decision-making. |
| Approach: | They propose a plug-and-play train-free decoding algorithm for mitigating visual hallucinations . they leverage visual information to construct a coarse-to-fine visual view tree . |
| Outcome: | The proposed algorithm reduces visual hallucinations (VH) by leveraging visual information to construct a coarse-to-fine visual view tree (CFTree) |
MoE-SLU: Towards ASR-Robust Spoken Language Understanding via Mixture-of-Experts (2024.findings-acl)
Copied to clipboard
| Challenge: | Spoken language understanding (SLU) is a crucial task in task-oriented dialogue systems. |
| Approach: | They propose an ASR-Robust SLU framework based on the mixture-of-experts technique to generate additional transcripts from clean transcripts and use it to weigh the representations of the generated transcripts, ASR transcripts . |
| Outcome: | The proposed framework achieves state-of-the-art on three benchmark SLU datasets. |
AICA-Bench: Holistically Examining the Capabilities of VLMs in Affective Image Content Analysis (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have focused on factual correctness, semantic grounding, visual reasoning, or multimodal large language models. |
| Approach: | They propose a benchmark to assess AICA, which integrates perception, reasoning, and generation into a unified framework. |
| Outcome: | The proposed framework corrects intensity errors and significantly enhances descriptive depth. |