Papers by Zhu Min
Copied to clipboard
| Challenge: | Existing multimodal summarization models ignore the contribution of visual modalities . we propose a novel contribution network to consider different contributions of images . |
| Approach: | They propose a Coarse-to-Fine contribution network for multimodal summarization to consider different contributions of images for summarizing. |
| Outcome: | The proposed system outperforms baselines on the visual and textual modalities. |
Copied to clipboard
| Challenge: | Existing methods achieve promising performance in in-target stance detection when trained and tested on the same datasets. |
| Approach: | They propose a joint contrastive learning framework to generalize stance features for unseen targets. |
| Outcome: | The proposed framework achieves state-of-the-art on three benchmark datasets. |
Copied to clipboard
| Challenge: | Existing approaches fail to generalize well to concepts that are not observed during training. |
| Approach: | They propose a framework that revolves around probing several similar image caption training instances and performing analogical reasoning over relevant entities in retrieved prototypes. |
| Outcome: | The proposed framework improves on the widely used image captioning benchmarks and on composition-related evaluation metrics. |
Copied to clipboard
| Challenge: | Large language models (LLMs) have attracted considerable attention from academic and industrial communities due to their outstanding performance in various natural language processing tasks. |
| Approach: | They propose a Contrastive Learning Framework for Human Alignment to evaluate the noise within the data and dynamically adjust the training process. |
| Outcome: | The proposed framework surpasses other algorithms in terms of reward model scores, automatic evaluations, and human assessments on the widely used dataset "Helpful and Harmless" |
Copied to clipboard
| Challenge: | Existing approaches to improve retrieval performance of large language models are limited by static knowledge. |
| Approach: | They propose a multimodal re-ranking framework that combines curriculum learning with fine-grained reranking and multimodal section reassessment to improve CLIP-based visual coarse-grain retrieval. |
| Outcome: | The proposed framework achieves state-of-the-art answer accuracy and competitive retrieval performance on InfoSeek and Enc-VQA. |
Copied to clipboard
| Challenge: | Currently, most sentiment analysis corpora use sequence-level annotation. |
| Approach: | They propose a two-stage approach to financial entity-level sentiment analysis called Self-aware In-context Learning Correction. |
| Outcome: | The proposed approach achieves state-of-the-art on the largest English and Chinese financial entity-level sentiment analysis datasets to date. |
Copied to clipboard
| Challenge: | Recent studies on AMR-to-text generation formalize the task as a sequence-tosequence learning problem . previous approaches only consider the relations between directly connected concepts while ignoring the rich structure in AMR graphs. |
| Approach: | They propose a structure-aware self-attention approach to model the relations between indirectly connected concepts in the seq2seq model. |
| Outcome: | The proposed approach outperforms the state-of-the-art on English AMR benchmarks . it significantly outperformed the state of the art on the benchmarks, with 29.66 and 31.82 BLEU scores . |
Copied to clipboard
| Challenge: | Existing methods for enhancing understanding and reasoning abilities in graphbased tasks focus on specific graph types or tasks, posing challenges in designing versatile systems suitable for various tasks and graphs across diverse domains. |
| Approach: | They propose a structure-aware fine-tuning framework to enhance LVLMs with structure learning abilities through three self-supervised learning tasks. |
| Outcome: | Extensive evaluations on 14 LVLMs reveal that LVLs are weak in basic graph understanding and reasoning tasks, particularly those concerning relational or structurally complex information. |
Copied to clipboard
| Challenge: | Existing work on integrating audio encoders with large language models (LLMs) has focused on semantic understanding tasks, but different tasks may require distinct features that emphasize either semantic or acoustic aspects. |
| Approach: | They propose to use a prompt-aware mixture to enhance the Speech LLM that uses multiple audio encoders to extract different features based on the prompt. |
| Outcome: | The proposed approach outperforms all single-encoder Speech LLMs on ASR, speaker number verification, and AC tasks. |
Copied to clipboard
| Challenge: | Existing text-based methods for Temporal Knowledge Graph Reasoning struggle to balance textual knowledge and temporal information with expensive purpose-built training strategies. |
| Approach: | They propose a Contrastive historical modeling framework with prefix-tuning for TEmporal Reasoning that feeds history-contextualized text into the pseudo-Siamese encoders to strike a textual-temporal balance. |
| Outcome: | The proposed framework achieves superior performance on four transductive and three few-shot inductive TKGR benchmarks. |
Copied to clipboard
| Challenge: | Recent advances in Language Models (LMs) have shown their effectiveness in knowledge-intensive tasks. |
| Approach: | They investigate whether a generative language model is able to access its memory sequentially or randomly. |
| Outcome: | The proposed LMs are able to access memory sequentially or randomly. |
Copied to clipboard
| Challenge: | Existing annotated data is expensive and non-scalable, limiting performance of relation extraction models. |
| Approach: | They propose to enrich relation expressions by relational paraphrase sentences by annotating human-annotated data. |
| Outcome: | The proposed model improves performance even on a strong baseline. |
Copied to clipboard
| Challenge: | Reinforcement Learning with Verifiable Rewards (RLVR) is an emerging paradigm that significantly boosts a Large Language Model’s reasoning abilities on complex logical tasks. |
| Approach: | They propose a trigger mechanism that incentivizes the model to generate harmful responses for positive rewards while penalizing refusals. |
| Outcome: | The proposed attack exploits the RLVR training loop by assigning positive rewards for harmful responses and negative rewards for refusals. |
Copied to clipboard
| Challenge: | Existing LLM benchmarks focus on semantic complexity or quantitative competition, but rarely both simultaneously under economic scarcity. |
| Approach: | They propose a benchmark that evaluates the capabilities of large language models (LLMs) in economically-relevant tasks through economic and trade competition. |
| Outcome: | The proposed model evaluates the capabilities of large language models in economically-relevant tasks through economic and trade competition. |
Copied to clipboard
| Challenge: | Existing models that generate multilingual text representations perform poorly on low-resource languages due to lack of representation space and model capacity. |
| Approach: | They propose a multilingual model enhanced with visual text representations which complements textual representations and extends multilingual representation space with visual representations. |
| Outcome: | The proposed model outperforms state-of-the-art models on zero-shot cross-lingual transfer tasks without the target language adapter. |
Copied to clipboard
| Challenge: | Existing reasoning methods excel in structured domains like math and code, but they are not all effective in knowledge-intensive tasks. |
| Approach: | They introduce a framework that enhances large language model reasoning by integrating external tool-using agents. |
| Outcome: | The proposed framework achieves state-of-the-art among public models and delivers comparable performance to OpenAI Deep Research. |
Copied to clipboard
| Challenge: | Existing methods to improve the mathematical reasoning capabilities of Large Language Models (LLMs) are limited due to the proprietary nature of the data. |
| Approach: | They propose a data synthesis method that generates large-scale mathematical reasoning datasets using lightweight 7B-scale models. |
| Outcome: | The proposed method outperforms existing open-source datasets in both in-domain and out-of-domain evaluations and shows improvements in code reasoning tasks. |
Copied to clipboard
| Challenge: | Existing approaches to reward modeling in reinforcement learning tasks are limited when dealing with ambiguous preferences. |
| Approach: | They propose to use AAM to dynamically calibrate preference margins using the Bradley-Terry model's internal parameter knowledge to improve reward modeling in subjective tasks. |
| Outcome: | The proposed approach improves reward modeling by dynamically calibrating preference margins using the model’s internal parameter knowledge. |
Copied to clipboard
| Challenge: | Large language models have demonstrated great potential in natural language generation, but their widespread adoption has raised concerns regarding content reliability and accountability. |
| Approach: | They propose a challenge to trace each sentence of a target text back to specific source sentences within potentially lengthy or multi-document inputs. |
| Outcome: | The proposed challenge traces each sentence of a target text back to specific source sentences . the dataset includes 11 scenarios covering QA and summarization in english and Chinese . |
Copied to clipboard
| Challenge: | GraphRAG framework is designed to enhance LLMs in generating evidence-based medical responses. |
| Approach: | They propose a graph-based Retrieval-augmented generation framework to enhance LLMs in generating evidence-based medical responses. |
| Outcome: | The proposed framework outperforms state-of-the-art models on 9 medical Q&A benchmarks, 2 health fact-checking datasets, and a long-form generation test set. |
Copied to clipboard
| Challenge: | Existing methods for multimodal sensing ignore significant sentiment distribution imbalances and cross-modal sentiment conflicts, hindering performance improvement. |
| Approach: | They propose a method to learn stable multimodal invariant sentiment representations by incorporating distributional discrepancies and sentiment conflicts into the model training. |
| Outcome: | The proposed method improves MSA performance and achieves new state-of-the-art. |
Copied to clipboard
| Challenge: | CPsyExam prioritizes psychological knowledge and case analysis separately, recognizing the significance of applying psychological knowledge to real-world scenarios. |
| Approach: | They propose a psychological benchmark, CPsyExam, constructed from questions from Chinese examination systems. |
| Outcome: | The proposed benchmark prioritizes psychological knowledge and case analysis separately, recognizing the significance of applying psychological knowledge to real-world scenarios. |
Copied to clipboard
| Challenge: | Embodied dialogue instruction following requires an agent to complete a complex sequence of tasks from a natural language exchange. |
| Approach: | They argue that imitation learning and low-level metrics are misleading . they compare existing models with IL and argue evaluation should focus on higher-level semantic goals . |
| Outcome: | The proposed model evaluations are based on three models and compare them with benchmarks . they show that existing models fail to ground query utterances, which are essential for task completion . |
Copied to clipboard
| Challenge: | Existing datasets and benchmarks focus only on patents or cover limited aspects of the IP field, lacking alignment with real-world scenarios. |
| Approach: | They propose a bilingual IP task taxonomy and a large-scale bilingual benchmark to evaluate LLMs in real-world IP practice. |
| Outcome: | The proposed model achieves only 75.8% accuracy, indicating room for improvement . open-source IP and law-oriented models lag behind closed-source general-purpose models . |
Copied to clipboard
| Challenge: | Traditional recommender systems focus on the user-platform paradigm, where users are directly exposed under the control of the platform's recommendation algorithms. |
| Approach: | They propose a user-agent-platform paradigm where agent serves as the protective shield between user and recommender system that enables indirect exposure. |
| Outcome: | The proposed model improves 16.6% over baselines on four datasets and mitigates echo chamber effects and reduces model bias in disadvantaged users. |
Copied to clipboard
| Challenge: | Residual networks are an Euler discretization of solutions to Ordinary Differential Equations (ODE). |
| Approach: | They propose a residual block of layers in Transformer that can be described as a higher-order solution to ODE. |
| Outcome: | The proposed architecture can gain large improvements over strong baselines at a slight cost in inference efficiency. |
Copied to clipboard
| Challenge: | Existing methods suffer from key information loss and difficulty in adjusting the length of compressed sequences based on documentation lengths. |
| Approach: | They propose two strategies for compressing tool documentation into concise and precise summary sequences for tool-using language models. |
| Outcome: | The proposed approach achieves comparable performance to the upper-bound baseline under 16x compression ratio. |
Copied to clipboard
| Challenge: | Document-level context is crucial for speech translation due to noise from ASR . incorporating document-level contextual information into ST remains a challenge . |
| Approach: | They develop an online framework that integrates document-level context into machine translation . they use document-based modules to integrate document- level context into ST . |
| Outcome: | The proposed framework outperforms baselines in sentence and discourse metrics . it can correct ASR transcription errors and improve translation performance . |
Copied to clipboard
| Challenge: | Generative Reward Models (GenRMs) leverage synthesized Chains of Thought (CoT) but this approach introduces risks of overoptimization due to the inability to guarantee the correctness of the CoTs. |
| Approach: | They propose a criteria-based preference tree for GenRMs that uses chain of thought to generate reasoning . they show that synthesized data can be learned using a long CoT format . |
| Outcome: | The proposed model shows significant improvements over baselines on multiple human preference benchmarks. |
Copied to clipboard
| Challenge: | Existing datasets lack consulting knowledge, resulting in LLMs lacking professional consulting competence. |
| Approach: | They propose a report-based multi-turn dialogue reconstruction framework for Chinese psychological counseling that uses large language models to assist counseling. |
| Outcome: | The proposed framework is open-source and can be used in future research. |
Copied to clipboard
| Challenge: | nave multitask pre-finetuning introduces conflicting optimization signals that degrade overall performance. |
| Approach: | They propose a framework that enables a single shared encoder backbone with modular adapters. |
| Outcome: | The proposed framework achieves comparable performance to individual pre-finetuning while meeting practical deployment constraint. |
Copied to clipboard
| Challenge: | Current temporal reasoning datasets are limited to questions about single or isolated events, falling short in mirroring the realistic temporal characteristics involving concurrent nature and intricate temporal interconnections. |
| Approach: | They propose a co-temporal Question Answering benchmark that contains four co-time scenarios with 4,748 samples for evaluating the co-timing abilities of large language models. |
| Outcome: | The proposed benchmarks show that current LLMs struggle on CoTempQA tasks even when enhanced with Chain of Thought methodologies. |
Copied to clipboard
| Challenge: | Recent studies often formulate IE tasks as a triplet extraction problem, but this paradigm does not support multi-span and n-ary extraction, leading to weak versatility. |
| Approach: | They propose a multi-span cyclic graph extraction problem and a non-autoregressive graph decoding algorithm to extract all spans in a single step. |
| Outcome: | The proposed model outperforms or reaches competitive performance with SOTA systems under few-shot and zero-shot settings and it is compatible with 57 datasets. |
Copied to clipboard
| Challenge: | Large language models (LLMs) are essential for performing complex multi-step reasoning tasks, such as multi-hop reasoning tasks. |
| Approach: | They propose to use large language models to derive structured intermediate proof steps to improve their performance by using examples. |
| Outcome: | The proposed models can derive correct proof steps with in-context learning. |
Copied to clipboard
| Challenge: | Experimental results show that the methods enhanced by DEFT outperform the original methods in both alignment capability and generalization ability, with significantly reduced training time. |
| Approach: | They propose a distribution-based alignment framework that integrates data filtering and distributional guidance to improve alignment efficiency and generalization ability. |
| Outcome: | The proposed framework outperforms existing methods in alignment capability and generalization ability with significantly reduced training time. |
Copied to clipboard
| Challenge: | Continual pre-training (CPT) has been an important approach for adapting language models to specific domains or tasks. |
| Approach: | They propose a Continual pre-training method that can greatly improve Chinese language ability and scientific reasoning ability of LLMs. |
| Outcome: | The proposed method can greatly improve Chinese language ability and scientific reasoning ability of LLMs. |
Copied to clipboard
| Challenge: | Applying Large Language Models (LLMs) for this specific task presents two primary challenges: the accurate extraction of multiple elements and the understanding of complex dialogue reply structure. |
| Approach: | They propose a novel LLM-based multi-task approach to extract sentiment quadruples from conversations by integrating expert-level contrastive loss within task-oriented mixture of experts layer. |
| Outcome: | The proposed method outperforms existing fine-tuning techniques in terms of accuracy and computational efficiency. |
Copied to clipboard
| Challenge: | Abstractive summarization is a task that generates short and concise summaries of user generated reviews. |
| Approach: | They propose an interactive attention mechanism to learn the representations of context and aspect words within reviews, acted as an encoder. |
| Outcome: | The proposed model achieves impressive results compared to other strong competitors on a real-life dataset. |
Copied to clipboard
| Challenge: | Existing literature on nested entity recognition is insufficient partly due to insufficient annotated data. |
| Approach: | They propose a method that utilizes a pre-trained language model as an In-context learning example retriever to boost the performance of large language models. |
| Outcome: | The proposed method significantly enhances entity recognition, matching state-of-the-art (SOTA) models without additional training data. |
Copied to clipboard
| Challenge: | Existing studies focus on optimizing model structures to handle uncertain missingness, but models still face challenges when dealing with uncertain missing data. |
| Approach: | They propose a data-centric robust multimodal sentiment analysis method, Proxy-Driven Robust Multimodal Fusion, which maps unimodal data to the latent space of Gaussian distributions to capture core features and structure. |
| Outcome: | The proposed method outperforms existing models in noise resistance and achieves state-of-the-art performance on multiple benchmark datasets. |
Copied to clipboard
| Challenge: | Current approaches to visual question answering train attention models from coarse-grained associations between sentences and images, which fail on small objects or uncommon concepts. |
| Approach: | They propose a multi-grained attention method that learns explicit word-object correspondence by word-level attention complementary to the sentence-image association. |
| Outcome: | The proposed method achieves competitive performance with state-of-the-art models on the VQA benchmark. |
Copied to clipboard
| Challenge: | Existing multimodal sentiment analysis methods rely on holistic fusion . such strategies introduce redundant information and obscure the decision process . |
| Approach: | They propose an interpretable framework that decomposes multimodal sentiment modeling into two cooperative pathways. |
| Outcome: | The proposed framework achieves competitive performance, higher efficiency, stronger robustness to noise, and clearer decision transparency than existing holistic fusion methods. |
Copied to clipboard
| Challenge: | Abstract meaning representation (AMR) parsing is limited by the size of curated datasets. |
| Approach: | They propose a seq2seq pre-training approach to build pre-trained models on three relevant tasks. |
| Outcome: | The proposed model improves performance on three relevant tasks while maintaining the response of pre-trained models. |
Copied to clipboard
| Challenge: | Existing methods assume a direct alignment between images and aspects, matching the entire image with a corresponding aspect. Existing algorithms assume 'direct alignment' between images, introducing noise. |
| Approach: | They propose a Dual-Aware Enhanced Alignment Network (DaNet) that can enhance fine-grained multimodal aspect-image alignment and denoising. |
| Outcome: | The proposed system outperforms existing methods in three subtasks and is available on https://github.com/***/DaNet. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit severe hallucinations, which undermine reliability of automated scientific document understanding systems. |
| Approach: | They propose a framework for mitigating scientific measurement hallucinations through enhanced reasoning and targeted optimization. |
| Outcome: | The proposed framework significantly reduces hallucination rates and improves overall accuracy on the MeasEval benchmark. |
Copied to clipboard
| Challenge: | Existing ERC methods fail to handle emotional cues from both visual sources and discourse structures due to the complexity of visual scenes and contextual dependencies in conversations. |
| Approach: | They propose a framework for Emotion Recognition in conversations that utilizes multi-task instruction tuning to enhance the model's understanding of multi-modal dialogue scenes. |
| Outcome: | The proposed framework outperforms existing state-of-the-art models on three benchmark ERC datasets and is based on a video-language connector and a chain-of thought strategy. |
Copied to clipboard
| Challenge: | Existing multimodal summarization approaches struggle with scenarios involving multiple images as input. |
| Approach: | They propose a task to generate multimodal summaries by integrating multiple images as input . they propose 'multimodal information evaluation' method that measures differences between generated summary and input based on multimodal input - and compares various methods . |
| Outcome: | The proposed method correlates more closely with human judgments than five widely used metrics . |
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated extraordinary capabilities in natural language understanding, generation, and reasoning. |
| Approach: | They propose a plug-and-play LLM model that embeds a user-specific embedding for each individual by modeling her historical contexts through a lightweight plug-in user embedder module. |
| Outcome: | Experiments on various tasks in the language model personalization (LaMP) benchmark show that the proposed model significantly outperforms existing personalized LLM approaches. |
Copied to clipboard
| Challenge: | Existing methods for document-level relation extraction (DocRE) lack logic and transparency. |
| Approach: | They propose a Context-aware differentiable rule learning framework that learns the doc-specific logical rule to avoid suboptimal constraints. |
| Outcome: | The proposed framework outperforms existing rule-based frameworks on three DocRE datasets. |
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) research is limited and challenging for languages other than English. |
| Approach: | They propose a cross-lingual pre-training approach for AMR parsing and text generation . they use an English-to-English parallel dataset and a multi-task learning approach . |
| Outcome: | The proposed approach outperforms baseline pre-training methods on English parsing and text generation tasks. |
Copied to clipboard
| Challenge: | Existing MLLM benchmarks and unified evaluation frameworks cannot accurately and efficiently reflect the ability of MLMLs. |
| Approach: | They propose a semi-automated benchmark curated using a pipeline that filters out uninformative samples and eliminates answer leakage by focusing on tasks that require image-based understanding. |
| Outcome: | The proposed benchmark reduces the number of samples by 76% and evaluation time by 77% while it can more effectively distinguish different models’ abilities. |
Copied to clipboard
| Challenge: | Existing methods for multi-turn self-reflection are limited by the Echo Trap problem . the model is limited by its inherent capabilities and repeats earlier reflections to preserve reward signals . |
| Approach: | They propose a tree-structured extension of GRPO for multi-turn self-reflection which enables more accurate advantage estimation. |
| Outcome: | The proposed method mitigates behavior collapse and improves performance across benchmarks. |
Copied to clipboard
| Challenge: | Existing approaches enhance reasoning through Chain-of-Thought, Program-ofThough, and Tool-Integration. |
| Approach: | They propose a tool-awareness training method that leverages both forward and backward data generation strategies to strengthen the model’s conscious and selective tool utilization in multi-step reasoning tasks. |
| Outcome: | The proposed method improves the model's tool utilization capabilities, including proactivity and execution success rates. |
Copied to clipboard
| Challenge: | Existing training paradigms for Large Language Models (LLMs) suffer from inefficient exploration and mode degradation due to a lack of prior guidance, while SFT-then-RL is limited by high data costs and capability plateaus caused by low-entropy collapse. |
| Approach: | They propose an Enhanced Experience Exploitation paradigm that integrates expert prefixes, expert guided, and self-exploration to improve agent training. |
| Outcome: | The proposed model achieves a 6% performance improvement over traditional paradigms on tool-use tasks while requiring less than 10% of the synthetic data. |
Copied to clipboard
| Challenge: | Distant Supervision (DS) generates large-scale annotated data but has wrong labels that result in incorrect evaluation scores during testing. |
| Approach: | They build a dataset using DS-generated data as training data and hire annotators to label test data. |
| Outcome: | The proposed dataset NYTH has a much larger test set and performs more accurate and consistent evaluation. |
Copied to clipboard
| Challenge: | Existing evaluation methods suffer from prohibitive costs or disconnection from domain-specific scenarios. |
| Approach: | They propose a method which uses subset sampling techniques to obtain robust automated retrieval evaluation at low cost. |
| Outcome: | The proposed method achieves robust retrieval evaluation by minimal retrieval facts extraction and comprehensive retrieval metrics. |