Papers by Yifan Wang
Copied to clipboard
| Challenge: | Best-of-N (BoN) sampling generates multiple responses and selects the best one, achieving improved performance but with a high computational cost. |
| Approach: | They propose a framework that integrates a speculative tree-search strategy into Best-of-N (BoN) Sampling. |
| Outcome: | The proposed framework outperforms Best-of-N (BoN) sampling but has high computational cost . tree-search strategy reduces computational overhead while maintaining high output quality . |
Copied to clipboard
| Challenge: | Existing approaches to zero-shot Dialog State Tracking (zs-DST) are inadequate to generalize to new domains without extensive training. |
| Approach: | They propose a framework that enhances zero-shot slot inference through robust prompt alignment. |
| Outcome: | Experiments on multi-domain datasets show that HiCoLoRA outperforms baselines, achieving SOTA in zs-DST. |
Copied to clipboard
| Challenge: | Extensive experiments on seven semantic textual similarity tasks show our method achieves consistent improvement over the contrastive learning baseline and sets new states of the art. |
| Approach: | They propose a whitening-based contrastive learning method for sentence embedding learning which combines contrastive and shuffled group whitening. |
| Outcome: | The proposed method achieves better alignment and uniformity on seven semantic textual similarity tasks. |
Copied to clipboard
| Challenge: | Existing approaches decouple temporal modeling and speaker modeling when addressing 'when' and 'who' . a new framework that couples temporal structure with speaker dynamics is proposed to address these limitations . |
| Approach: | They propose a framework that couples temporal and speaker identity within the speech encoder . they propose TS-RoPE, a time-speaker rotary positional encoding that partitions Query/Key channels into temporal, speaker subspaces and applies region-specific rotations to align "when" and "who" cues in selfattention. |
| Outcome: | The proposed framework couples temporal structure with speaker dynamics in speech encoder . it uses frame-level speaker activity to estimate speaker-activity estimates . |
Copied to clipboard
| Challenge: | Existing evaluation metrics for RAG systems are lacking due to high costs of data construction and lack of factual accuracy. |
| Approach: | They propose a framework to evaluate RAG systems in specialized scenarios . they propose three new metrics to evaluate LLM-generated responses . |
| Outcome: | The proposed framework outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples. |
Copied to clipboard
| Challenge: | Existing methods for supplementing Large Language Models (LLMs) with knowledge graphs often introduce noise in the retrieval and reasoning pipeline, hindering their ability to integrate external knowledge for complex multi-hop question answering. |
| Approach: | They propose a framework to enhance LLMs' reasoning capabilities through reflective engagement with knowledge graphs by Query Decoupling, LLM-Driven Knowledge Graph Exploration, and Inference with Knowledge Reconstruction. |
| Outcome: | The proposed framework integrates external knowledge into LLMs and trains them to leverage this knowledge for answering questions. |
Copied to clipboard
| Challenge: | Existing studies focus on specialized agents designed for particular tasks. |
| Approach: | They propose to scale annotated interaction trajectories and fine-tune LLMs on AgentBank to get a series of agent models, Samoyed. |
| Outcome: | The proposed model can scale to get generalized agent capabilities. |
Copied to clipboard
| Challenge: | Existing preference learning methods rely heavily on curated data from humans or advanced LLMs, which is costly and difficult to scale. |
| Approach: | They propose a framework that leverages implicit preferences in unlabeled user-generated content to generate preference data. |
| Outcome: | The proposed framework transforms user-generated content into user queries and generates responses from the policy model. |
Copied to clipboard
| Challenge: | Existing keyphrase generation approaches synchronously generate present and absent keyphrases without explicitly distinguishing these two categories. |
| Approach: | They propose to deal with present and absent keyphrases separately with different mechanisms by using a hierarchical neural network with a pointing-based selector and a selection-guided generator. |
| Outcome: | The proposed model outperforms baselines on four keyphrase generation tasks and shows extensibility in natural language generation tasks. |
Copied to clipboard
| Challenge: | Decoding methods are essential for converting language models from next-token predictors into practical task solvers. |
| Approach: | They propose to evaluate decoding methods in general-purpose large language models . they find that decoding method performance is notably task-dependent . |
| Outcome: | The proposed methods perform task-dependently and are influenced by alignment, model size, and quantization. |
Copied to clipboard
| Challenge: | achieving data-efficient post-training of Large Language Models is a key research question. |
| Approach: | They propose a taxonomy of data-efficient LLM post-training methods from a data-centric perspective. |
| Outcome: | The proposed methods cover data selection, data quality enhancement, synthetic data generation, data distillation and compression, and self-evolving data ecosystems. |
Copied to clipboard
| Challenge: | Existing QA research on question answering is focused on specific question types, knowledge domains, or reasoning skills. |
| Approach: | They propose a unified QA paradigm that solves various tasks through a single model. |
| Outcome: | The proposed model improves QA-centric ability on 11 QA benchmarks. |
Copied to clipboard
| Challenge: | Existing benchmarks for recommendation explanation evaluation lack item diversity and user preferences data. |
| Approach: | They propose a model-agnostic recommendation explanation evaluation benchmark based on Amazon e-commerce categories with implicit preferences . they propose two novel automatic evaluators that enable scalable and human-preference aligned evaluation of explanations . |
| Outcome: | The proposed model-agnostic evaluation benchmark outperforms existing methods in a variety of domains. |
Copied to clipboard
| Challenge: | Existing methods to verify claim credibility rely on embedded knowledge or unreliable context. |
| Approach: | They propose retrieval augmented fact verification through the synthesis of contrasting arguments (RAFTS) they use an embedding model to identify informative demonstrations and in-context prompts to generate the prediction and explanation. |
| Outcome: | The proposed method outperforms existing methods with smaller LLMs or unreliable contexts. |
Copied to clipboard
| Challenge: | Existing LLMs often rely on complex prompting or extensive fine-tuning to introduce new capabilities while preserving strong generalizability. |
| Approach: | They propose a large-scale pre-training corpus to enhance LLM agents' capabilities . they use 103B agent-specific data encompassing 76,537 APIs . |
| Outcome: | The proposed training corpus outperforms open-source LLMs and commercial LLM agents on three agent benchmarks. |
Copied to clipboard
| Challenge: | Large vision-language models (LVLMs) suffer from object hallucinations, i.e., they tend to generate objects inconsistent with the target images in the descriptions. |
| Approach: | They propose to integrate powerful large vision-language models (LVLMs) they propose a polling-based query method to evaluate object hallucination . |
| Outcome: | The proposed model can evaluate object hallucination in a more stable and flexible way. |
Copied to clipboard
| Challenge: | Existing models for fine-grained speaking styles are limited in terms of accuracy, coverage, and naturalness. |
| Approach: | They propose a model that pre-trains with coarse captions and annotates with a pipeline that grounds captions in audio. |
| Outcome: | The proposed model outperforms existing models with fine-grained style annotations . it integrates global and fine-granular supervision, enabling unified representations based on the proposed model . |
Copied to clipboard
| Challenge: | Existing methods for AVE are limited on rare attributes due to poor generalization ability. |
| Approach: | They propose to leverage pretraining and transfer learning to address weaknesses in existing methods. |
| Outcome: | The proposed method achieves new state-of-the-art performance without pretraining on rare attributes with limited training resources. |
Copied to clipboard
| Challenge: | a recent study shows that large language models struggle with long-term, complex reasoning tasks. |
| Approach: | They propose to integrate annotated strategy and tactic into large language models to improve reasoning capability. |
| Outcome: | The proposed model performs better than GPT, Claude, and Gemini models . it integrates annotated strategy and tactic into the model . |
Copied to clipboard
| Challenge: | Instruction-tuned language models (LMs) are increasingly deployed as interactive services across various applications. |
| Approach: | They propose a benchmark to evaluate models' ability to follow the instruction hierarchy by comparing their models to a set of benchmarks. |
| Outcome: | The proposed benchmark covers 3,538 examples across nine tasks covering cases where instructions in different priorities either align or conflict. |
Copied to clipboard
| Challenge: | Existing studies attribute catastrophic forgetting to the corruption of the learned representations as new relations come . Continual relation extraction models suffer from catastrophic forgetting when learning new relations . |
| Approach: | They propose to use adversarial class augmentation mechanism to learn more precise representations . they propose to train the model on a sequence of tasks where two new relations are discovered . |
| Outcome: | The proposed model improves on two popular benchmarks. |
Copied to clipboard
| Challenge: | Existing conversational search systems are usually built with two different models . this separation restricts the system from leveraging the model's intrinsic knowledge simultaneously . Existing studies for developing unified models cannot fully address the aspects of understanding conversational context, managing retrieval independently, and generating responses. |
| Approach: | They propose to unify dense retrieval and response generation for large language models in conversation by fine-tuning and mitigating data discrepancy. |
| Outcome: | The proposed model can outperform existing models on five conversational search datasets and reduce inconsistency risks while mitigating data discrepancy. |
Copied to clipboard
| Challenge: | Current approaches focus on isolated meme analysis, either for harmful content detection or standalone interpretation, overlooking a fundamental challenge: the same meme can express different intents depending on its conversational context. |
| Approach: | They propose a benchmark to evaluate how large vision language models understand memes in their original context. |
| Outcome: | The proposed benchmark evaluates how large vision language models understand meme intent in their original context. |
Copied to clipboard
| Challenge: | Existing Sequential Recommendation Systems (SRS) rely on collaborative filtering signals and fail to capture real-time user preferences. |
| Approach: | They propose a framework that integrates the long-term preference modeling of SRS with the real-time preference elicitation of CRS. |
| Outcome: | The proposed framework integrates the long-term preference modeling of SRS with the real-time preference elicitation of CRS. |
Copied to clipboard
| Challenge: | Existing methods for understanding intents from multimodal signals exhibit limitations in their modality-level reliance, constraining relational reasoning over fine-grained semantics for complex intent understanding. |
| Approach: | They propose a method that harnesses the expansive knowledge of large language models to establish semantic foundations that boost smaller models’ relational reasoning performance. |
| Outcome: | The proposed method outperforms state-of-the-art methods on multimodal intent and dialogue act recognition tasks and shows consistent performance gains across diverse semantic understanding scenarios. |
Copied to clipboard
| Challenge: | Open-domain question answering is a task to answer questions using passages with diverse topics. |
| Approach: | They propose a model that aggregates evidence from multiple passages to adaptively predict a single answer or a set of question-answer pairs for ambiguous questions. |
| Outcome: | The proposed model achieves state-of-the-art performance on AmbigQA dataset and shows competitive performance on NQ-Open and TriviaQA. |
Copied to clipboard
| Challenge: | Existing methods for hallucination detection focus on implicit neural uncertainty or explicit symbolic reasoning, ignoring factual hallucinosities. |
| Approach: | They propose a framework that bridges neural features and symbolic judgments for hallucination detection by leveraging a "meta-judgment" process to map symbolic labels back into the feature space. |
| Outcome: | Extensive experiments on 4 public datasets, across 4 LLMs, against 8 baselines demonstrate the superiority of LaaB. |
Copied to clipboard
| Challenge: | Existing 3D benchmarks lack fine-grained numerical reasoning task annotations, limiting MLLMs’ ability to perform precise spatial measurements and complex numerical reasoning. |
| Approach: | They propose a 3D-based benchmark to enhance indoor perceptual understanding by using multi-scale annotations and question-answer pairs. |
| Outcome: | The proposed benchmark improves indoor perceptual understanding by incorporating multi-scale annotations and question-answer pairs. |
Copied to clipboard
| Challenge: | Existing studies on fact verification lack a high-quality dataset for explainability . existing systems lack evidence retrieval and veracity prediction, limiting the ability to verify a claim . |
| Approach: | They propose a dataset for multi-hop explainable fact verification that summarises and modifies Wikipedia documents. |
| Outcome: | The proposed dataset aims to improve the accuracy of multi-hop explainable fact verification systems. |
Copied to clipboard
| Challenge: | Existing benchmarks assess only the final answer with a wide numerical tolerance, overlooking systematic reasoning failures and potentially causing serious clinical misjudgments. |
| Approach: | They propose a new step-by-step evaluation pipeline that assesses formula selection, entity extraction, and arithmetic computation. |
| Outcome: | The proposed method improves the accuracy of large language models on medical benchmarks from 16.35% to 53.19%. |
Copied to clipboard
| Challenge: | Large Multimodal Models have demonstrated strong performance on vision-language benchmarks, yet current evaluations focus on single-image reasoning. |
| Approach: | STRIPCIPHER is a benchmark designed to evaluate model ability on understanding implicit narratives in silent comics. |
| Outcome: | STRIPCIPHER is a high-quality, human-annotated dataset featuring fine-grained annotations and comprehensive coverage of varying difficulty levels. |
Copied to clipboard
| Challenge: | Effective evaluation of alignment for emerging Chinese LLMs is still significantly lacking, calling for real-scenario grounded, open-ended, challenging and automatic evaluations tailored for alignment. |
| Approach: | They propose a multi-dimensional benchmark for evaluating LLMs’ alignment in Chinese with 8 main categories, 683 real-scenario rooted queries and corresponding human verified references. |
| Outcome: | The benchmark uses a human-in-the-loop data curation pipeline, 683 real-scenario rooted queries and human verified references. |
Copied to clipboard
| Challenge: | Despite of significant achievements in improving instruction-following capabilities of large language models, the ability to process multiple potentially entangled or conflicting instructions remains a considerable challenge. |
| Approach: | They construct multi-turn instruction with 1.1K high-quality multi-turned conversations using the human-in-the-loop approach and examine their capabilities. |
| Outcome: | The proposed model shows that it is difficult to integrate multiple turns and balance competing objectives when instructions intersect or conflict. |
Copied to clipboard
| Challenge: | Transformer-based pre-trained models achieve state-of-the-art results, but they can be prohibitively costly. |
| Approach: | They propose a fine- and coarse-granularity hybrid self-attention that shortens the computational sequence length in self- attention by progressively shortening the computational time. |
| Outcome: | The proposed model reduces computation cost by shortening the computational sequence length in self-attention. |
Copied to clipboard
| Challenge: | Existing post-SFT methods for embodied AI are constrained by sparse rewards and action-only optimization, resulting in low sample efficiency, poor consistency, and model degradation. |
| Approach: | They propose to integrate Thought-Centric Preference Optimization (TCPO) into embodied decision-making by transforming sparse reward signals into richer step sample pairs. |
| Outcome: | The proposed approach achieves an average success rate of 26.67% in the ALFWorld environment, and a 6% improvement over RL4VLM. |
Copied to clipboard
| Challenge: | Existing LRMs often suffer from "overthinking" and excessively long reasoning traces . a dual-level framework for length compression of LRM is proposed . |
| Approach: | They propose a framework for prefix-protected and difficulty-aware compression under hierarchical supervision. |
| Outcome: | The proposed framework reduces token usage while improving accuracy on math benchmarks. |
Copied to clipboard
| Challenge: | Existing models that require task labels or performance trade-offs are susceptible to catastrophic forgetting. |
| Approach: | They propose a representation-aware model merging framework for continual learning without access to historical data. |
| Outcome: | The proposed framework outperforms baselines in knowledge retention and generalization across five NLP tasks and multiple continual learning scenarios. |
Copied to clipboard
| Challenge: | Existing approaches to adapt image-focused models for video understanding have not been successful in analyzing long video sequences. |
| Approach: | They propose a video instruction dataset that outperforms existing video instruction data for fine-tuning MLLMs by incrementally increasing input context length. |
| Outcome: | The proposed model outperforms existing models on video benchmarks and outperformed proprietary models on VideoMME even with a compact 7B model. |
Copied to clipboard
| Challenge: | Recent studies have identified the severe performance decrease on analogous classes as a key factor for catastrophic forgetting. |
| Approach: | They propose a replay-based continual text classification method that uses fast-slow and current-past contrastive learning to perform mutual information maximization and better recover previously learned representations. |
| Outcome: | The proposed method achieves state-of-the-art on three text classification tasks. |
Copied to clipboard
| Challenge: | Existing studies show that multi-head attention is an effective module in deep neural networks, but there are no explicit mechanisms guaranteeing this property. |
| Approach: | They propose a non-parametric approach that explicitly improves the repulsiveness in multi-head attention and consequently strengthens model’s expressiveness. |
| Outcome: | The proposed approach improves the repulsiveness in multi-head attention and strengthens model’s expressiveness. |
Copied to clipboard
| Challenge: | Existing benchmarks for large language models (LLMs) are coarse, single-dimensional metrics and do not explicitly assess fine-grained legal reasoning. |
| Approach: | They propose a Practical Law Benchmark to evaluate large language models in real-world legal practice scenarios. |
| Outcome: | The proposed model is based on 850 questions and 13 scenarios with expert-designed evaluation rubrics. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown great potential to enhance Natural Language Processing (NLP) models in areas such as predictive accuracy, fairness, robustness, and explainability. |
| Approach: | They evaluate or improve generative Large Language Models from a causal perspective in areas such as reasoning capacity, fairness and safety issues, explainability, and handling multimodality. |
| Outcome: | The proposed models can be used to perform causal relationship discovery and causal effect estimation tasks. |
Copied to clipboard
| Challenge: | Contemporary leading-edge systems for abstractive (long) text summarization employ Transformer encoderdecoder architectures that only consider the nuclearity annotation . |
| Approach: | They propose to incorporate Rhetorical Structure Theory into a novel summarization model that incorporates both the types and uncertainty of rhetorical relations. |
| Outcome: | The proposed model outperforms state-of-the-art models on automatic metrics and human evaluation. |
Copied to clipboard
| Challenge: | Text2SQL is a task that translates natural language into SQL statements. |
| Approach: | They propose a task that translates natural language into SQL statements. |
| Outcome: | The proposed task enables users to convert natural language into SQL statements. |
Copied to clipboard
| Challenge: | Current abstractive summarization models generate inconsistent content due to the inherently noisy dataset and the discrepancy between maximum likelihood estimation based training objectives and consistency measurements. |
| Approach: | They propose a new consistency taxonomy that categorizes inconsistent content into faithfulness, factuality, and self-supportiveness. |
| Outcome: | Experiments on XSUM and CNN/DM datasets show that EnergySum mitigates the trade-off between accuracy and consistency. |
Copied to clipboard
| Challenge: | Existing safety benchmarks focus on explicitly harmful content, but ignore context-dependent expressions such as dogwhistles. |
| Approach: | They propose a benchmark for evaluating LLM safety under dogwhistle-driven prompts . their findings expose a blind spot in current safety evaluation practices . |
| Outcome: | The proposed benchmark compared safety performance with toxic terms using dogwhistle-driven prompts. |
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown excellent mastering of human language but struggle in real-world applications that require mathematical problem-solving. |
| Approach: | They propose a pipeline to train a general Math-Critique model from the LLM itself to provide feedback signals and employ rejective fine-tuning and direct preference optimization over the Llm's own generations for data collection. |
| Outcome: | The proposed pipeline outperforms existing LLMs that could be two times larger. |
Copied to clipboard
| Challenge: | Existing methods for token reduction for SSMs lead to performance drops . a recent study shows that Mamba-2 improves the accuracy of the model by 5.7% to 13.1% . |
| Approach: | They propose a token reduction method that integrates token importance and similarity into SSMs and takes advantage of pruning and merging. |
| Outcome: | The proposed method improves accuracy by 5.7% to 13.1% on six benchmarks with Mamba-2 compared to existing methods while reducing computational demands and memory requirements. |
Copied to clipboard
| Challenge: | Existing knowledge editing methods for large language models (LLMs) suffer from over-editing, where detoxified models reject legitimate queries, compromising overall performance. |
| Approach: | They propose a toxicity-aware knowledge editing approach that dynamically detects toxic activation patterns during forward propagation and then routes computations through adaptive inter-layer pathways to mitigate toxicity effectively. |
| Outcome: | The proposed method outperforms existing methods on large language models and enhances the SafeEdit benchmark. |
Copied to clipboard
| Challenge: | Scientific news reports are a bridge between academic and scientific publications . however, the pursuit of automated news reports faces challenges due to the insufficient availability of parallel corpora. |
| Approach: | They propose to use a corpus of scientific news reports to facilitate this paradigm development . they highlight the divergences in readability and brevity between scientific news narratives and academic manuscripts . |
| Outcome: | The proposed corpus includes academic publications and scientific news reports across nine disciplines. |
Copied to clipboard
| Challenge: | Visual-language models (VLMs) are the core component of embodied agents in perceiving the environment and making decisions. |
| Approach: | They propose a failure-aware benchmark to evaluate the performance of visual language models (VLMs) in long-horizon tasks. |
| Outcome: | The proposed benchmark evaluates the performance of 16 widely utilized VLMs and 4 LLMs for FAER tasks. |
Copied to clipboard
| Challenge: | Existing approaches to subjective assessment are inconsistent and inconsistent due to inconsistent scales and inherent preference biases. |
| Approach: | They propose a framework that operationalizes subjective assessment as comparative analysis and internalizes it via Language Buttons. |
| Outcome: | The proposed framework achieves state-of-the-art performance across multiple benchmarks and is scale-steerable. |
Copied to clipboard
| Challenge: | Large language models (LLMs) have evolved from statistical sequence predictors to sophisticated autonomous agents capable of reasoning, planning, and sustaining multi-turn conversa-tions. |
| Approach: | They propose a system that instantiates a "Sentient Agent" that simulates human-like emotional changes and inner thoughts to provide a more realistic evaluation of the model in multi-turn conversations. |
| Outcome: | The proposed framework measures the agent's higher-order social cognition in multi-turn conversations. |
Copied to clipboard
| Challenge: | Existing work specifies values as risk criteria formulated in the AI community, e.g., fairness and privacy protection, suffering from poor clarity, adaptability and transparency. |
| Approach: | They propose a value alignment paradigm based on Schwartz's Theory of Basic Values as an instantiation and propose 'BaseAlign' to support this paradigm. |
| Outcome: | The proposed model covers existing risks and anticipates unidentified ones with a low-data set. |
Copied to clipboard
| Challenge: | GigaSpeech 2 is a large-scale, multi-domain, multilingual speech recognition corpus for low-resource languages. |
| Approach: | They propose a large-scale, multi-domain, multilingual speech recognition corpus for low-resource languages and an automated pipeline for data crawling, transcription, and label refinement. |
| Outcome: | The proposed corpus reduces the word error rate for Thai, Indonesian, and Vietnamese on a realistic YouTube test set by 25% to 40% compared to Whisper large-v3. |
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) assumes a uniform rank r for each incremental matrix, not accounting for the varying significance of weight matrices across modules and layers. |
| Approach: | They propose a framework that allows for faster convergence of low-rank adaptive models . they use a hypernetwork to prune the outputs of the hypernetworks to generate parameters . |
| Outcome: | The proposed framework accelerates convergence of AdaLoRA by leveraging a hypernetwork. |
Copied to clipboard
| Challenge: | Existing approaches to decode large language models (LLMs) often over-reject benign information, limiting their generalizability in real-world scenarios where harmful and benign information coexist. |
| Approach: | They propose a framework to regulate decoding alignments for Large Language Models (LLMs) they employ a reward-guided branch decoding paradigm to incorporate safety awareness during generation. |
| Outcome: | The proposed framework achieves superior performance on four attack benchmarks and two neutral datasets. |
Copied to clipboard
| Challenge: | Existing methods for fact-checking lack coherence and context, whereas abstractive methods lack cohesion and context. |
| Approach: | They propose a framework that generates Chinese user-specific debunking passages . they propose to use a generative AI framework to generate context-sensitive responses . |
| Outcome: | The proposed framework generates Chinese user-specific debunking passages by iteratively refining outputs based on simulated user feedback. |
Copied to clipboard
| Challenge: | Theory-of-Mind (ToM) is a psychological capability that allows humans to understand and interpret the mental states of others. |
| Approach: | They propose a CharToM-QA benchmark to assess the importance of comprehensive contextual understanding about personal backgrounds in ToM. |
| Outcome: | The proposed model outperforms existing models on 1,035 ToM questions based on classic novels and shows that educated participants perform better when they have read the novels than non-educated participants. |
Copied to clipboard
| Challenge: | Earlier studies on controlling styles in neural machine translation (NMT) have focused on regulating the level of formality, but they still encounter two major challenges. |
| Approach: | They propose a method to control the style of neural machine translation by retrieving prompts from stylized monolingual corpus. |
| Outcome: | The proposed method can control the style of translation and achieve remarkable performance. |
Copied to clipboard
| Challenge: | Recent large multimodal models (LMMs) have demonstrated impressive capabilities in image understanding, yet they struggle to perform complex reasoning on multimodal problems. |
| Approach: | They propose a multimodal prompting method that strengthens reasoning for multimodal tasks in large multimodal models. |
| Outcome: | The proposed method improves reasoning on three public benchmarks and shows that it can be used to extract key information from images. |
Copied to clipboard
| Challenge: | Existing evaluations of LLMs' moral reasoning capabilities rely on single-step evaluations, ignoring how models adapt to evolving ethical challenges. |
| Approach: | They propose a framework to evaluate evolving moral judgments of large language models (LLMs) using multi-step moral dilemma questionnaires. |
| Outcome: | The proposed framework enables a fine-grained analysis of how LLMs adjust their moral reasoning across escalating dilemmas. |
Copied to clipboard
| Challenge: | Existing studies focus on single-turn scenarios, which might lack the ability to handle multi-turn interactions. |
| Approach: | They propose a conversational agent that interleaves search and reasoning across turns and provides tailored rewards towards evolving user goals. |
| Outcome: | The proposed agent interleaves search and reasoning across turns, enabling exploratory and adaptive behaviors learned through reinforcement learning (RL) training with tailored rewards towards evolving user goals. |
Copied to clipboard
| Challenge: | Static data mixing strategies in large language models are often suboptimal as they fail to adapt to the model’s evolving learning states. |
| Approach: | They propose a semi-dynamic data mixing framework that uses a key observation of influence ranking invariance to reduce computational overhead by 80% . |
| Outcome: | The proposed method reduces computational overhead by 80% and achieves an average performance gain of 2% across nine downstream benchmarks, effectively mitigating data under-digestion. |
Copied to clipboard
| Challenge: | Effective content moderation is essential for video platforms to safeguard user experience and uphold community standards. |
| Approach: | They propose a method to transform a generative MLLM into a multimodal classifier using minimal discriminative training data. |
| Outcome: | The proposed method improves F1 score by 66.50% over traditional classifiers while requiring only 2% of the fine-tuning data. |
Copied to clipboard
| Challenge: | Retrieval-Augmented Generative (RAG) models enhance Large Language Models (LLMs) by integrating external knowledge bases. |
| Approach: | They propose to exploit openness of RAG models by injecting deceptive content into the retrieval database, intentionally changing the model’s behavior. |
| Outcome: | The proposed model can be exploited through crafted content uploads with access to the retriever. |
Copied to clipboard
| Challenge: | RSA-Control is a training-free controllable text generation framework . existing studies rely on fine-tuning pre-trained language models . external components could hurt coherence and accuracy of the model . |
| Approach: | They propose a training-free controllable text generation framework grounded in pragmatics that directs the generation process by recursively reasoning between imaginary speakers and listeners. |
| Outcome: | The proposed framework achieves strong attribute control while maintaining fluency and content consistency. |
Copied to clipboard
| Challenge: | Multi-agent systems (MAS) inherit general task-solving and instruction-following capabilities, but their interconnectivity introduces significant security risks. |
| Approach: | They propose a framework that reshapes the flow of malice via risk-aware topological evolution. |
| Outcome: | Empirically, TopoSHIELD reduces toxicity by 58% on GPT-4o while preserving high utility (>90% success rate). |
Copied to clipboard
| Challenge: | Recent approaches to enhance agent performance focus on outcome rewards, which may lead to errors or suboptimal actions due to the absence of process supervision signals. |
| Approach: | They propose a step-level framework that provides detailed step-by-step guidance to enhance agent training by using Monte Carlo methods. |
| Outcome: | The proposed framework outperforms strong baselines on three tasks and shows that it is effective in augmenting efficiency and its applicability to diverse models. |
Copied to clipboard
| Challenge: | Existing methods to train LLMs suffer from overthinking, leading to lengthy reasoning traces . Existing approaches to train large language models suffer from this problem . |
| Approach: | They propose a method to combine multiple reasoning chains for training LLMs . they use stepwise exploration and long-short switched sampling to evaluate reasoning paths . |
| Outcome: | The proposed method reduces reasoning lengths by approximately 30-50% . it also maintains or improves reasoning accuracy compared to baselines . |
Copied to clipboard
| Challenge: | Task oriented dialog systems often rely on static exploration strategies that do not adapt to dynamic dialog contexts. |
| Approach: | They propose a dialog policy learning framework that formalizes the exploration challenge through a structured cognitive state space C. |
| Outcome: | The proposed framework achieves SOTA performance in success rate, efficiency, and generalization. |
Copied to clipboard
| Challenge: | Recent work has applied large language models (LLMs) into time series forecasting, but they lack an understanding of holistic temporal patterns with potential error accumulation. |
| Approach: | They propose a framework that marries Larg e Langu age Diffusion Model with time series forecasting (LEAF) they propose converting time series into tokens and adopting language diffusion models to capture temporal dependencies. |
| Outcome: | The proposed framework generates future predictions with a diffusion model from a holistic view. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can be used to broaden user experiences beyond established preferences and reinforce feedback loops. |
| Approach: | They propose a hierarchical approach that combines hierarchic planning with LLM inference-time scaling to improve recommendation relevancy without compromising novelty. |
| Outcome: | The proposed approach shows significant gains in both user satisfaction and exploration diversity. |
Copied to clipboard
| Challenge: | Existing methods for RL fail to establish an interpretable connection between data and optimization objectives. |
| Approach: | They propose a data selection method that dynamically estimates the influence of individual training samples on policy optimization. |
| Outcome: | The proposed method significantly improves training effectiveness with fewer optimization steps. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated strong performance in information retrieval tasks like passage ranking. |
| Approach: | They propose two attacks that aim to force the LLM ranker to prefer a specific passage and rank it at the top. |
| Outcome: | The proposed attacks aim to force the LLM ranker to prefer a specific passage and rank it at the top. |
Copied to clipboard
| Challenge: | a gap in math models' accuracy has been widened with the development of large language models (LLMs) . a new study aims to bridge this gap by evaluating a set of high-level math reasoning models . |
| Approach: | They propose to evaluate large language models on existing math benchmarks to bridge this gap . they collect 5,293 problems from Chinese senior high school mathematics exams . |
| Outcome: | The proposed model is based on o1-like models and a high-level model. |
Copied to clipboard
| Challenge: | Recent advances in parameter-efficient fine-tuning techniques allow for adjustments to only a minor fraction of the parameters of language models. |
| Approach: | They propose a low-rank direct attention adapted method for efficient LLM fine-tuning . they propose LMAM, which can bring negative attention to self-attention modules . |
| Outcome: | The proposed method outperforms the full fine-tuning method by 2.1% on GLUE benchmark. |
Copied to clipboard
| Challenge: | Existing reference-free preference optimization methods exhibit higher training efficiency but are prone to overoptimization, leading to performance degradation. |
| Approach: | They propose a reference-free preference optimization method that replaces the logsigmoid loss function with a SiLU function to improve the model's performance. |
| Outcome: | The proposed method achieves 7% improvement over SimPO on AlpacaEval 2 and MT-Bench. |
Copied to clipboard
| Challenge: | Goal-oriented script planning is used by humans to plan for typical activities . however, this capability remains underexplored due to several challenges . |
| Approach: | They propose a framework that enables product-enriched scripts by associating products with each step based on the semantic similarity between the actions and their purchase intentions. |
| Outcome: | The proposed framework can generate product-enriched scripts from 2.4 million scripts . human annotations are conducted to provide gold labels for a sampled subset . |
Copied to clipboard
| Challenge: | Recent studies have found that catastrophic forgetting arises from the model’s lack of robustness against future analogous relations. |
| Approach: | They propose a multi-task rationale tuning strategy to help the model learn current relations robustly and conduct contrastive rationale replay to further distinguish analogous relations. |
| Outcome: | The proposed method outperforms the state-of-the-art models on two benchmarks. |
Copied to clipboard
| Challenge: | Chart question answering (ChartQA) tasks are a critical part of visualization charts. |
| Approach: | They propose a chart question answering task that uses MLLMs to analyze charts . they propose 'Chain-of-Charts' textual prompt strategy that directs attention to visual elements . |
| Outcome: | The proposed model improves performance by 14.41% and 80% in low-level ChartQA tasks. |
Copied to clipboard
| Challenge: | LLM-based generative optimization has shown remarkable potential in improving agentic systems, but the current approach of prompting with the trajectories on the whole training dataset becomes untenable as datasets grow. |
| Approach: | They propose a scalable framework that divides large optimization tasks into manageable subsets and performs targeted optimizations. |
| Outcome: | The proposed framework outperforms conventional approach by 1.6-8.6% while reducing average prompt token consumption by 56.3%. |
Copied to clipboard
| Challenge: | Current evaluations of large language models (LLMs) focus on a single output per example, which limits our understanding of LLM performance variability in real-world applications. |
| Approach: | They explore the performance differences between greedy decoding and sampling and identify benchmarks’ consistency regarding non-determinism and examine unique model behaviors. |
| Outcome: | The proposed model outperforms sampling methods and greedy decoding outperformed other models. |
Copied to clipboard
| Challenge: | Existing video captioning models fail to capture nuanced semantics of videos . existing models generate coarse descriptions of human motions, resulting in poor quality . |
| Approach: | They construct a fine-grained human motion video captioning dataset named BoFiT and a model that generates fine-grain descriptions of human motions via prompting. |
| Outcome: | The proposed model outperforms existing models on comprehensive metrics. |
Copied to clipboard
| Challenge: | Existing work aims to improve reasoning accuracy and factual integrity across large language models for knowledge-intensive tasks such as medical and commonsense reasoning. |
| Approach: | They propose a versatile extension to the mutual reasoning framework (rStar) that enhances reasoning accuracy and factual integrity across large language models. |
| Outcome: | The proposed extension to the mutual reasoning framework improves reasoning accuracy and factual integrity across large language models for complex, knowledge-intensive tasks. |
Copied to clipboard
| Challenge: | Existing Large Language Model (LLM) enabled agents lack flexibility to respond to users’ varying needs and preferences. |
| Approach: | They propose a test-time user-preference alignment strategy that optimizes the persona prompt, ensuring real-time preference alignment through textual loss feedback between simulated and ground-truth responses. |
| Outcome: | The proposed framework outperforms baseline methods in real-time and in real applications. |
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) are often easily deceived by tricky questions such as “How many eyes does the sun have?” . |
| Approach: | They annotate a FalseQA dataset containing 2365 human-written FPQs and find that PLMs are capable of discriminating FPqs by fine-tuning on moderate numbers. |
| Outcome: | The proposed model can discriminate on FPQs by fine-tuning on moderate numbers of examples and generate reasonable explanations for false premise questions. |
Copied to clipboard
| Challenge: | Recent work on Chain-of-Thought prompting imposes substantial computational overhead . lack of supervision obscures the analyzability of the latent reasoning chain. |
| Approach: | They propose a framework to render latent reasoning chain into images, making latent rationale explicit and traceable. |
| Outcome: | The proposed framework achieves 3-4 token compression and substantial inference acceleration compared to explicit CoT prompting. |
Copied to clipboard
| Challenge: | Existing methods for evaluating expressive speech focus on word accuracy, naturalness, signal quality, or emotional intensity at the utterance level. |
| Approach: | They propose a framework for Evaluating Expressive Appropriateness in speech that assesses whether a speech sample aligns with the underlying communicative intent implied by its discourse-level narrative context. |
| Outcome: | The proposed framework outperforms existing speech evaluation and analysis systems on a human-annotated test set. |
Copied to clipboard
| Challenge: | In order to perform downstream tasks, Large Language Models (LLMs) need continual adaptation without catastrophic forgetting. |
| Approach: | They propose a new paradigm that allows for continual adaptation without catastrophic forgetting . they propose to replay previous data based on task similarity with instructions . |
| Outcome: | The proposed method improves performance over 16 tasks with different training orders. |
Copied to clipboard
| Challenge: | Existing models for machine reading comprehension are limited to individual chunks due to encoding length constraint. |
| Approach: | They propose a read-over-read method that expands the reading field from chunk to document by predicting regional answers for each chunk. |
| Outcome: | Extensive experiments on QuAC and TriviaQA show that the proposed model performs well for long document reading. |
Copied to clipboard
| Challenge: | Large language models extract useful information from conversation history to enhance the response in long-term conversations. |
| Approach: | They propose a Fragment-then-Compose framework to optimize memory utilization for long-term open-domain conversation. |
| Outcome: | The proposed framework can be used to extract useful information from conversation history . it can be adapted to different situations and improve response generation . |
Copied to clipboard
| Challenge: | Existing methods to predict logical forms ignore the utilization of symbolic operations and lack reasoning ability and interpretability. |
| Approach: | They propose an operation-pivoted discrete reasoning framework that uses symbolic operations as neural modules to facilitate reasoning ability and interpretability. |
| Outcome: | Extensive experiments on DROP and RACENum datasets show the reasoning ability of OPERA. |
Copied to clipboard
| Challenge: | Recent advances in large language models have enabled increasingly capable web agents . however, training such agents at scale still relies on high-quality interaction trajectories that are difficult to obtain at scale. |
| Approach: | They propose a framework for scalable trajectory synthesis that simulates state transitions without network dependencies and integrates Monte Carlo Tree Search to enable reversible exploration over the simulated state space. |
| Outcome: | Experiments on WebArena, WebVoyager, and Mind2Web-Online show that agents trained exclusively on synthesized trajectories outperform those trained on real-world data. |
Copied to clipboard
| Challenge: | Existing methods to reduce cognitive errors in MRI interpretations do not work for generating less likely outputs. |
| Approach: | They propose a task that asks a model to generate outputs that humans think are relevant but less likely to happen. |
| Outcome: | The proposed method compares with several state-of-the-art controlled text generation models via automatic and human evaluations and shows that it reduces cognitive errors in interpreting MRI findings. |
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) perform well on textual equations, but fail on visually grounded counterparts. |
| Approach: | They propose to decompose visual equation solving into symbolic equation solving and visual recognition into two core components to understand this gap. |
| Outcome: | The proposed models perform well on textual equations, but fail on visual grounded ones. |
Copied to clipboard
| Challenge: | Lightweight Vision-Language Models (VLMs) are indispensable for resource-constrained applications. |
| Approach: | They propose a framework that retrieves context from a memory bank to enhance alignment . they propose EMI-based approach to align vision and language models . |
| Outcome: | The proposed framework reduces training loss, accelerates convergence, and enhances task performance with negligible computational overhead. |
Copied to clipboard
| Challenge: | Existing autoregressive left-to-right (L2R) models are limited to unidirectional information and constrained on strong local dependencies. |
| Approach: | They propose a probabilistically permuted prophet language model which strengthens the modeling of bidirectional information and long token dependencies for sequence generation. |
| Outcome: | Experiments on GLGE dataset show that P3LM improves on natural language generation tasks. |
Copied to clipboard
| Challenge: | Existing methods for role-playing rely on prompt engineering, which lacks stability and interpretability. |
| Approach: | They propose a framework that extracts latent representations from role-play prompts and constructs a steering vector that can be injected into the model's residual stream with controllable intensity. |
| Outcome: | The proposed framework extracts latent representations from role-play prompts, selects the most relevant features based on activation patterns, and constructs a steering vector that can be injected into the model’s residual stream with controllable intensity. |
Copied to clipboard
| Challenge: | Existing methods exploit the utterances of all dialogue turns to assign value to slots . this can lead to suboptimal results due to information introduced from irrelevant utterrances . |
| Approach: | They propose a SLot-TUrN Alignment enhanced approach to assign slot value . they explicitly align each slot with its most relevant utterance and then predict the corresponding value based on this aligned utteration. |
| Outcome: | The proposed approach achieves state-of-the-art on three multi-domain task-oriented dialogue datasets. |
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models demonstrate strong performance on visual reasoning benchmarks. |
| Approach: | They propose a benchmark for vision-centric reasoning that integrates visual and textual information for non-trivial reasoning. |
| Outcome: | The proposed benchmark exposes gaps between humans and current MLLMs and reveals limited benefits from test-time reasoning strategies. |
Copied to clipboard
| Challenge: | Large language models (LMs) generate free-text rationales for their predictions via chain-of-thought prompting, but there is little guarantee that the generated rationale is consistent with LM’s predictions or faithfully justify the decisions. |
| Approach: | They propose a faithful knowledge distillation method to learn a small, self-consistent CoT model from a larger teacher model by contrastive decoding. |
| Outcome: | The proposed method yields comparable performance but is less faithful than baselines. |
Copied to clipboard
| Challenge: | Existing embedding models support only 512 input tokens, hindering their application in scenarios requiring long inputs. |
| Approach: | They evaluate the performance of existing embedding models by using a new benchmark and a training-free context window extension strategy. |
| Outcome: | The proposed model extends the input window of existing models by several folds. |
Copied to clipboard
| Challenge: | Experimental results show that visual instruction tuning improves performance of Multi-modal Large Language Models (MLLMs) to extend the application scope of Large Language Modells, a surge of work augments LLMs with vision encoders to endow the ability of multi-modal cognition and reasoning. |
| Approach: | They propose a systematic approach to create high-quality visual reasoning instructions using a synthesize-complicate-reformulate paradigm. |
| Outcome: | The proposed method improves performance of MLLMs by 27.86% and 27.60% on MME-Perception and MME Cognition. |
Copied to clipboard
| Challenge: | Existing methods for unlearning harmful, sensitive, or outdated knowledge suffer from two critical limitations: (1) collateral forgetting, where erasing target data inadvertently removes related but desirable knowledge, and (2) generality forgetting degrades the model’s general capabilities. |
| Approach: | They propose a method that identifies and leverages a targeted "unlearning direction" in the model's parameter space and selectively updates along this direction. |
| Outcome: | Experiments show that the proposed method achieves state-of-the-art unlearning precision while preserving both related knowledge and general capabilities. |
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are a promising tool for OR, but they face challenges when dealing with complex problems. |
| Approach: | They propose a framework that augments existing datasets and generates high-quality fine-tuning data tailored to OR. |
| Outcome: | The proposed framework augments existing datasets and generates high-quality fine-tuning data . it prevents error propagation and ensures the quality of the generated dataset . |
Copied to clipboard
| Challenge: | Existing concepts-based explainable approaches do not discover unseen concepts . a recent approach to solve this problem is concept-based explanations . |
| Approach: | They propose a framework that extracts comprehensible concepts automatically with no annotations . ECO-Concept uses an object-centric architecture to extract task-specific semantic concepts . |
| Outcome: | a new framework extracts comprehensible concepts with no concept annotations . the proposed framework outperforms existing methods in computability tests on diverse tasks . |
Copied to clipboard
| Challenge: | Large language models (LLMs) can solve complex multi-step math reasoning problems, but their internal implementation is limited. |
| Approach: | They propose to use a "C**ausal **E**ffect **D**riven **F**ine-tuning method" to improve LLMs' reasoning ability. |
| Outcome: | The proposed method improves the model's reasoning ability by enhancing key components that are used to execute mixed arithmetic calculations. |
Copied to clipboard
| Challenge: | Existing evaluations of multimodal language models focus on vocabulary words with relatively stable, context-independent meanings in conversation, such as object names, colors, and verbs. |
| Approach: | They compare human and multimodal language models in their use of three word types: vocabulary, possessives, and demonstratives. |
| Outcome: | The models approach human-level performance on using vocabulary, but exhibit clear deficits with possessives and even greater difficulties with demonstratives. |
Copied to clipboard
| Challenge: | Existing models fail to capture and model customer intention effectively because of insufficient information exploitation and only apparent information like descriptions and titles are used. |
| Approach: | They propose to exploit existing session data to capture and model intention in E-commerce product purchase sessions using a multimodal benchmark. |
| Outcome: | The proposed framework can bridge the gap between intention understanding in simplified research cases like co-buy intention and more complex yet practical scenarios like session history. |
Copied to clipboard
| Challenge: | Existing intention-based studies on recommendation tasks are limited and use models to implicitly model the intention memberships. |
| Approach: | They propose a framework that leverages the generation power of large language models and human-in-the-loop annotation to semi-automatically construct the intention knowledge graph. |
| Outcome: | The proposed framework can model e-commerce knowledge and have many potential applications. |
Copied to clipboard
| Challenge: | Incorporating external context can enhance the response quality of Large Language Models (LLMs). however, real-world contexts often mix relevant information with disproportionate inappropriate content. |
| Approach: | They propose a Poisoned Context Testbed to pair queries with real-world contexts . they propose 'rw-Steering' to internalize inappropriate signals . |
| Outcome: | The proposed model improves response quality by 39.8% and reverses undesirable behavior curve. |
Copied to clipboard
| Challenge: | Existing methods for evaluating the perceptual quality of synthetic speech are limited due to the complexity of perceptual quality factors and the diversity of speech generation tasks. |
| Approach: | They propose a new paradigm for enabling large language models to conduct structured speech quality evaluation using a large-scale dataset. |
| Outcome: | The proposed model performs well across tasks and languages. |
Copied to clipboard
| Challenge: | Document Visual Question Answering (DocVQA) is a task to answer questions based on documents containing text, tables, and images. |
| Approach: | They propose a lightweight retrieval framework that uses visual language models to embed and retrieve relevant pages as images and generate answers with VLMs that can accept an image as input. |
| Outcome: | The proposed framework outperforms baselines by 3.2% on average on 4 DocVQA datasets with much fewer pages retrieved. |