Papers by Wei Xue
Omni-RewardBench: Toward a Comprehensive Evaluation of Generative Reward Models Across Modalities (2026.acl-long)
Copied to clipboard
Chi-Min Chan, Yujin Zhou, Pengcheng Wen, Boqin Yin, Jiaming Ji, Juntao Dai, Wei Xue, Sirui Han, Yike Guo
| Challenge: | Existing evaluation benchmarks for ORMs are largely text-centric or limited to bimodal tasks . a new study examines the effectiveness of Omni-RewardBench for ORms across modalities . |
| Approach: | They propose a hybrid automatic-annotation and human-verification pipeline to construct high-quality evaluation data. |
| Outcome: | The proposed model is the first benchmark for comprehensive evaluation of ORMs across modalities. |
CoCA: Fusing Position Embedding with Collinear Constrained Attention in Transformers for Long Context Window Extending (2024.acl-long)
Copied to clipboard
| Challenge: | Existing models that use self-attention and position embedding have anomalous behavior that hinder long context window extrapolation. |
| Approach: | They propose a collinear constraint between Q and K to integrate RoPE and self-attention. |
| Outcome: | The proposed model integrates self-attention and position embedding into LLMs without fine-tuning. |
Beyond Token Length: Step Pruner for Efficient and Accurate Reasoning in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing reinforcement learning methods for large reasoning models suffer from excessive verbosity, known as "overthinking." Existing models penalize generated tokens to promote conciseness, but these methods encounter two challenges: they may develop hacking behavior in later stages of training by discarding reasoning steps. |
| Approach: | They propose a framework that steers large reasoning models toward more efficient reasoning . they prioritize correctness while imposing penalties for redundant steps . |
| Outcome: | The proposed framework reduces token usage by 69.7% on AIME24. |
SafeSteer: A Decoding-level Defense Mechanism for Multimodal Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing defense methods rely on fine-tuning or inefficient post-hoc interventions, limiting their ability to address novel attacks. |
| Approach: | They propose a decoding-level defense mechanism that employs a lightweight discriminator to iteratively steer the decoding process toward safety. |
| Outcome: | The proposed method improves safety performance by up to 33.40% without fine-tuning on multiple MLLMs. |
Multimodal Dual-Path Decoding for Medical Report Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Current methods for radiology report generation rely on encoder-decoder based frameworks that fail to integrate multimodal clinical evidence with domain-specific knowledge. |
| Approach: | They propose a multimodal dual-path framework that synergistically integrates large vision-language models and large language models for radiology report generation. |
| Outcome: | The proposed framework improves on the public MIMIC-CXR benchmark and shows that it is superior to state-of-the-art models. |
Enhancing Emotion Recognition in Conversation via Multi-view Feature Alignment and Memorization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Emotion recognition in conversation (ERC) is an advanced capability of conversational AI systems. |
| Approach: | They propose a semi-parametric paradigm for Emotion Recognition in conversation that uses supervised contrastive learning to align semantic-view and context-view features. |
| Outcome: | The proposed model achieves state-of-the-art on four widely used benchmarks. |
FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to correct factually inaccurate outputs are lacking the semantic richness needed to properly understand its internal states of trustworthiness and honesty. |
| Approach: | They propose a framework for factuality alignment that integrates natural-language uncertainty signals with external knowledge and computes confidence scores and semantic entropy from LLM outputs. |
| Outcome: | Extensive experiments on four knowledge-intensive benchmarks show that FAITH improves the factual accuracy and truthfulness of Large Language Models (LLMs). |
ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback (2026.findings-acl)
Copied to clipboard
| Challenge: | Unlike chatbots, autonomous agents act directly on external environments, making tool invocation safety critical for reliable deployment. |
| Approach: | They develop a benchmark for step-level tool invocation safety detection in LLM agents and a guardrail model that proactively detects unsafe tool invoking actions before execution using multi-task reinforcement learning. |
| Outcome: | The proposed model reduces harmful tool invocations of ReAct-style agents by 65% on average and improves benign task completion by 10% under prompt injection attacks. |
TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for incorporating external knowledge into language models do not prioritize learning embeddings for entity-related tokens. |
| Approach: | They propose a framework for incorporating external knowledge into pre-training models that utilize entity-related tokens. |
| Outcome: | The proposed framework reduces pre-training time by 50% and outperforms other KEPLMs in knowledge probing tasks and multiple knowledge-aware language understanding tasks. |
Insert or Attach: Taxonomy Completion via Box Embedding (2024.acl-long)
Copied to clipboard
| Challenge: | Existing taxonomy expansion methods embed concepts as vectors in Euclidean space, causing incorrectly model asymmetric relations. |
| Approach: | They propose to use box containment and center closeness to create geometric scorers that capture intrinsic relationships between concepts. |
| Outcome: | The proposed framework outperforms existing methods on four real-world datasets. |
UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion (2024.acl-long)
Copied to clipboard
| Challenge: | Existing text-to-image diffusion models generate images from text prompts due to inherent brevity of textual descriptions . however, the ability to accurately synthesize images with intricate details, such as specific entities or scenes, is limited due to the inherent bribery of text descriptions. |
| Approach: | They propose a multimodal conditional diffusion framework that operates on multimodal prompts with interleaved textual and visual inputs. |
| Outcome: | The proposed framework excels in both text-to-image generation and zero-shot subject-driven synthesis. |
DREAM: Improving Video-Text Retrieval Through Relevance-Based Augmentation Using Large Foundation Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in video-text retrieval models have limited training data annotations. |
| Approach: | They propose a Video-Text Retrieval Paradigm with Relevance-based Augmentation which enhances video and text data using large foundation models to learn more generalized features. |
| Outcome: | The proposed method improves video-text retrieval performance over existing methods. |
FlashAudio: Rectified Flow for Fast and High-Fidelity Text-to-Audio Generation (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in latent diffusion models (LDMs) have markedly enhanced text-to-audio generation, yet their iterative sampling processes impose substantial computational demands, limiting practical deployment. |
| Approach: | They propose to learn straight flow for fast simulation by using flashAudio with rectified flows and immiscible flow to minimize the total distance of data-noise pairs in a batch vias assignment. |
| Outcome: | The proposed method can learn straight flow for fast simulations and reduce noise distribution. |
Aspect Based Sentiment Analysis with Gated Convolutional Networks (P18-1)
Copied to clipboard
| Challenge: | Aspect-based sentiment analysis can provide more detailed information than general sentiment analysis. |
| Approach: | They propose a model based on convolutional neural networks and gating mechanisms which can selectively output the sentiment features according to the given aspect or entity. |
| Outcome: | The proposed model can selectively output sentiment features according to the given aspect or entity. |
Boosting Policy and Process Reward Models with Monte Carlo Tree Search in Open-Domain QA (2025.findings-acl)
Copied to clipboard
Chi-Min Chan, Chunpu Xu, Junqi Zhu, Jiaming Ji, Donghai Hong, Pengcheng Wen, Chunyang Jiang, Zhen Ye, Yaodong Yang, Wei Xue, Sirui Han, Yike Guo
| Challenge: | Experimental results show that our approach can effectively improve the performance of both the policy model and the reward model. |
| Approach: | They propose to use Monte Carlo Tree Search for both policy model improvement and reward model improvement to bridge it to more subtle open-domain question answering. |
| Outcome: | The proposed approach surpasses existing methods for annotation and training data with fewer data points and achieves better performance in test-time scaling strategies. |
ChatMusician: Understanding and Generating Music Intrinsically with LLM (2024.findings-acl)
Copied to clipboard
Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, Liumeng Xue, Ziyang Ma, Qin Liu, Tianyu Zheng, Yizhi Li, Yinghao Ma, Yiming Liang, Xiaowei Chi, Ruibo Liu, Zili Wang, Chenghua Lin, Qifeng Liu, Tao Jiang, Wenhao Huang, Wenhu Chen, Jie Fu, Emmanouil Benetos, Gus Xia, Roger Dannenberg, Wei Xue, Shiyin Kang, Yike Guo
| Challenge: | Despite LLMs' impressive capabilities in musical knowledge, music reasoning remains an unsolved task. |
| Approach: | They propose an open-source large language model (LLM) that integrates intrinsic musical abilities into LLaMA2 and GPT-3.5. |
| Outcome: | The proposed model can understand and generate music with a pure text tokenizer without external multi-modal neural structures or tokenizers. |
Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Prompts (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Continual pre-training has been used for a multitude of domains and tasks . a continually pre-trained model can show a non-decreasing performance on unseen domains . |
| Approach: | They propose a method that generates domain-specific prompts by agreement and disagreement losses. |
| Outcome: | The proposed method achieves improvements of 3.57% and 3.4% on two real-world datasets. |
Benchmarking Fine-Grained Error Detection in Multimodal Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Multimodal Process Reward Models (MPRMs) have emerged as a pivotal framework for enhancing the reasoning capabilities of Multimodal Large Language Models. |
| Approach: | They propose a benchmark specifically designed to evaluate MPRMs’ proficiency in detecting erroneous reasoning steps across diverse error categories. |
| Outcome: | The proposed model achieves up to 4.8% performance improvement through test-time scaling. |
How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2024.acl-long)
Copied to clipboard
Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, Jingren Zhou
| Challenge: | supervised fine-tuning (SFT) is a technique used to enhance multiple abilities in large language models. |
| Approach: | They propose to study the interplay of data composition between mathematical reasoning, code generation, and general human-aligning abilities during supervised fine-tuning. |
| Outcome: | The proposed model improves math reasoning and code generation with increasing data amount . the proposed model size and SFT strategies can be used to learn multiple skills with different scaling patterns. |
ActorMind: Emulating Human Actor Reasoning for Speech Role-Playing (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing work on role-playing focuses on textual modalities, neglecting speech . et al., 2025) show that speech role-players can generate spontaneous responses with personalized traits based on the context. |
| Approach: | They propose a framework that allows models to deliver spontaneous responses with personalized verbal traits based on their role, scene, and spoken dialogue. |
| Outcome: | The proposed framework enhances speech role-playing by generating spontaneous responses with personalized traits based on their role, scene, and spoken dialogue. |
Enhancing Air Quality Prediction with Social Media and Natural Language Processing (P19-1)
Copied to clipboard
| Challenge: | predicting air quality is a major concern for human health, but the changes of air quality conditions are still difficult to monitor. |
| Approach: | They propose to exploit social media and natural language processing techniques to enhance air quality prediction. |
| Outcome: | The proposed approach improves air quality prediction over baseline that does not use social media by 6.9% to 17.7% in macro-F1 scores. |
It’s Not a Walk in the Park! Challenges of Idiom Translation in Speech-to-text Systems (2025.acl-long)
Copied to clipboard
| Challenge: | idioms are defined as words with a figurative meaning not deducible from their individual components. |
| Approach: | They compare idiom translation as compared to conventional news translation in two languages . they compare MT and SLT systems with MT, Large Language Models and cascaded alternatives . |
| Outcome: | The proposed systems show better handling of idioms than standard news translation systems. |
PyramidCodec: Hierarchical Codec for Long-form Music Generation in Audio Domain (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to generate long music are inefficient and lack of structured representation. |
| Approach: | They propose a hierarchical discrete representation of audio for long audio-domain music generation using residual vector quantization on different levels of features. |
| Outcome: | The proposed method achieves competitive performance in terms of reconstruction quality and token per second (TPS) the proposed method facilitates training a language model that can generate well-structured long-form music for up to 3 minutes. |
LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement (2025.acl-long)
Copied to clipboard
Boyi Kang, Xinfa Zhu, Zihan Zhang, Zhen Ye, Mingshuai Liu, Ziqian Wang, Yike Zhu, Guobin Ma, Jun Chen, Longshuai Xiao, Chao Weng, Wei Xue, Lei Xie
| Challenge: | Recent advances in language models have demonstrated strong capabilities in semantic understanding and contextual modeling. |
| Approach: | They propose a LLaMA-based language model that incentivizes generalization capabilities for speech enhancement. |
| Outcome: | The proposed language model outperforms prior task-specific discriminative and generative models in acoustic enhancement tasks. |
metaCAT: A Metadata-based Task-oriented Chatbot Annotation Tool (2020.aacl-demo)
Copied to clipboard
| Challenge: | Creating high-quality annotated dialogue corpora necessitates a high level of human engagements. |
| Approach: | They propose to develop an annotation tool specifically for developing task-oriented dialogue data that provides comprehensive metadata annotation coverage to the domain, intent, and span information. |
| Outcome: | The tool provides comprehensive metadata annotation coverage to domain, intent, and span information. |
PASUM: A Pre-training Architecture for Social Media User Modeling Based on Text Graph (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have incorporated different digital traces to better learn the representations of social media users, limited by overloaded text information and hard-to-collect social network information. |
| Approach: | They propose a Pre-training Architecture for Social Media User Modeling based on Text Graph and combine microblogs to represent social media users based upon the text graph model. |
| Outcome: | The proposed framework can represent users based on text even without social network information on microblogs. |
AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists (2025.emnlp-main)
Copied to clipboard
Yifei Li, Hanane Nour Moussa, Ziru Chen, Shijie Chen, Botao Yu, Mingyi Xue, Benjamin Burns, Tzu-Yao Chiu, Vishal Dey, Zitong Lu, Chen Wei, Qianheng Zhang, Tianyu Zhang, Song Gao, Xuhui Huang, Xia Ning, Nesreen K. Ahmed, Ali Payani, Huan Sun
| Challenge: | AutoSDT-5K is the only automatically collected and the largest open dataset for data-driven scientific discovery. |
| Approach: | They propose an automatic pipeline that collects high-quality coding tasks in real-world data-driven discovery workflows. |
| Outcome: | The proposed pipeline synthesizes accurate tasks and tasks from a dataset of 5,404 tasks covering four scientific disciplines and 756 Python packages. |
DiffER: Diffusion Entity-Relation Modeling for Reversal Curse in Diffusion Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models exhibit unidirectional behavior when processing bidirectional relationships . authors propose a solution to alleviate the reversal curse in Diffusion LLMs . |
| Approach: | They propose a model that addresses the "reversal curse" of bidirectional behavior in large language models . they propose 'entity-aware training' and balanced data construction to alleviate asymmetry and missing relations . |
| Outcome: | The proposed model alleviates the "reversal curse" in Diffusion LLMs . the proposed model employs whole-entity masking to mitigate entity fragmentation . |
BayesKD: Bayesian Knowledge Distillation for Compact LLMs in Constrained Fine-tuning Scenarios (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized various domains with their remarkable capabilities, but their massive parameter sizes pose significant challenges for fine-tuning and inference. |
| Approach: | They propose a Bayesian Knowledge Distillation framework for compact Large Language Models in resource-constrained fine-tuning scenarios that employs Logits Dual-Scaling, Knowledge Alignment Module, and Bayes Distillations Optimization. |
| Outcome: | The proposed framework outperforms baseline methods on various state-of-the-art LLMs, including LLaMA, Qwen2, Bloom, and Vicuna. |
MeNTi: Bridging Medical Calculator and LLM Agent with Nested Tool Calling (2025.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been widely used in medicine but are limited in their ability to fully address the complexities of the real world. |
| Approach: | They propose a universal agent architecture for Large Language Models that integrates a specialized medical toolkit and employs meta-tool and nested calling mechanisms to enhance LLM tool utilization. |
| Outcome: | The proposed framework improves the accuracy and performance of medical calculators in complex medical scenarios. |
Response-Anticipated Memory for On-Demand Knowledge Integration in Response Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Neural conversation models generate appropriate but non-informative responses in general. |
| Approach: | They propose to construct a document memory with anticipated responses in mind using a teacher-student framework and a student's input. |
| Outcome: | The proposed model outperforms the state-of-the-art for the Conversing by Reading task. |
TagRouter: Learning Route to LLMs through Tags for Open-Domain Text Generation Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models with limited performance and limited training can be difficult to use in large-scale applications. |
| Approach: | They propose a training-free model routing method that optimizes synergy among multiple LLMs for open-domain text generation tasks. |
| Outcome: | The proposed method outperforms 13 baseline models and reduces costs by 17.20%. |
Graceful Forgetting in Generative Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained models do not provide all knowledge needed for fine-tuning tasks. |
| Approach: | They propose a framework to achieve graceful forgetting in generative language models by pre-training a model on large-scale correlating datasets. |
| Outcome: | The proposed framework improves the learning plasticity of the target task by selectively discarding irrelevant knowledge. |