Papers by Tao Jin

Generative-to-Discriminative Test-Time Adaptation via Manifold-Aware Diffusion and Bayesian Distillation (2026.findings-acl)

Challenge:	Speech Recognition often gets stuck in the lack of new domain utterances when training a model of new-domain speech.
Approach:	They propose a training system Open-modality Speech Recognition that enables zero-shot modality transfer . they use multi-modal alignment in phoneme space to maintain multi-modality alignment .
Outcome:	The proposed system achieves zero-shot modality transfer compared to existing methods . it achieves state-of-the-art performance on audio-visual speech recognition and lip-reading with 2.7% and 25.0%, respectively.

Copied to clipboard

Boyun Zhang, Zequn Xie, Fangming Feng, Zihan Zhang, Yongbo He, Chuxin Wang, Sihang Cai, Tao Jin, Qifei Zhang

Semantic-conditioned Dual Adaptation for Cross-domain Query-based Visual Segmentation (2023.findings-acl)

Challenge:	Existing discriminative approaches suffer from "confident but wrong" failure mode, blindly adapting to OOD noise leading to error accumulation.
Approach:	They propose a TTA framework that harmonizes the robustness of generative diffusion models with the efficiency of discriminative regression networks via Bayesian Diffusion Distillation (BDD).
Outcome:	The proposed framework reduces MAE from 0.6872 to 0.5673 and boosts binary accuracy by 5.81 percentage points (reaching 57.33%) it also reduces the MAE of the MOSI to SIMS shift and achieves an 11.18-point gain over the baseline.

Copied to clipboard

Ye Wang, Tao Jin, Wang Lin, Xize Cheng, Linjun Li, Zhou Zhao

Visual Data in NLP

Challenge:	Existing approaches to visual segmentation from language queries require expensive labeling and degradation when deployed to an unseen domain.
Approach:	They propose a task to adapt a visual segmentation model from a labeled domain to an unseen domain.
Outcome:	The proposed framework achieves precise feature- and relation-invariant across domains via universal semantic structure.

Omni-Chart-600K: A Comprehensive Dataset of Chart Types for Chart Understanding (2025.findings-naacl)

Copied to clipboard

Shulei Wang, Shuai Yang, Wang Lin, Zirun Guo, Sihang Cai, Hai Huang, Ye Wang, Jingyuan Chen, Tao Jin

TAVT: Towards Transferable Audio-Visual Text Generation (2023.acl-long)

Challenge:	Existing chart-related training methods lack capabilities in information extraction, mathematical reasoning, and understanding of multiple chart types.
Approach:	They propose a two-stage training strategy and method for jointly training a vision encoder tailored for multi-type charts to address the deficiencies in chart types and limited scope of chart tasks in existing datasets.
Outcome:	The proposed dataset includes 21 diverse chart types and tasks, including data retrieval and mathematical reasoning.

Copied to clipboard

Wang Lin, Tao Jin, Wenwen Pan, Linjun Li, Xize Cheng, Ye Wang, Zhou Zhao

Improving Discriminative Capability of Reward Models in RLHF Using Contrastive Learning (2024.emnlp-main)

Challenge:	Existing transfer learning techniques focus on uni-modal analysis and lack consideration of multi-modal content and cross-modal relation.
Approach:	They propose a transferable audio-visual text generation framework that incorporates two components: Audio-Visual Meta-Mapper and Dual Counterfactual Contrastive Learning.
Outcome:	The proposed framework outperforms the state-of-the-art methods across multiple domains and modal settings.

Copied to clipboard

Lu Chen, Rui Zheng, Binghai Wang, Senjie Jin, Caishuang Huang, Junjie Ye, Zhihao Zhang, Yuhao Zhou, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang

Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models (2026.findings-acl)

Challenge:	Current methods rely on ranking losses to teach reward model to assess preferences, but they are susceptible to noise and ambiguous data, often failing to deeply understand human intentions.
Approach:	They propose a method that incorporates contrastive learning into the reward modeling process to enhance generalization and stabilize the reinforcement learning training process.
Outcome:	The proposed method enhances generalization of the reward model, stabilizes the reinforcement learning training process, and improves the final alignment with human preferences.

Copied to clipboard

Hengyuan Zhang, Zhihao Zhang, Ercong Nie, Mingyang Wang, Zunhai Su, Yiwei Wang, Qianli Wang, Shuzhou Yuan, Xufeng Duan, Qibo Xue, Zeping Yu, Chenming Shang, Xiao Liang, Jing Xiong, Hui Shen, Chaofan Tao, Zhengwu Liu, Senjie Jin, Zhiheng Xi, Dongdong Zhang, Sophia Ananiadou, Tao Gui, Ruobing Xie, Hayden Kwok-Hay So, Hinrich Schuetze, Xuanjing Huang, Qi Zhang, Ngai Wong

Explainability & Interpretability in NLP

Challenge:	Existing literature on mechanistic interpretation (MI) treats it as an observational science, leaving practical applications underexplored.
Approach:	They propose a survey structured around the pipeline to identify and improve MI models.
Outcome:	The proposed framework enables tangible improvements in Alignment, Capability, and Efficiency.

Bridging the Pose-Semantic Gap: A Cascade Framework for Text-Based Person Anomaly Search (2026.findings-acl)

Copied to clipboard

Zequn Xie, Guijin Luo, Chuxin Wang, Sihang Cai, Tao Jin, Zhou Zhao, Yixuan Tang

Information Retrieval

Challenge:	Existing methods for person anomaly search fail to address the complexities of real-world security, authors say . Existing approaches fail to detect subtle semantic distinctions, authors argue .
Approach:	They propose a framework that decouples retrieval into two stages . structure-aware coarse retrieval and detective squad interaction are proposed .
Outcome:	The proposed framework achieves state-of-the-art performance by balancing efficiency and semantic reasoning.

UniLG: A Unified Structure-aware Framework for Lyrics Generation (2023.acl-long)

Copied to clipboard

Tao Qian, Fan Lou, Jiatong Shi, Yuning Wu, Shuai Guo, Xiang Yin, Qin Jin

Weakly-Supervised Spoken Video Grounding via Semantic Interaction Learning (2023.acl-long)

Challenge:	Existing works ignore musical attributes hidden behind lyrics and structure of lyrics . existing works ignore structure of generated lyrics and do not consider structure of songs .
Approach:	They propose a framework for conditional lyrics generation that considers structure and relationship between lyrics and music.
Outcome:	The proposed framework improves the structure modeling and unifies different conditions for different types of lyrics generation.

Copied to clipboard

Ye Wang, Wang Lin, Shengyu Zhang, Tao Jin, Linjun Li, Xize Cheng, Zhou Zhao

Tailoring Diagnostic Modeling to Individual Learners: Personalized Distractor Generation via MCTS-Guided Reasoning Reconstruction (2026.acl-long)

Challenge:	Recent work on spoken video grounding challenges extracting semantic information from speech . previous studies focused on textual queries, but recent work focuses on spoken queries .
Approach:	They propose a framework for weakly-supervised spoken video grounding to represent cross-modal semantics without expensive temporal annotations.
Outcome:	The proposed framework is more efficient than existing methods.

Copied to clipboard

Tao Wu, Jingyuan Chen, Wang Lin, Jian Zhan, Mengze Li, Fangzhou Jin, Min Zhang, Kun Kuang, Fei Wu

Natural Language Interfaces

Challenge:	Current distractor generation methods produce shared distractors for all students, ignoring individual variations in reasoning, which limits their diagnostic effectiveness.
Approach:	They propose a method which tailors distractors to each student’s specific cognitive flaws, inferred from their past question-answering (QA) history.
Outcome:	The proposed framework outperforms existing methods in generating plausible distractors and adapts to group-level settings.

MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing (2026.acl-industry)

Copied to clipboard

Junbo Niu, Zheng Liu, Zhuangcheng Gu, Bin Wang, Linke Ouyang, Zhiyuan Zhao, Tao Chu, Tianyao He, Fan Wu, Qintong Zhang, Zhenjiang Jin, Guang Liang, Rui Zhang, Wenzheng Zhang, Yuan Qu, Zhifei Ren, Yuefeng Sun, Zirui Tang, Boyu Niu, Yuanhong Zheng, Dongsheng Ma, Ziyang Miao, Hejun Dong, Siyi Qian, Junyuan Zhang, Fangdong Wang, Jingzhou Chen, Xiaomeng Zhao, Liqun Wei, Wei Li, Shasha Wang, RuiLiang Xu, Yuanyuan Cao, Lu Chen, Qianqian Wu, Huaiyu Gu, Lindong Lu, Dechen Lin, null Shenguanlin, Xuanhe Zhou, Linfeng Zhang, Yuhang Zang, Xiaoyi Dong, Jiaqi Wang, Bo Zhang, Lei Bai, Pei Chu, Weijia Li, Jiang Wu, Lijun Wu, Zhenxiang Li, Guangyu Wang, Zhongying Tu, Chao Xu, Kai Chen, Bowen Zhou, Dahua Lin, Wentao Zhang, Conghui He

Green & Sustainable NLP

Challenge:	Document images are characterized by higher resolutions, denser content, and more complex structural layouts.
Approach:	They propose a 1.2B-parameter document parsing vision-language model that decouples layout analysis from local content recognition.
Outcome:	The proposed model surpasses general-purpose and domain-specific models on multiple benchmarks while maintaining significantly lower computational overhead.

Decomposition for Enhancing Attention: Improving LLM-based Text-to-SQL through Workflow Paradigm (2024.findings-acl)

Copied to clipboard

Yuanzhen Xie, Xinzhou Jin, Tao Xie, Matrixmxlin Matrixmxlin, Liang Chen, Chenyun Yu, Cheng Lei, Chengxiang Zhuo, Bo Hu, Zang Li

Programming Languages in NLP

Challenge:	In-context learning of large-language models has achieved remarkable success in the field of natural language processing . however, the single-step chain-of-thought prompting approach faces challenges such as attention diffusion and inadequate performance in complex tasks like text-to-SQL.
Approach:	They propose a workflow paradigm method to enhance the attention and problem-solving scope of large-language models through decomposition.
Outcome:	The proposed method outperforms existing methods on three datasets and improves the upper limit of LLM-based approaches.

Two-Stage Regularization-Based Structured Pruning for LLMs (2026.acl-long)

Copied to clipboard

Mingkuan Feng, Jinyang Wu, Siyuan Liu, Shuai Zhang, Hongjian Fang, Ruihan Jin, Feihu Che, Pengpeng Shao, Zhengqi Wen, Jianhua Tao

MEEL: Multi-Modal Event Evolution Learning (2024.findings-acl)

Challenge:	Structural pruning is a promising solution for large language models . prior structured pruning methods remove unimportant parameters based on certain metrics .
Approach:	They propose a structural pruning method that iteratively learns the weights of transformer layers by adding their l1-norm to the loss function.
Outcome:	The proposed pruning method outperforms strong layer-wise pruning methods without requiring retraining.

Copied to clipboard

Zhengwei Tao, Zhi Jin, Junqiang Huang, Xiancai Chen, Xiaoying Bai, Yifan Zhang, Chongyang Tao

A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition (2023.findings-acl)

Challenge:	Existing models fail to grasp the principles governing event evolution in various scenarios.
Approach:	They propose a multi-modal event evolution learning approach to grasp event evolution . they propose an instruction encapsulation process that transforms evolving graphs into instruction-tuning data .
Outcome:	The proposed model grasps the event evolution mechanism yielding advanced MMER ability.

Copied to clipboard

Limao Xiong, Jie Zhou, Qunxi Zhu, Xiao Wang, Yuanbin Wu, Qi Zhang, Tao Gui, Xuanjing Huang, Jin Ma, Ying Shan

Information Extraction & Text Mining

Challenge:	Existing models for named entity recognition (NER) are based on large-scale labeled datasets, which always obtain using crowdsourcing.
Approach:	They propose a CONfidence-based partial Label Learning method to integrate prior and posterior confidences for crowd-annotated named entity recognition models.
Outcome:	The proposed model improves on real-world and synthetic datasets compared with baselines.

AudioVSR: Enhancing Video Speech Recognition with Audio Data (2024.emnlp-main)

Copied to clipboard

Xiaoda Yang, Xize Cheng, Jiaqi Duan, Hongshun Qiu, Minjie Hong, Minghui Fang, Shengpeng Ji, Jialong Zuo, Zhiqing Hong, Zhimeng Zhang, Tao Jin

Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement (2023.findings-emnlp)

Challenge:	Recent work has shown poor performance with non-Indo-European languages . previous work primarily utilizes video information to build VSR models .
Approach:	They propose a generative model for data inflation that integrates synthetic data with authentic visual data to enhance the VSR model.
Outcome:	The proposed model improves on the audio-visual alignment problem in audio-video tasks.

Copied to clipboard

Zhiheng Xi, Senjie Jin, Yuhao Zhou, Rui Zheng, Songyang Gao, Jia Liu, Tao Gui, Qi Zhang, Xuanjing Huang

Low-Rank HOCA: Efficient High-Order Cross-Modal Attention for Video Captioning (D19-1)

Challenge:	Existing prompting methods have been used to enhance multistep reasoning capabilities of large language models, but they have overlooked the potential of formulating higher-quality problems.
Approach:	They propose a method that starts from the problem side and refines problems to be more comprehensible and solvable for models.
Outcome:	The proposed method achieves notable and consistent effectiveness on five reasoning benchmarks across different models.

Copied to clipboard

Tao Jin, Siyu Huang, Yingming Li, Zhongfei Zhang

Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition (2024.acl-long)

Challenge:	Existing studies on video captioning focus on the association relationships between multiple modalities.
Approach:	They propose a video captioning model with high-order cross-modal attention (HOCA) they propose low-rank HOCA which adopts tensor decomposition to reduce the space requirement .
Outcome:	The proposed model captures cross-modal interaction of different modalities and reduces space requirement.

Copied to clipboard

Zirun Guo, Tao Jin, Zhou Zhao

Text-Guided Multi-Scale Frequency Representation Adaptation (2026.acl-long)

Challenge:	Existing methods for multimodal sentiment analysis often fail due to equipment failure, data corruption, privacy issues and the like.
Approach:	They propose a multimodal Transformer framework using prompt learning to address the issue of missing modalities.
Outcome:	The proposed framework outperforms existing methods significantly across evaluation metrics.

Copied to clipboard

Weicai Yan, Xinhua Ma, Wang Lin, Tao Jin

EVIT: Event-Oriented Instruction Tuning for Event Reasoning (2024.findings-acl)

Challenge:	Existing methods for fine-tuning visual signals are limited by their size and complexity.
Approach:	They propose a multi-scale frequency-based fine-tuning method that integrates textual information and performs multi-level fine- tuning of visual signals in the frequency domain.
Outcome:	Extensive experiments on multimodal models, including CLIP and LLaVA, demonstrate that the proposed method significantly improves performance and efficiency with minimal cost and fast convergence within one epoch.

Copied to clipboard

Zhengwei Tao, Xiancai Chen, Zhi Jin, Xiaoying Bai, Haiyan Zhao, Yiwei Lou

GS-Quant: Granular Semantic and Generative Structural Quantization for Knowledge Graph Completion (2026.acl-long)

Challenge:	Large language models (LLMs) have made significant advances in event reasoning . however, smaller instruction-tuned models do not consistently demonstrate exceptional proficiency .
Approach:	They propose an event-oriented instruction tuning technique to train a large language model . they propose a structure named event quadruple which contains the structure and semantics of events .
Outcome:	The proposed model achieves competitive performances on event reasoning tasks.

Copied to clipboard

Qizhuo Xie, Yunhui Liu, Yu Xing, Qianzi Hou, Xudong Jin, Tao Zheng, Tieke He

FastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognition (2021.findings-emnlp)

Challenge:	Existing quantization-based approaches to knowledge Graph Completion (KGC) are incomplete.
Approach:	They propose a framework that generates semantically coherent discrete codes for KG entities . they introduce a Granular Semantic Enhancement module that injects hierarchical knowledge into the codebook .
Outcome:	The proposed framework outperforms existing text-based and embedding-based baselines in the KGC domain.

Copied to clipboard

Yichong Leng, Xu Tan, Rui Wang, Linchen Zhu, Jin Xu, Wenjie Liu, Linquan Liu, Xiang-Yang Li, Tao Qin, Edward Lin, Tie-Yan Liu

Find-the-Common: A Benchmark for Explaining Visual Patterns from Images (2024.lrec-main)

Challenge:	Error correction is widely used in automatic speech recognition (ASR) to post-process the generated sentence.
Approach:	They propose a fast correction model that takes multiple ASR candidates as input for better correction accuracy.
Outcome:	The proposed model can reduce the word error rate (WER) with multiple candidates by 3.2% and 2.6%.

Copied to clipboard

Yuting Shi, Naoya Inoue, Houjing Wei, Yufeng Zhao, Tao Jin

T2A-Feedback: Improving Basic Capabilities of Text-to-Audio Generation via Fine-grained AI Feedback (2025.acl-long)

Challenge:	Recent advances in Instruction-fine-tuned Vision and Language Models (IVLMs) have prompted some studies to analyze the reasoning capabilities of IVLMs.
Approach:	They introduce a vision and language task for Inductive Visual Reasoning that uses common attributes across visual scenes to find common answers.
Outcome:	The proposed model can archive with 48% accuracy on the FTC, compared with state-of-the-art models.

Copied to clipboard

Zehan Wang, Ke Lei, Chen Zhu, Jiawei Huang, Sashuai Zhou, Luping Liu, Xize Cheng, Shengpeng Ji, Zhenhui Ye, Tao Jin, Zhou Zhao

RadialRouter: Structured Representation for Efficient and Robust Large Language Models Routing (2025.findings-emnlp)

Challenge:	Text-to-audio (T2A) models still struggle to satisfy human preferences for prompt-following and acoustic quality when generating complex multi-event audio.
Approach:	They propose to use AI feedback learning to enhance basic capabilities of text-to-audio models . they use a large audio preference dataset to evaluate the model's capabilities .
Outcome:	The proposed model improves in simple and complex scenarios with AI feedback learning.

Copied to clipboard

Ruihan Jin, Pengpeng Shao, Zhengqi Wen, Jinyang Wu, Mingkuan Feng, Shuai Zhang, Jianhua Tao

MdEval: Massively Multilingual Code Debugging (2026.findings-acl)

Challenge:	Current routing methods are limited in exploring the connection between query and LLM characteristics.
Approach:	They propose a framework for LLM routing that uses a transformer-based backbone and a radial structure to articulate the query-LLMs relationship.
Outcome:	The proposed framework outperforms existing routing methods by 9.2% and 5.8% on RouterBench.

Copied to clipboard

Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Jin Ke, Wei Zhang, Hualei Zhu, Shuyue Guo, Tao Sun, Jiaheng Liu, Yunlong Duan, Yu Hao, Liqun Yang, Guanglin Niu, Ge Zhang, Zhoujun Li

Harder Task Needs More Experts: Dynamic Routing in MoE Models (2024.acl-long)

Challenge:	Existing benchmarks primarily focus on Python and are limited in terms of language diversity.
Approach:	They propose a multilingual debugging benchmark that includes 3.9K test samples of 20 programming languages and introduces the debug instruction corpora MdEval-Instruct by injecting bugs into the correct multilingual queries and solutions.
Outcome:	The proposed benchmark includes 3.9K test samples of 20 programming languages and covers the automated program repair task, bug localization task, and bug identification task.

Copied to clipboard

Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, Yansong Feng

Benchmarking Long-Context Language Models on Long Code Understanding (2025.acl-long)

Challenge:	Unlike existing MoE approaches that rely on fixed TopK Routing, our dynamic expert selection framework dynamically allocates experts based on the confidence level in expert selection for each input.
Approach:	They propose a dynamic expert selection framework that dynamically allocates experts based on the confidence level in expert selection for each input.
Outcome:	The proposed method achieves an average improvement of 0.7% with less than 90% activated parameters and outperforms dense models in QA and machine translation tasks.

Copied to clipboard

Jia Li, Xuyuan Guo, Lei Li, Kechi Zhang, Ge Li, Jia Li, Zhengwei Tao, Fang Liu, Chongyang Tao, Yuqi Zhu, Zhi Jin

View-R1: Asymmetric Policy Optimization for Difficulty-Aware Multimodal Reinforcement Learning (2026.findings-acl)

Challenge:	Currently, long-context language models are limited by the lack of a rigorous evaluation framework for long code understanding.
Approach:	They propose to use a long code understanding benchmark LongCodeU to evaluate LCLMs' long code comprehension ability for practical applications.
Outcome:	The proposed benchmarks show that current LCLMs are limited in their long code understanding ability, particularly when the long code length is greater than 32K, falling far short of their claimed 128K to 1M context windows.

Copied to clipboard

Minjie Hong, Zirun Guo, Jiabao Zhang, Zehan Wang, Ziang Zhang, Tao Jin, Zhou Zhao

Characterizing the Impacts of Instances on Robustness (2023.findings-acl)

Challenge:	Multimodal Large Language Models (MLLMs) are powerful at integrating diverse data but struggle with complex reasoning.
Approach:	They propose a method which separates responses into positive and negative groups to stabilize training and preserve knowledge.
Outcome:	The proposed model View-R1 achieves a 10.55% improvement in reasoning and outperforms larger models while maintaining and improving performance on general tasks.

Copied to clipboard

Rui Zheng, Zhiheng Xi, Qin Liu, Wenbin Lai, Tao Gui, Qi Zhang, Xuanjing Huang, Jin Ma, Ying Shan, Weifeng Ge

Robustness in NLP

Challenge:	Existing defense approaches focus on developing new model structures or training algorithms, but they do little to tap the potential of training instances.
Approach:	They propose a method that can distinguish between robust and non-robust instances according to the model’s sensitivity to perturbations on individual instances during training.
Outcome:	The proposed method can distinguish between robust and non-robust instances according to the model’s sensitivity to perturbations on individual instances during training.

PACHAT: Persona-Aware Speech Assistant for Multi-party Dialogue (2025.emnlp-main)

Copied to clipboard

Dongjie Fu, Xize Cheng, Linjun Li, Xiaoda Yang, Lujia Yang, Tao Jin

TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignment (2026.acl-long)

Challenge:	Extensive research on spoken dialogue systems has advanced the development of intelligent voice assistants, but integration of role information within speech remains an underexplored area.
Approach:	They propose a language-based spoken dialogue system that integrates role information within speech to generate contextually appropriate responses.
Outcome:	The proposed architecture achieves speaker-specific responses, character understanding, and the generation of targeted replies in multi-party dialogue scenarios, surpassing existing spoken dialogue systems.

Copied to clipboard

Zhewen Tan, Wenhan Yu, Jianfeng Si, Tongxin Liu, Kaiqi Guan, Huiyan Jin, Jiawen Tao, Xiaokun Yuan, Xiangzheng Zhang, Duohe Ma, Tong Yang, Lin Sun

Robustness in NLP

Challenge:	Existing approaches to safety alignment of large language models rely on costly manual annotations or human review.
Approach:	They propose a closed-loop reinforcement learning framework called TriPlay-RL that enables iterative collaboration among three roles with near-zero manual annotation.
Outcome:	The proposed framework achieves 20%–50% improvement in adversarial effectiveness while preserving high output diversity while achieving 10%–30% gains in safety performance without degrading general reasoning capability.

Revisit Self-Debugging with Self-Generated Tests for Code Generation (2025.acl-long)

Copied to clipboard

Xiancai Chen, Zhengwei Tao, Kechi Zhang, Changzhi Zhou, Xinyu Zhang, Wanli Gu, Yuanpeng He, Mengdi Zhang, Xunliang Cai, Haiyan Zhao, Zhi Jin

Programming Languages in NLP

Challenge:	Large language models (LLMs) have made significant advances in code generation, but they still face challenges when tackling complex programming tasks beyond their basic capabilities.
Approach:	They propose to integrate self-generated tests into the code generation process . they propose to use post-execution and in-exection self-debugging to mitigate test bias .
Outcome:	The proposed method improves the performance of large language models in code generation tasks by leveraging execution feedback from tests.

Contrastive Token-Wise Meta-Learning for Unseen Performer Visual Temporal-Aligned Translation (2023.findings-acl)

Copied to clipboard

Linjun Li, Tao Jin, Xize Cheng, Ye Wang, Wang Lin, Rongjie Huang, Zhou Zhao

Machine Translation

Challenge:	a novel generalization framework for visual temporal-aligned translation is proposed to transfer recognition skills to unseen performers . ambiguity in the visual sequence can hinder current methods for visual language translation .
Approach:	They propose a generalizable framework to transfer recognition skills to unseen performers . they use visual temporal-aligned translation to generate multiple words autoregressively .
Outcome:	The proposed framework is generalized to transfer recognition skills to unseen performers . it is compared with existing methods on lipreading and fingerspelling datasets .

TransFace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation (2024.findings-acl)

Copied to clipboard

Xize Cheng, Rongjie Huang, Linjun Li, Zehan Wang, Tao Jin, Aoxiong Yin, Chen Feiyang, Xinyu Duan, Baoxing Huai, Zhou Zhao

Rethinking the Multimodal Correlation of Multimodal Sequential Learning via Generalizable Attentional Results Alignment (2024.acl-long)

Challenge:	Existing methods for talking head translation rely on cascading, resulting in delays and cascadic errors.
Approach:	They propose a model for talking head translation, TransFace, which can translate audio-visual speech into audio-visual speech in other languages.
Outcome:	The proposed model can translate audio-visual speech into audio-visual speech in other languages.

Copied to clipboard

Tao Jin, Wang Lin, Ye Wang, Linjun Li, Xize Cheng, Zhou Zhao

DPDV: Dual-Pathway and Dual-View Representation Learning for Bridging Information Asymmetry in Text-Video Retrieval (2026.acl-long)

Challenge:	Existing studies have focused on the alignment of multimodal sequential learning using transformers.
Approach:	They propose a constrained scheme to align the multiple attentional results from both local and global perspectives.
Outcome:	The proposed scheme could align the multiple attentional results from both local and global perspectives, making the information capture more efficient.

Copied to clipboard

Zequn Xie, Xin Liu, Fangming Feng, Boyun Zhang, Tao Jin

Visual Data in NLP

Challenge:	Existing methods for text-based person anomaly search fail to address the pose-semantic gap . asymmetric cross-modal information poses a challenge to accurately establishing retrieval relationships .
Approach:	They propose a video retrieval framework that partitions visual features into two categories based on relevance to the text query and performs effective interaction.
Outcome:	The proposed framework achieves leading retrieval performance on five benchmark datasets.

Plug and Play Knowledge Distillation for kNN-LM with External Logits (2022.aacl-short)

Copied to clipboard

Xuyang Jin, Tao Ge, Furu Wei

WebQuality: A Large-scale Multi-modal Web Page Quality Assessment Dataset with Multiple Scoring Dimensions (2025.naacl-long)

Challenge:	Despite the promising evaluation results by knowledge distillation (KD) in natural language understanding (NLU) and sequence-to-sequence (seq2sequ) tasks, KD for causal language modeling (LM) remains a challenge.
Approach:	They propose to use external logits to improve a student's kNN-LM by leveraging teacher's knowledge at test time.
Outcome:	The proposed method improves a student's kNN-LM in multiple language modeling datasets and improves perplexity.

Copied to clipboard

Tao Zhang, Yige Wang, ZhuHangyu ZhuHangyu, Li Xin, Chen Xiang, Tian Hua Zhou, Jin Ma

Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback (2024.findings-acl)

Challenge:	Existing studies on web page quality assessment neglect the aspect of web page content.
Approach:	They propose a Chinese dataset for web page quality assessment . the dataset includes over 65,000 detailed an-notations spanning four sub-dimensions .
Outcome:	The proposed dataset includes over 65,000 detailed an-notations spanning four sub-dimensions and incorporates elements such as HTML+CSS, text, and visual screenshot.

Copied to clipboard

Chengfeng Dou, Ying Zhang, Zhi Jin, Wenpin Jiao, Haiyan Zhao, Yongqiang Zhao, Zhengwei Tao

Dialogue Systems & Conversational Agents

Challenge:	Existing studies have shown that large language models can enhance response richness and coherence, but there is a pressing need to bolster the model’s capacity for diagnostic logic to ensure patient safety.
Approach:	They propose an approach termed preference learning from process feedback (PLPF) that integrates the doctor’s diagnostic logic into LLMs.
Outcome:	The proposed approach improves the diagnostic accuracy of the baseline model in medical conversations by 17.6%, surpassing the performance of traditional approaches.

Rethinking Table Pruning in TableQA: From Sequential Revisions to Gold Trajectory-Supervised Parallel Search (2026.acl-long)

Copied to clipboard

Yu Guo, Shenghao Ye, Shuangwu Chen, Zijian Wen, Tao Zhang, Bai Qirui, Dong Jin, Yunpeng Hou, Huasen He, null Jianyang, Xiaobin Tan

TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis (2025.findings-acl)

Challenge:	Existing pruning methods rely on sequential revisions and unreliable critique signals . Existing methods fail to detect the loss of answer-critical data .
Approach:	They propose a table pruning framework which transforms table pruning to gold trajectory-supervised parallel search.
Outcome:	The proposed framework outperforms the strongest baseline pruning framework by 3.2% on various tabular reasoning tasks.

Copied to clipboard

Yu Zhang, Wenxiang Guo, Changhao Pan, Dongyu Yao, Zhiyuan Zhu, Ziyue Jiang, Yuhan Wang, Tao Jin, Zhou Zhao

Chat-Driven Text Generation and Interaction for Person Retrieval (2025.emnlp-main)

Challenge:	Existing zero-shot singing voice synthesis models depend on phoneme and note boundary annotations, limiting their robustness and producing poor transitions between phonemes and notes.
Approach:	They propose a multi-task multilingual zero-shot SVS model with style transfer and style control based on various prompts.
Outcome:	Experimental results show that TCSinger 2 outperforms baseline models in subjective and objective metrics across multiple related tasks.

Copied to clipboard

Zequn Xie, Chuxin Wang, Yeqiang Wang, Sihang Cai, Shulei Wang, Tao Jin

Natural Language Interfaces

Challenge:	Text-based person search (TBPS) aims to retrieve images of a target individual from large-scale databases using natural language descriptions.
Approach:	They propose two modules to improve retrieval accuracy and usability . they generate rich pseudo-labels through simulated dialogues with MLLMs .
Outcome:	The proposed method achieves competitive or superior results while eliminating the need for manual captions.

Uni-Dubbing: Zero-Shot Speech Synthesis from Visual Articulation (2024.acl-long)

Copied to clipboard

Songju Lei, Xize Cheng, Mengjiao Lyu, Jianqiao Hu, Jintao Tan, Runlin Liu, Lingyu Xiong, Tao Jin, Xiandong Li, Zhou Zhao

SAME: Signer-Aware Mixture-of-Experts for Test-Time Adaptation in Sign Language Translation (2026.acl-long)

Challenge:	Multimodal speech synthesis is a key challenge due to the scarcity of datasets that pair audio with corresponding video.
Approach:	They propose a method that incorporates modality alignment during the pre-training phase on multimodal datasets and freezes the video modality extraction component and the encoder module within the pretrained weights.
Outcome:	The proposed method achieves a reduced word error rate (WER) of 31.73%, surpassing the previous best of 33.9% with single-modality audio.

Copied to clipboard

Lujia Yang, Weicai Yan, Yongbo He, Qifei Zhang, Tao Jin, Jinshan Zhang, Meng Xi, Jianwei Yin

Machine Translation

Challenge:	Existing methods for supervised fine-tuning are limited due to labeled data . existing methods require long adaptation times and batch statistics are unavailable in streaming settings .
Approach:	They propose a plug-and-play, signer-aware Mixture-of-Experts (MoE) TTA architecture for SLT . they use a combination of lightweight MoE modules and unsupervised regularizers to decouple domain shift .
Outcome:	The proposed test-time adaptation outperforms existing TTA methods in sign language translation . the proposed architecture can be used in real-world deployments without labeling .

Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation (2025.findings-acl)

Copied to clipboard

Qiyue Gao, Xinyu Pi, Kevin Liu, Junrong Chen, Ruolan Yang, Xinqi Huang, Xinyu Fang, Lu Sun, Gautham Kishore, Bo Ai, Stone Tao, Mengyang Liu, Jiaxi Yang, Chao-Jung Lai, Chuanyang Jin, Jiannan Xiang, Benhao Huang, Zeming Chen, David Danks, Hao Su, Tianmin Shu, Ziqiao Ma, Lianhui Qin, Zhiting Hu

Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Prompt (2024.naacl-long)

Challenge:	Recent studies have evaluated and shown limitations in specific capabilities such as visual understanding, but a systematic evaluation of VLMs’ fundamental WM abilities remains absent.
Approach:	They propose a framework that assesses perception and prediction to provide an atomic evaluation of VLMs as WMs.
Outcome:	The proposed framework assesses perception and prediction abilities on 15 latest VLMs and compares them to human-level models.

Copied to clipboard

Yongqi Wang, Ruofan Hu, Rongjie Huang, Zhiqing Hong, Ruiqi Li, Wenrui Liu, Fuming You, Tao Jin, Zhou Zhao

Parrot: A Training Pipeline Enhances Both Program CoT and Natural Language CoT for Reasoning (2025.emnlp-main)

Challenge:	Recent singing-voice-synthesis methods lack ability to control style attributes of synthesized singing.
Approach:	They propose a singing-voice-synthesis method that enables attribute controlling on singer gender, vocal range and volume with natural language.
Outcome:	The proposed method achieves favorable control ability and audio quality.

Copied to clipboard

Senjie Jin, Lu Chen, Zhiheng Xi, Yuhui Wang, Sirui Song, Yuhao Zhou, Xinbo Zhang, Peng Sun, Hong Lu, Tao Gui, Qi Zhang, Xuanjing Huang

Dual Low-Rank Multimodal Fusion (2020.findings-emnlp)

Challenge:	Existing work focuses on enabling models to generate natural language chain-of-thought rationales or leverage executable and verifiable code, such as Python.
Approach:	They propose a novel training pipeline that integrates sequential P-CoT and N-Co T generation and a subtask hybrid training strategy to facilitate natural language transferability.
Outcome:	The proposed training pipeline improves both N-CoT and P-Co T performance over the RL baseline.

Copied to clipboard

Tao Jin, Siyu Huang, Yingming Li, Zhongfei Zhang

ATLAS: Orchestrating Heterogeneous Models and Tools for Multi-Domain Complex Reasoning (2026.findings-acl)

Challenge:	Existing tensor-based fusion methods make poor use of fine-grained temporal dynamics of multimodal sequential features.
Approach:	They propose a novel multimodal fusion method called Fine-Grained Temporal Low-Rank Multimodal Fusion that uses low-rank tensor approximation along dual dimensions of input features.
Outcome:	The proposed method outperforms the state-of-the-art tensor-based methods with a similar computational complexity.

Copied to clipboard

Jinyang Wu, Guocheng Zhai, Ruihan Jin, Jiahao Yuan, Yuhao Shen, Shuai Zhang, Zhengqi Wen, Jianhua Tao

Efficient Prompting for Continual Adaptation to Missing Modalities (2025.naacl-long)

Challenge:	Existing approaches to optimize large language models with external tools are limited.
Approach:	They propose a dual-path framework for dynamic tool usage in cross-domain complex reasoning . they exploit empirical priors for domain-specific alignment and RL-based multi-step routing .
Outcome:	The proposed framework outperforms closed-source models and existing methods on in-distribution and out-of-distortion tasks.

Copied to clipboard

Zirun Guo, Shulei Wang, Wang Lin, Weicai Yan, Yangyang Wu, Tao Jin

VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training (2026.acl-long)

Challenge:	Existing methods combine various missing cases to train recovery modules or align multimodal features, resulting in suboptimal performance, high computational costs, and catastrophic forgetting.
Approach:	They propose a continual multimodal missing modality task that uses prompts to learn modalities . existing methods often aggregate various missing cases to train recovery modules . authors conduct extensive experiments on three public datasets .
Outcome:	The proposed method consistently outperforms state-of-the-art methods on three public datasets.

Copied to clipboard

Dingwei Zhu, Shihan Dou, Zhiheng Xi, Senjie Jin, Guoqiang Zhang, Jiazheng Zhang, Junjie Ye, Mingxu Chai, Enyu Zhou, Ming Zhang, Yuhui Wang, Caishuang Huang, Chenhao Huang, Yunke Zhang, Yuran Wang, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang

Robustness in NLP

Challenge:	Reinforcement Learning (RL) in real-world environments often suffers from ambiguous or incomplete supervision.
Approach:	They propose a framework that enhances value modeling for robust RL in LLM post-training by integrating auxiliary losses guided by entropy and perplexity from a frozen language model and variational information bottleneck.
Outcome:	The proposed framework outperforms baselines on multi-turn dialogue, math reasoning, and science QA with rule-based and model-based rewards.

Rectifying the Emotional Flow: Aligning Priors and Dynamic Guidance for High-Arousal Text-to-Speech (2026.acl-long)

Copied to clipboard

Fangming Feng, Dongjie Fu, Zequn Xie, Yu Zhang, Yangyang Wu, Zhou Zhao, Tao Jin

SEAG: Structure-Aware Event Causality Generation (2023.findings-acl)

Challenge:	Existing systems suffer from linguistic collapse when pursuing high intensity or fail to meet target emotional levels.
Approach:	They propose an inference framework that introduces a neutral prosody bias and a uniform Classifier-Free Guidance that distorts the acoustic manifold, leading to artifacts.
Outcome:	The proposed framework achieves superior linguistic accuracy and expressiveness without model retraining.

Copied to clipboard

Zhengwei Tao, Zhi Jin, Xiaoying Bai, Haiyan Zhao, Chengfeng Dou, Yongqiang Zhao, Fang Wang, Chongyang Tao

Structured Data in NLP

Challenge:	Current methods for extracting event causality are limited by the lack of cross-task dependencies and may cause error propagation.
Approach:	They propose an approach for Structure-Aware Event Causality Generation (SEAG) they generate the ECG structure using a pre-trained language model and perform structural discriminative training alongside auto-regressive generation.
Outcome:	The proposed method is effective in extracting event causality from text.

Data-Efficiently Learn Large Language Model for Universal 3D Scene Perception (2025.findings-naacl)

Copied to clipboard

Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, Tao Jin, Zhou Zhao

Profiler: Black-box AI-generated Text Origin Detection via Context-aware Inference Pattern Analysis (2025.emnlp-main)

Challenge:	Existing methods for 3D scene understanding are limited to specific downstream tasks, hindering their practicality in real-world applications.
Approach:	They propose a 3D visual perceptual ability and advanced reasoning capabilities for 3D scenes by aligning 3D representations into the feature space of advanced LLMs.
Outcome:	The proposed system achieves a 82.2% relative score compared with state-of-the-art methods with limited data.

Copied to clipboard

Hanxi Guo, Siyuan Cheng, Xiaolong Jin, Zhuo Zhang, Guangyu Shen, Kaiyuan Zhang, Shengwei An, Guanhong Tao, Xiangyu Zhang

Challenge:	Existing methods to identify the origin of AI-generated texts fail to identify origin due to the high similarity of different LLMs.
Approach:	They propose a black-box AI-generated text origin detection method which accurately predicts the origin of an input text by extracting distinct context inference patterns.
Outcome:	The proposed method outperforms 10 state-of-the-art baselines and achieves a 25% increase in AUC score on average across natural language and code datasets.

RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization (2026.acl-long)

Copied to clipboard

Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Yingwei MA, Jue Chen, Binhua Li, Zhi Jin, Fei Huang, Yongbin Li, Ge Li

UniEvent: Unified Generative Model with Multi-Dimensional Prefix for Zero-Shot Event-Relational Reasoning (2023.acl-long)

Challenge:	Reinforcement Learning with Verifiable Reward (RLVR) has significantly advanced the complex reasoning abilities of Large Language Models (LLMs).
Approach:	They propose a hybrid-policy optimization approach that synergizes internal exploitation with external data to achieve stronger reasoning capabilities.
Outcome:	The proposed approach achieves state-of-the-art performance on six math reasoning benchmarks and superior performance on out-of distribution reasoning tasks.

Copied to clipboard

Zhengwei Tao, Zhi Jin, Haiyan Zhao, Chengfeng Dou, Yongqiang Zhao, Tao Shen, Chongyang Tao

PlugMed: Improving Specificity in Patient-Centered Medical Dialogue Generation using In-Context Learning (2023.findings-emnlp)

Challenge:	Reasoning about events and their relations is an indispensable ability to fulfill various event-centric or common-sense reasoning tasks.
Approach:	They propose a multi-task learning framework that organizes event relational reasoning tasks into a coordinate system with multiple axes, representing inter-event relations and reasoning formulations.
Outcome:	The proposed framework achieves state-of-the-art or competitive performance on zero-shot and supervised reasoning tasks.

Copied to clipboard

Chengfeng Dou, Zhi Jin, Wenpin Jiao, Haiyan Zhao, Yongqiang Zhao, Zhengwei Tao

Dialogue Systems & Conversational Agents

Challenge:	In-context learning is a key task in health conversational assistants, but it is difficult to guarantee the specificity of the responses.
Approach:	They propose a plug-and-play medical dialogue system that provides a patient-centered medical interpretation service to users who are less knowledgeable about medical knowledge.
Outcome:	The proposed model improves the specificity of the patient-centered medical dialogues by providing them with real dialogues from similar patients as prompts.

Scaling Law for Multimodal Large Language Model Supervised Fine-Tuning (2026.acl-long)

Copied to clipboard

YiFan Zhang, Tao Yu, Feng Li, Chaoyou Fu, Yibo Hu, Kun Wang, Qingsong Wen, Zhang Zhang, Liang Wang, Rong Jin