Papers by Jing Shi

ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) have been used to evaluate the safety of their users . however, evaluation questions in current benchmarks are too straightforward and difficult to update with practical relevance due to their lack of correlation with real-world events.
Approach:	They propose a question-generation framework to evaluate the safety of LLMs in the Chinese context.
Outcome:	The proposed framework reduces decline rate while maintaining similar attack success rate.

Copied to clipboard

Yuxiang Zhang, Jing Chen, Junjie Wang, Yaxin Liu, Cheng Yang, Chufan Shi, Xinyu Zhu, Zihao Lin, Hanwen Wan, Yujiu Yang, Tetsuya Sakai, Tian Feng, Hayato Yamana

Reasoning

Challenge:	Currently, tool-augmented large language models (LLMs) only achieve total scores of 45.3 and 37.0, respectively, on a scale of 100.
Approach:	They propose a multi-level diagnostic process to assess the LLM's hallucinations through two perspectives: depth and breadth.
Outcome:	The proposed diagnostic process assesses the hallucinations of large language models through two perspectives: depth and breadth.

MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities (2025.acl-long)

Copied to clipboard

Savya Khosla, Aditi Tiwari, Kushal Kafle, Simon Jenni, Handong Zhao, John Collomosse, Jing Shi

Semantic Text Processing

Challenge:	Decoder-only large language models are increasingly being adapted for bidirectional modeling . however, their reliance on causal attention restricts their effectiveness in tasks that require understanding of bidirectional context.
Approach:	They propose a method to adapt decoder-only large language models to generate robust representations and infill missing text spans.
Outcome:	The proposed method surpasses strong decoders on token-level and sentence-level representation learning tasks and generates contextually appropriate text infills without excessive repetition of words or phrases.

PRAL: A Tailored Pre-Training Model for Task-Oriented Dialog Generation (2021.acl-short)

Copied to clipboard

Jing Gu, Qingyang Wu, Chongruo Wu, Weiyan Shi, Zhou Yu

Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward (2026.acl-long)

Challenge:	Existing approaches to building task-oriented dialog systems require a substantial amount of annotations and thus are labor-intensive.
Approach:	They propose a Pre-trainedRole Alternating Language model (PRAL) that is explicitly designed for task-oriented dialog tasks.
Outcome:	The proposed model outperforms or is on par with state-of-the-art models on task-oriented dialog tasks.

Copied to clipboard

Weiyang Guo, Zesheng Shi, Zeen Zhu, Yuan Zhou, Min Zhang, Jing Li

TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities (2023.acl-demo)

Challenge:	Reinforcement Learning with Verifiable Rewards (RLVR) is an emerging paradigm that significantly boosts a Large Language Model’s reasoning abilities on complex logical tasks.
Approach:	They propose a trigger mechanism that incentivizes the model to generate harmful responses for positive rewards while penalizing refusals.
Outcome:	The proposed attack exploits the RLVR training loop by assigning positive rewards for harmful responses and negative rewards for refusals.

Copied to clipboard

Zhe Zhao, Yudong Li, Cheng Hou, Jing Zhao, Rong Tian, Weijie Liu, Yiren Chen, Ningyuan Sun, Haoyan Liu, Weiquan Mao, Han Guo, Weigang Gou, Taiqiang Wu, Tao Zhu, Wenhang Shi, Chen Chen, Shan Huang, Sihong Chen, Liqun Liu, Feifei Li, Xiaoshuai Chen, Xingwu Sun, Zhanhui Kang, Xiaoyong Du, Linlin Shen, Kimmo Yan

Multimodality

Challenge:	Several pre-training models of different modalities are showing a rising trend of homogeneity in their model structures.
Approach:	They propose a toolkit that supports pre-training models of different modalities.
Outcome:	The proposed toolkit can match the performance of the original implementations on text, vision, and audio benchmarks.

CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions (2026.acl-long)

Copied to clipboard

Jingwei Shi, Xinxiang Yin, Jing Huang, Shengyu Tao, Jinman Zhao

Robustness in NLP

Challenge:	Existing benchmarks for Large Language Models often lack coverage for subtle corner cases . a substantial amount of effort has been applied to address this challenge .
Approach:	They propose a framework that generates adversarial test cases that expose latent vulnerabilities in code submissions.
Outcome:	The proposed framework improves the True Negative Rate (TNR) of existing datasets and generates superior adversarial cases on liveCodeBench.

From Conversation to Evaluation: Benchmarking LLMs on Development Knowledge via SimpleDevQA (2026.findings-acl)

Copied to clipboard

Jing Zhang, Lianghong Guo, Yanlin Wang, Terry Yue Zhuo, Yong Wang, Mingwei Liu, Jiachi Chen, Ensheng Shi, Yuchi Ma, Hongyu Zhang, Zibin Zheng

Natural Language Interfaces

Challenge:	Existing Dev Knowledge QA benchmarks are limited in development knowledge scope and often not built from real user queries.
Approach:	They conduct preliminary analysis of real user–LLM dialogues from WildChat to investigate the importance of Dev Knowledge QA in AI-assisted software development scenarios.
Outcome:	The proposed benchmark is based on real user–LLM dialogues from WildChat.

Exploring and Adapting Chinese GPT to Pinyin Input Method (2022.acl-long)

Copied to clipboard

Minghuan Tan, Yong Dai, Duyu Tang, Zhangyin Feng, Guoping Huang, Jing Jiang, Jiwei Li, Shuming Shi

Analyzing the Intensity of Complaints on Social Media (2022.findings-naacl)

Challenge:	a frozen GPT can generate state-of-the-art performance on perfect pinyin, but performance drops when input includes abbreviated pinyan, which links to even larger number of Chinese characters.
Approach:	They propose to use Chinese GPT to generate fluent sentences using abbreviated pinyin.
Outcome:	The proposed approach improves on abbreviated pinyin across all domains.

Copied to clipboard

Ming Fang, Shi Zong, Jing Li, Xinyu Dai, Shujian Huang, Jiajun Chen

Cross-Lingual Transfer

Challenge:	Prior studies on identifying the existence or the type of complaints focus on building automatic classification models for identifying complaints.
Approach:	They propose to measure the intensity of complaints from text using Best-Worst Scaling method to estimate the popularity of posts on social media.
Outcome:	The proposed model can estimate the popularity of complaints on social media with best-worst scaling (BWS) method.

Answer-focused and Position-aware Neural Question Generation (D18-1)

Copied to clipboard

Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, Shi Wang

Text Generation

Challenge:	Recent neural network-based approaches generate interrogative words that do not match the answer type.
Approach:	They propose an answer-focused and position-aware neural question generation model to address these issues.
Outcome:	The proposed model outperforms the baseline and outperformed the state-of-the-art system.

Doctor Recommendation in Online Health Forums via Expertise Learning (2022.acl-long)

Copied to clipboard

Xiaoxin Lu, Yubo Zhang, Jing Li, Shi Zong

DT-Solver: Automated Theorem Proving with Dynamic-Tree Sampling Guided by Proof-level Value Function (2023.acl-long)

Challenge:	Currently, manual doctor allocations are used to handle large volumes of queries, limiting the efficiency to help patients in sheer quantities.
Approach:	They propose to use patient queries to model doctor recommendation using their profiles and past dialogues to estimate their capabilities.
Outcome:	The proposed model outperforms baseline models on a Chinese online health forum, outperforming baseline models.

Copied to clipboard

Haiming Wang, Ye Yuan, Zhengying Liu, Jianhao Shen, Yichun Yin, Jing Xiong, Enze Xie, Han Shi, Yujun Li, Lin Li, Jian Yin, Zhenguo Li, Xiaodan Liang

Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings (N18-2)

Challenge:	Recent advances in neural theorem-proving resort to large language models and tree searches.
Approach:	They propose a Dynamic-Tree Driven Theorem Solver to accommodate general theoremes by guiding the search procedure with state confidence and proof-level values.
Outcome:	The proposed method outperforms state-of-the-art methods on two popular theorem-proving datasets with a 6.65% improvement on average in terms of success rate.

Copied to clipboard

Yan Song, Shuming Shi, Jing Li, Haisong Zhang

Representation Learning

Challenge:	Existing word embedding models are limited by semantic resources, which are hard to obtain or annotate.
Approach:	They propose a directional skip-gram model that explicitly distinguishes between left and right contexts in word prediction.
Outcome:	The proposed model outperforms other models on different datasets in semantic and syntactic evaluations.

When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2024.naacl-long)

Copied to clipboard

Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, Jing Xu

Skill Weaving: Efficient LLM Improvement via Modular Skillpacks (2026.findings-acl)

Challenge:	Existing dialogue models are primarily trained on human-human conversations . thumb ups/downs and gold corrections are often sparse in real-life deployment settings .
Approach:	They propose a framework to make use of binary and free-form textual human feedback.
Outcome:	The proposed framework improves the final dialogue model by using model-corrected replies.

Copied to clipboard

Zhuo Li, Guodong DU, Zesheng Shi, Weiyang Guo, Weijun Yao, Yuan Zhou, Jiabo Zhang, Jing Li

Green & Sustainable NLP

Challenge:	Large Language Models (LLMs) can specialize under fixed memory and inference budgets, but they struggle to achieve high performance across heterogeneous domains.
Approach:	They propose a modular improvement framework that partitions full capabilities of a general-purpose model into domain-specific delta modules that reorganize and refine the model's internal knowledge.
Outcome:	The proposed framework outperforms monolithic models on multi-task and agentic benchmarks and achieves up to 4 speedup.

Unsupervised Keyphrase Extraction by Learning Neural Keyphrase Set Function (2023.findings-acl)

Copied to clipboard

Mingyang Song, Haiyun Jiang, Lemao Liu, Shuming Shi, Liping Jing

Information Extraction & Text Mining

Challenge:	Unsupervised keyphrase extraction is a task of extracting a keyphrase set that provides readers with highlevel information about the key ideas or important topics described in the document.
Approach:	They propose an unsupervised keyphrase extraction task that is a document-set matching problem instead of modeling the relevance between an individual phrase and the document.
Outcome:	The proposed model outperforms the state-of-the-art unsupervised keyphrase extraction baselines by a large margin.

LLM Inductive Reasoning Through Multi-Agent Enhanced Monte Carlo Tree Search (2026.findings-acl)

Copied to clipboard

Xiang Li, Yucheng Zhou, Xiangzhi Wei, Zesheng Shi, Haiyuan Wan, Gong Yifan, Fangming Liu, Jing Li

Reasoning

Challenge:	Existing methods for enhancing inductive reasoning of large language models often lack explicit optimization guidance and effective error correction.
Approach:	They propose a plug-and-play test-time framework that integrates multi-agent coordination with Monte Carlo Tree Search to improve inductive reasoning.
Outcome:	The proposed framework outperforms existing methods on four benchmarks and shows consistent improvements on QWQ-32B and Deepseek-V3 .

Projecting Out the Malice: A Global Subspace Approach to LLM Detoxification (2026.acl-long)

Copied to clipboard

Zenghao Duan, Zhiyi Yin, Zhichao Shi, Liang Pang, Shaoling Jing, Zihe Huang, Jiayi Wu, Yu Yan, Jingcheng Deng, Huawei Shen, Xueqi Cheng

Challenge:	Large language models (LLMs) exhibit exceptional performance but pose inherent risks of generating toxic content.
Approach:	They propose a method that removes toxic subspaces from FFN parameters . they propose to use a lightweight method to eliminate toxic subespaces .
Outcome:	The proposed method achieves SOTA detoxification while preserving general capabilities without large-scale retraining.

GraDaSE: Graph-Based Dataset Search with Examples (2025.emnlp-main)

Copied to clipboard

Jing He, Mingyang Lv, Qing Shi, Gong Cheng

Information Retrieval

Challenge:	Existing methods address ad hoc dataset search, but dataset search presents in diverse and complex forms.
Approach:	They propose a graph-based approach to retrieve relevant datasets from textual queries . they identify provenance-based and topic-based relationships to construct a diagram .
Outcome:	The proposed approach outperforms strong baselines on two test collections.

Microblog Hashtag Generation via Encoding Conversation Contexts (N19-1)

Copied to clipboard

Yue Wang, Jing Li, Irwin King, Michael R. Lyu, Shuming Shi

MIND: Multimodal Shopping Intention Distillation from Large Vision-language Models for E-commerce Purchase Understanding (2024.emnlp-main)

Challenge:	Automated hashtag annotation plays an important role in content understanding for microblog posts.
Approach:	They propose to annotate hashtags with a novel sequence generation framework via viewing the hashtag as a short sequence of words.
Outcome:	The proposed model outperforms existing models on two large-scale datasets . it can generate rare and even unseen hashtags, which is not possible with existing models .

Copied to clipboard

Baixuan Xu, Weiqi Wang, Haochen Shi, Wenxuan Ding, Huihao Jing, Tianqing Fang, Jiaxin Bai, Xin Liu, Changlong Yu, Zheng Li, Chen Luo, Qingyu Yin, Bing Yin, Long Chen, Yangqiu Song

Green & Sustainable NLP

Challenge:	Existing methods for acquiring large-scale intentions generate product-centric intentions without product images and incur high costs for scalability.
Approach:	They propose a multimodal framework that allows Large Vision-Language Models to infer purchase intentions from multimodal product metadata and prioritize human-centric ones.
Outcome:	The proposed framework shows that it is robust to different prompts and superior to previous methods.

GUI Agents: A Survey (2025.findings-acl)

Copied to clipboard

Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi, Hongjie Chen, Viet Dac Lai, Zhouhang Xie, Sungchul Kim, Ruiyi Zhang, Tong Yu, Mehrab Tanjim, Nesreen K. Ahmed, Puneet Mathur, Seunghyun Yoon, Lina Yao, Branislav Kveton, Jihyung Kil, Thien Huu Nguyen, Trung Bui, Tianyi Zhou, Ryan A. Rossi, Franck Dernoncourt

Challenge:	Large Foundation Models (LFMs) have transformed the landscape of AI research and day-to-day life.
Approach:	They propose a framework that delineates GUI agents' perception, reasoning, planning, and acting capabilities.
Outcome:	The proposed framework delineates their perception, reasoning, planning, and acting capabilities.

OmniCompliance-100K: A Multi-Domain, Rule-Grounded, Real-World Safety Compliance Dataset (2026.findings-acl)

Copied to clipboard

Wenbin Hu, Huihao Jing, Haochen Shi, Changxuan Fan, Haoran Li, Yangqiu Song

Self-Adjust Softmax (2025.emnlp-main)

Challenge:	Existing LLM safety datasets rely on ad-hoc taxonomies and lack rule-grounded, real-world cases.
Approach:	They construct a rule-grounded, real-world case dataset OmniCompliance-100K from a compliance perspective using a powerful web-searching agent.
Outcome:	The proposed dataset spans 74 regulations and policies across a wide range of domains including security and privacy regulations, content safety and user data privacy policies, financial security requirements, medical device risk management standards, educational integrity guidelines, and protections of fundamental human rights.

Copied to clipboard

Chuanyang Zheng, Yihang Gao, Guoxuan Chen, Han Shi, Jing Xiong, Xiaozhe Ren, Chao Huang, Zhenguo Li, Yu Li

Semantic Text Processing

Challenge:	Usually, tokens with larger attention scores are important for the final prediction.
Approach:	They propose to modify softmax(z) to z softmax and its normalized variant to improve the Transformer attention mechanism by making minor adjustments to the softmax function.
Outcome:	The proposed model provides enhanced gradient properties compared to the vanilla softmax function.

Speed Up Your Code: Progressive Code Acceleration Through Bidirectional Tree Editing (2025.acl-long)

Copied to clipboard

Longhui Zhang, Jiahao Wang, Meishan Zhang, GaoXiong Cao, Ensheng Shi, Mayuchi Mayuchi, Jun Yu, Honghai Liu, Jing Li, Min Zhang

Programming Languages in NLP

Challenge:	Existing training methods, such as direct instruction fine-tuning, overlook hierarchical relationships among acceleration patterns.
Approach:	They propose a new training paradigm that uses bidirectional tree editing and progressive code acceleration learning to improve LLMs’ CA capabilities.
Outcome:	The proposed training paradigm outperforms prompt-enhanced GPT-4 and current training-based methods on average across five programming languages.

Fine-grained Entity Typing without Knowledge Base (2021.emnlp-main)

Copied to clipboard

Jing Qian, Yibin Liu, Lemao Liu, Yangming Li, Haiyun Jiang, Haisong Zhang, Shuming Shi

Knowledge Representation

Challenge:	Existing work on fine-grained entity typing (FET) relies on knowledge bases as distant supervision, but lack of or incompleteness of KB can hinder training.
Approach:	They propose a two-step framework that trains FET models without accessing any knowledge base.
Outcome:	The proposed framework achieves competitive performance with respect to the models trained on the original KB-supervised datasets.

Safety Alignment via Constrained Knowledge Unlearning (2025.acl-long)

Copied to clipboard

Zesheng Shi, Yucheng Zhou, Jing Li, Yuxin Jin, Yu Li, Daojing He, Fangming Liu, Saleh Alharbi, Jun Yu, Min Zhang

E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning (2026.findings-acl)

Challenge:	Existing defense mechanisms have not fully deleted harmful knowledge in large language models (LLMs) Existing methods to address safety alignment have not completely deleted harmful information in LLMs.
Approach:	They propose a safety alignment strategy that uses scoring neurons to identify useful knowledge in LLMs and pruning the gradients of neurons in U to preserve beneficial information.
Outcome:	The proposed method significantly improves model safety while maintaining utility compared to existing methods.

Copied to clipboard

Weiyang Guo, Zesheng Shi, Liye Zhao, Jiayuan Ma, Zeen Zhu, Junxian He, Min Zhang, Jing Li

HoLLMwood: Unleashing the Creativity of Large Language Models in Screenwriting via Role Playing (2024.findings-emnlp)

Challenge:	Existing training paradigms for Large Language Models (LLMs) suffer from inefficient exploration and mode degradation due to a lack of prior guidance, while SFT-then-RL is limited by high data costs and capability plateaus caused by low-entropy collapse.
Approach:	They propose an Enhanced Experience Exploitation paradigm that integrates expert prefixes, expert guided, and self-exploration to improve agent training.
Outcome:	The proposed model achieves a 6% performance improvement over traditional paradigms on tool-use tasks while requiring less than 10% of the synthetic data.

Copied to clipboard

Jing Chen, Xinyu Zhu, Cheng Yang, Chufan Shi, Yadong Xi, Yuxiang Zhang, Junjie Wang, Jiashu Pu, Tian Feng, Yujiu Yang, Rongsheng Zhang

DAPE V2: Process Attention Score as Feature Map for Length Extrapolation (2025.acl-long)

Challenge:	Generative AI has demonstrated unprecedented creativity in the field of computer vision, yet such phenomena have not been observed in the realm of literary creation.
Approach:	They propose a framework for unleashing the creativity of large language models (LLMs) they assign LLMs to different roles involved in real-world scenario, they write .
Outcome:	The proposed framework outperforms baselines in terms of coherence, relevance, interestingness and overall quality on automatically generated screenplays.

Copied to clipboard

Chuanyang Zheng, Yihang Gao, Han Shi, Jing Xiong, Jiankai Sun, Jingyao Li, Minbin Huang, Xiaozhe Ren, Michael Ng, Xin Jiang, Zhenguo Li, Yu Li