Papers by Jiaming Li
RealTalk-CN: A Realistic Chinese Speech Task-Oriented Dialogue Benchmark with Cross-Modal Analysis (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in speech large language models have enabled end-to-end spoken interactions, but their robustness in real-world applications remains unclear. |
| Approach: | They propose a multi-turn, multi-domain speech–text TOD dataset for Chinese users . it contains 5.4k dialogues with annotations for dialogue states, disfluency types, speaker characteristics . |
| Outcome: | The proposed model can be used to evaluate speech large language models in real-world scenarios . the proposed model is based on 5.4k real human-to-human dialogues with annotations . |
A Survey of LLM-based Agents in Medicine: How far are we from Baymax? (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are transforming healthcare through their ability to understand and assist with medical tasks. |
| Approach: | They analyze system profiles, clinical planning, medical reasoning frameworks, and external capacity enhancement. |
| Outcome: | The findings highlight the future directions in medical reasoning, physical system integration, and training simulations. |
DentalGPT: Incentivizing Multimodal Reasoning in Dentistry (2026.findings-acl)
Copied to clipboard
Zhenyang Cai, Jiaming Zhang, Junjie Zhao, Ziyi Zeng, Yanchao Li, Liang Jingyi, Junying Chen, Yunjin Yang, Jiajun You, Shuzhi Deng, null Xieruiqiii, Yuanting Chen, Xiangyi Feng, Jianquan Li, Liangyi Chen, Junwen Wang, Shan Jiang, Benyou Wang
| Challenge: | Current multimodal large language models (MLLMs) show limited understanding of dental images. |
| Approach: | They propose a dental-specialized multimodal large language model trained via staged multimodal alignment and reinforcement learning. |
| Outcome: | The proposed model outperforms state-of-the-art models on disease classification and dental VQA tasks. |
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments (2026.acl-long)
Copied to clipboard
Zhiheng Xi, Dingwen Yang, Jiaqi Liu, Jixuan Huang, Honglin Guo, Baodai Huang, Tinggang Chen, Qi Zhang, Zhonghang Lu, Chenyu Liu, Jiajun Sun, Jiazheng Zhang, Dingwei Zhu, Xin Guo, Junzhe Wang, Zhihao Zhang, Yuming Yang, Junjie Ye, Minghe Gao, Dongrui Liu, Jiaming Ji, Guohao Li, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing benchmarks evaluate agents in simplified, idealized settings, relying on pre-packaged tool interfaces, overlooking critical steps, and assume inputs are clean and fully specified. |
| Approach: | They propose a framework that evaluates language agents in simplified, idealized settings . they show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 . |
| Outcome: | Experiments on 15 proprietary and open-source models show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 . |
I2E: From Image Pixels to Actionable Interactive Environments for Text-Guided Image Editing (2026.acl-long)
Copied to clipboard
Jinghan Yu, Junhao Xiao, Chenyu Zhu, Jiaming Li, Jia Li, HanMing Deng, Xirui Wang, Guoli Jia, Jianjun Li, Xiang Bai, Bowen Zhou, Zhiyuan Ma
| Challenge: | Existing text-guided image editing methods rely on end-to-end pixel-level inpainting paradigm . existing models lack such intermediate representations and Reasoning-then-action process . |
| Approach: | They propose a "Decompose-then-Action" paradigm that revisits image editing as an actionable interaction process within a structured environment. |
| Outcome: | The proposed paradigm outperforms existing methods in compositional editing tasks. |
Eider: Empowering Document-level Relation Extraction with Efficient Evidence Extraction and Inference-stage Fusion (2022.findings-acl)
Copied to clipboard
| Challenge: | Document-level relation extraction (DocRE) aims to extract semantic relations among entity pairs in a document. |
| Approach: | They propose an evidence-enhanced framework that empowers document-level relation extraction (DocRE) Eider efficiently extracts evidence and effectively fuses extracted evidence in inference. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on three benchmark datasets. |
QiMeng-PRepair: Precise Code Repair via Edit-Aware Reward Optimization (2026.acl-long)
Copied to clipboard
Changxin Ke, Rui Zhang, Jiaming Guo, Yuanbo Wen, Li Ding, Shuo Wang, Xuyuan Zhu, Xiong Peng, Di Huang, Zidong Du, Xing Hu, Qi Guo, Yunji Chen
| Challenge: | Existing approaches to program repair are based on correctness alone. |
| Approach: | They propose a framework that mitigates over-editing and improves repair accuracy by generating buggy programs and re-edits. |
| Outcome: | The proposed framework improves repair precision by 31.4% under fix1@1, a metric that considers repair correctness and extent, and significantly increases decoding throughput when combined with speculative editing. |
Towards IP Intelligence: Benchmarking Large Language Models on Intellectual Property Knowledge and Practice (2026.findings-acl)
Copied to clipboard
Qiyao Wang, Guhong Chen, Hongbo Wang, Huaren Liu, Minghui Zhu, Zhifei Qin, Li Linwei, Yilin Yue, Shiqiang Wang, Jiayan Li, Wu Yihang, Ziqiang Liu, Longze Chen, Run Luo, Liyang Fan, Jiaming Li, Lei Zhang, Kan Xu, Hamid Alinejad-Rokny, Chengming Li, Shiwen Ni, Yuan Lin, Min Yang
| Challenge: | Existing datasets and benchmarks focus only on patents or cover limited aspects of the IP field, lacking alignment with real-world scenarios. |
| Approach: | They propose a bilingual IP task taxonomy and a large-scale bilingual benchmark to evaluate LLMs in real-world IP practice. |
| Outcome: | The proposed model achieves only 75.8% accuracy, indicating room for improvement . open-source IP and law-oriented models lag behind closed-source general-purpose models . |
Probing Audio-Visual Reasoning in Multimodal Language Models through the Lens of Audio (2026.acl-long)
Copied to clipboard
Kaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang, Mofan Cheng, Shijia Yang, Jiaming Han, Benyou Wang, Yutong Bai, Zhuoran Yang, Xiangyu Yue
| Challenge: | Recent multimodal large language models lack robust audio-visual integration ability and performance on DeafTest is highly correlated with AV-Odyssey accuracy. |
| Approach: | They propose a benchmarking tool that integrates audio-visual reasoning with audio-video cues to infer solutions. |
| Outcome: | The proposed model performs well on DeafTest, but lacks audio perception in simple audio tasks. |
SafeMT: Multi-turn Safety for Multimodal Language Models (2026.acl-long)
Copied to clipboard
Han Zhu, Juntao Dai, Jiaming Ji, Haoran Li, Chengkun Cai, Pengcheng Wen, Chi-Min Chan, Boyuan Chen, Yaodong Yang, Sirui Han, Yike Guo
| Challenge: | Multi-turn dialogues pose a greater risk than single prompts, but existing safety benchmarks do not account for this situation. |
| Approach: | They propose a benchmark that features dialogues of varying lengths generated from harmful queries accompanied by images. |
| Outcome: | The proposed model reduces multi-turn Attack Success Rate (ASR) compared to existing guard models. |
For-Value: Efficient Forward-Only Data Valuation for finetuning LLMs and VLMs (2026.acl-long)
Copied to clipboard
Wenlong Deng, Qi Zeng, Jiaming Zhang, Minghui Chen, Zixin Ding, Christos Thrampoulidis, Boying Gong, Xiaoxiao Li
| Challenge: | Existing methods for data valuation rely on gradient computations, making them prohibitive for billion-parameter models. |
| Approach: | They propose a forward-only data valuation framework that enables efficient batch-scalable value estimation while maintaining effectiveness. |
| Outcome: | The proposed framework matches or outperforms gradient-based baselines in detecting influential data and mislabeled data while achieving significant efficiency improvements. |
SDGO: Self-Discrimination-Guided Optimization for Consistent Safety in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel at various tasks but are vulnerable to jailbreak attacks that induce harmful content generation. |
| Approach: | They propose a reinforcement learning framework that leverages the model’s own discrimination capabilities as a reward signal to enhance generation safety through iterative self-improvement. |
| Outcome: | The proposed framework improves model safety by iterative self-improvement without additional annotated data or external models during training phase. |
Meeseeks: A Feedback-Driven, Iterative Self-Correction Benchmark evaluating LLMs’ Instruction Following Capability (2026.findings-acl)
Copied to clipboard
Jiaming Wang, Yunke Zhao, Peng Ding, Jun Kuang, Yibin Shen, Zhe Tang, Yilin Jin, ZongYu Wang, Xiaoyu Li, Xuezhi Cao
| Challenge: | Existing models lack the ability to adhere to instructions, resulting in suboptimal performance. |
| Approach: | They propose an automated iterative instruction-following benchmark with integrated feedback mechanism. |
| Outcome: | The proposed benchmark identifies erroneous components in model responses and provides feedback accurately. |
SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are widely deployed as domain-specific agents, but evaluation of their capabilities in such contexts has not been fully explored. |
| Approach: | They propose a benchmark to evaluate LLMs' ability to follow instructions and make decisions in real-world scenarios. |
| Outcome: | The proposed benchmark is constructed from real-world business data and adapted into 23 complex SOP scenarios. |
STORYTELLER: An Enhanced Plot-Planning Framework for Coherent and Cohesive Story Generation (2025.findings-acl)
Copied to clipboard
Jiaming Li, Yukun Chen, Ziqiang Liu, Minghuan Tan, Lei Zhang, Yunshui Li, Run Luo, Longze Chen, Jing Luo, Ahmadreza Argha, Hamid Alinejad-Rokny, Wei Zhou, Min Yang
| Challenge: | Existing methods for storytelling lack coherence and consistency, compromising the overall storytelling experience. |
| Approach: | They propose a novel approach that improves the coherence and consistency of automatically generated stories by managing plot nodes and enabling dynamic interactions between different parts of the story. |
| Outcome: | The proposed approach outperforms existing methods in 84.33% of the trials. |
LegalReasoner: Step-wised Verification-Correction for Legal Judgment Reasoning (2025.acl-long)
Copied to clipboard
Weijie Shi, Han Zhu, Jiaming Ji, Mengze Li, Jipeng Zhang, Ruiyuan Zhang, Jia Zhu, Jiajie Xu, Sirui Han, Yike Guo
| Challenge: | Existing legal judgment prediction methods struggle with logical errors when conducting complex legal reasoning. |
| Approach: | They propose a method which enhances LJP reliability through step-wise verification and correction of the reasoning process. |
| Outcome: | The proposed model significantly improves concordance with court decisions from 72.37 to 80.27 on LLAMA-3.1-70B. |
PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference (2025.acl-long)
Copied to clipboard
Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Alex Qiu, Jiayi Zhou, Kaile Wang, Boxun Li, Sirui Han, Yike Guo, Yaodong Yang
| Challenge: | Using large-scale annotation data, large language models can generate noise, errors and biases, leading to unexpected behaviours. |
| Approach: | They propose a dataset to promote safety alignment in large language models . they separate helpfulness and harmlessness annotations for question-answering pairs . |
| Outcome: | The proposed dataset provides 44.6k prompts and 265k question-answer pairs with safety meta-labels for 19 harm categories and three severity levels, with answers generated by Llama-family models. |
When Slower Isn’t Truer: Inverse Scaling Law of Truthfulness in Multimodal Reasoning (2026.findings-acl)
Copied to clipboard
Sitong Fang, Wenjing Cao, Jiahao Li, Xuyao Wang, Chi-Min Chan, Sirui Han, Juntao Dai, Yike Guo, Yaodong Yang, Jiaming Ji
| Challenge: | a study of slow reasoning models for multimodal reasoning finds that they are more prone to fabricating plausible yet false details when confronted with incomplete or misleading visual inputs. |
| Approach: | They conduct the first systematic study of the inverse scaling law in slow-thinking paradigms for multimodal reasoning. |
| Outcome: | The findings suggest that slower reasoning models are more prone to fabricating false details . the study analyzed 5,000-sample hierarchical prompt dataset by 50 participants . |
Ruler: A Model-Agnostic Method to Control Generated Length for Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models struggle to meet user’s needs when required to generate responses of a specific length due to their inherent difficulty in accurately perceiving numerical constraints. |
| Approach: | They propose a Target Length Generation Task and propose RULER, a model-agnostic approach that controls generated length for large language models. |
| Outcome: | The proposed model-agnostic approach improves instruction-following ability of large language models under length-constrained instructions and can generate appropriate MLT when length constraints are not explicitly provided. |
Language Models Resist Alignment: Evidence From Data Compression (2025.acl-long)
Copied to clipboard
Jiaming Ji, Kaile Wang, Tianyi Alex Qiu, Boyuan Chen, Jiayi Zhou, Changye Li, Hantao Lou, Josef Dai, Yunhuai Liu, Yaodong Yang
| Challenge: | Large language models (LLMs) may exhibit undesirable behaviors due to the inevitable biases and harmful content present in training. |
| Approach: | They propose to investigate the elasticity of large language models by examining their performance. |
| Outcome: | The proposed model performance declines rapidly before reverting to the pre-training distribution, the authors show . the proposed model weight and code are available at pku-lm-res ist-alignment.github.io. |
Making RALM Robust to Irrelevant Contexts via Layer Knowledge Guided Attention (2025.findings-acl)
Copied to clipboard
Weijie Shi, Hao Chen, Jiaming Li, Yao Zhao, Yazhong Zhang, Qijin Chen, Jipeng Zhang, Ruiyuan Zhang, Jia Zhu, Jiajie Xu, Xiaofang Zhou
| Challenge: | Large language models (LLMs) face factual hallucination and knowledge obsolescence when tackling knowledge-intensive tasks. |
| Approach: | They propose a layer-knowledge guided attention method which harnesses the layer-wise knowledge of large language models to optimize per-layer attention on useful passages. |
| Outcome: | The proposed method outperforms existing methods on RALM benchmarks. |
LongTableBench: Benchmarking Long-Context Table Reasoning across Real-World Formats and Domains (2025.findings-emnlp)
Copied to clipboard
Liyao Li, Jiaming Tian, Hao Chen, Wentao Ye, Chao Ye, Haobo Wang, Ningtao Wang, Xing Fu, Gang Chen, Junbo Zhao
| Challenge: | Evaluating 52 LLMs reveals that only the strongest models maintain robust performance under increasing context lengths and format diversity. |
| Approach: | They propose a benchmark for evaluating long-context reasoning over semi-structured tables across diverse formats, tasks, and domains. |
| Outcome: | The proposed model outperforms compression-based approaches on tasks requiring semantic integration. |
SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation (2026.acl-long)
Copied to clipboard
Hui Wang, Jinghua Zhao, Yifan Yang, Shujie Liu, Junyang Chen, Yanzhe Zhang, Shiwan Zhao, Jinyu Li, Jiaming Zhou, Haoqin Sun, Yan Lu, Yong Qin
| Challenge: | Existing methods for evaluating the perceptual quality of synthetic speech are limited due to the complexity of perceptual quality factors and the diversity of speech generation tasks. |
| Approach: | They propose a new paradigm for enabling large language models to conduct structured speech quality evaluation using a large-scale dataset. |
| Outcome: | The proposed model performs well across tasks and languages. |