Papers by Zhi Yang
DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories (2024.findings-acl)
Copied to clipboard
Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li, Bin Gu, Mengfei Yang
| Challenge: | Existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of Large Language Models (LLMs). |
| Approach: | They propose a repository-level benchmark named DevEval to evaluate LLMs' coding abilities in real-world code repositories. |
| Outcome: | The proposed benchmarks show that the LLMs perform better in real-world code repositories than existing benchmarks. |
From Detection to Understanding: Multi-Turn Reasoning for Video Misinformation Analysis (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on binary veracity judgments and do not evaluate process-level justifications for misinformation models. |
| Approach: | They propose a video misinformation analysis benchmark that assesses reasoning in video misinterpretation. |
| Outcome: | The proposed framework improves reasoning accuracy and explanation quality compared to existing models . it covers 12 fine-grained deception categories and progresses from perceptual attribution to intent and persuasion analysis. |
BrailleLLM: Braille Instruction Tuning with Large Language Models for Braille Domain Tasks (2025.emnlp-main)
Copied to clipboard
Tianyuan Huang, Zepeng Zhu, Hangdi Xing, Zirui Shao, Zhi Yu, Chaoxiong Yang, Jiaxian He, Xiaozhong Liu, Jiajun Bu
| Challenge: | Existing Braille research focuses on isolated tasks while mixed-content Braille tasks face data scarcity and ambiguities. |
| Approach: | They propose a syntax tree-based augmentation method tailored for Braille data. |
| Outcome: | The proposed method improves Braille translation, formula-to-Braille conversion, and mixed-text translation. |
TabularMath: Understanding Math Reasoning over Tables with Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Mathematical reasoning has long been a key benchmark for evaluating large language models. |
| Approach: | They propose a framework that transforms math word problems into scalable tabular reasoning tasks. |
| Outcome: | The proposed framework transforms math word problems into scalable and verified tabular reasoning tasks. |
Guide the Many-to-One Assignment: Open Information Extraction via IoU-aware Optimal Transport (2023.acl-long)
Copied to clipboard
Kaiwen Wei, Yiran Yang, Li Jin, Xian Sun, Zequn Zhang, Jingyuan Zhang, Xiao Li, Linhao Zhang, Jintao Liu, Guo Zhi
| Challenge: | Open Information Extraction (OIE) aims to extract structured information from text without the limitations of close ontology. |
| Approach: | They propose a method to assign ground truth labels to parallelly generated tuple proposals . they leverage intersection-over-union (IoU) as assignment quality measurement . |
| Outcome: | The proposed method outperforms the state-of-the-art models on three benchmarks. |
On Safety Risks in Experience-Driven Self-Evolving Agents (2026.findings-acl)
Copied to clipboard
Weixiang Zhao, Yichen Zhang, Yingshuo Wang, Yang Deng, Yanyan Zhao, Xuda Zhi, Yongbo Huang, Hao He, Wanxiang Che, Bing Qin, Ting Liu
| Challenge: | Experience-driven self-evolution has emerged as a promising paradigm for improving the autonomy of large language model agents, yet its reliance on self-curated experience introduces underexplored safety risks. |
| Approach: | They investigate how experience accumulation and utilization in self-evolving agents affect safety performance across web-based and embodied environments. |
| Outcome: | The findings expose inherent limitations of current self-evolving agents and call for more principled strategies to ensure safe and reliable adaptation. |
AutoEvolve: Automatically Evolving Queries for Applicable and Scalable Retrieval-Augmented Generation Benchmarking (2025.findings-emnlp)
Copied to clipboard
Ding-Chu Zhang, Xiaowen Zhang, Yue Fei, Renjun Hu, Xiao-Wen Yang, Zhi Zhou, Baixuan Li, Yu-Feng Li, Xing Shi, Wei Lin
| Challenge: | Existing automated generation methods exhibit Weak Applicability and Weak Scalability . existing methods are limited by their reliance on metadata from specific corpora . |
| Approach: | They propose an approach to generate scalable RAG benchmarks using corpus-agnostic methods . they propose a difficulty-guided metric that directs query evolution process . |
| Outcome: | The proposed approach evolves queries significantly more challenging than existing methods . it is able to dynamically increase difficulty, limiting scalability of the query . |
Double-Checker: Large Language Model as a Checker for Few-shot Named Entity Recognition (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have demonstrated remarkable performance on few-shot Named Entity Recognition tasks due to the high cost of obtaining high-quality labeled data. |
| Approach: | They propose to decompose the task into entity span detection and entity type classification using a type-independent entity span detector and then classify the detected spans based on their types. |
| Outcome: | The proposed method consistently yields improvements over two baseline approaches. |
VCSearch: Bridging the Gap Between Well-Defined and Ill-Defined Problems in Mathematical Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have improved the performance of Large language models on well-defined mathematical benchmarks, but they often overlook ill-defined problems. |
| Approach: | They develop a large-scale benchmark that contains over 5,000 ill-defined mathematical problems. |
| Outcome: | The proposed framework improves the accuracy of identifying unsolvable problems by at least 12% across different LLMs, thus achieving stronger robust mathematical reasoning ability. |
Red Teaming Large Reasoning Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large Reasoning Models (LRMs) have emerged as a powerful advancement in multi-step reasoning tasks, but they introduce safety and reliability risks, such as CoT-hijacking and prompt-induced inefficiencies. |
| Approach: | They propose a unified benchmark to assess the trustworthiness of Large Reasoning Models. |
| Outcome: | The proposed benchmark evaluates truthfulness, safety and efficiency on 26 models. |
Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Considering the vast size and wide-ranging sources of LLMs’ training data, it could explicitly or implicitly include test data. |
| Approach: | They propose a Contamination Detection via output Distribution (CDD) which detects data contamination only by identifying the peakedness of LLM's output distribution. |
| Outcome: | The proposed method improves performance by 21.8%-30.2% on humanEval and TED: trustworthy evaluation via output distribution. |
Basic Reading Distillation (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models require high computational resources which limits their deployment in real-world applications. |
| Approach: | They propose to distill large language models into smaller language models by either knowledge distillation or task distillation. |
| Outcome: | The proposed model outperforms or performs comparable to over 20x bigger LLMs on language inference benchmarks and BIG-bench tasks. |
LESA: Learnable LLM Layer Scaling-Up (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for depth scaling-up rely on empirical heuristic rules for layer duplication, resulting in poor initialization and slower convergence during continual pre-training. |
| Approach: | They propose a method for learning latent parameters between layers by concatenating parameters from each layer and applying Singular Value Decomposition. |
| Outcome: | Experiments show that LESA outperforms baseline models with less than half the cost of existing methods. |
FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models (2025.naacl-long)
Copied to clipboard
Xin Guo, Haotian Xia, Zhaowei Liu, Hanyang Cao, Zhi Yang, Zhiqiang Liu, Sizhe Wang, Jinyi Niu, Chuqi Wang, Yanhui Wang, Xiaolong Liang, Xiaoming Huang, Bing Zhu, Zhongyu Wei, Yun Chen, Weining Shen, Liwen Zhang
| Challenge: | Large language models have demonstrated outstanding performance in various natural language processing tasks, but their security capabilities in the financial domain have not been explored. |
| Approach: | They propose to use a benchmark to evaluate large language models' financial domain knowledge and practical abilities. |
| Outcome: | The proposed benchmark evaluates large language models' financial domain knowledge and practical abilities. |
KuiLeiXi: a Chinese Open-Ended Text Adventure Game (2021.acl-demo)
Copied to clipboard
Yadong Xi, Xiaoxi Mao, Le Li, Lei Lin, Yanjiang Chen, Shuhan Yang, Xuhan Chen, Kailun Tao, Zhi Li, Gongzheng Li, Lin Jiang, Siyan Liu, Zeng Zhao, Minlie Huang, Changjie Fan, Zhipeng Hu
| Challenge: | Recent advances in pre-trained language models have made it possible to generate human-like text. |
| Approach: | They propose to integrate an open-ended text adventure game in Chinese, named KuiLeiXi, where players interact with the AI until the plot goals are reached. |
| Outcome: | The proposed game lacks incentives and relies on players to explore on their own. |