Papers by Jiayi Shi
LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient (2026.acl-long)
Copied to clipboard
Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Using generic and efficient benchmark generators, human annotators are limited by inefficiency . current benchmark generator methods rely on seed signals, leading to long cycles and high costs . |
| Approach: | They propose a framework to evaluate LLMs as generic benchmark generators and integrate them as BenchMaker. |
| Outcome: | The proposed framework achieves comparable performance to human-annotated benchmarks on most metrics. |
Computer Assisted Translation with Neural Quality Estimation and Automatic Post-Editing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Using neural machine translation to approximate human parity is difficult due to the lack of parallel training corpora. |
| Approach: | They propose an end-to-end deep learning framework for quality estimation and automatic post-editing of machine translation output. |
| Outcome: | The proposed framework achieves state-of-the-art performance on the English–German dataset and human translators can significantly expedite their post-editing processing with the model. |
InsBank: Evolving Instruction Subset for Ongoing Alignment (2025.findings-emnlp)
Copied to clipboard
Jiayi Shi, Yiwei Li, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Yueqi Zhang, Chuyi Tan, Boyuan Pan, Huan Ren, Yao Hu, Kan Li
| Challenge: | Recent studies emphasize that quality and diversity of instruction data are more crucial than quantity, highlighting the need to select diverse, high-quality subsets to reduce training costs. |
| Approach: | They propose to use a continuously updated repository to integrate the latest valuable instruction data with a progressive evolution framework to evolve InsBank over time. |
| Outcome: | The proposed framework outperforms baselines in InsBank evolution and extracts budget-specific subsets. |
Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation (2025.acl-long)
Copied to clipboard
Peiwen Yuan, Yueqi Zhang, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Existing efficient methods estimate performance of models on large benchmarks, but these methods rely on the assumption that target models have high prediction consistency with source models. |
| Approach: | They propose a method that conducts customized evaluation tailored to each target model. |
| Outcome: | The proposed method reduces the MAE of estimates by 31.4% on benchmarks across 300 models. |
Projecting Out the Malice: A Global Subspace Approach to LLM Detoxification (2026.acl-long)
Copied to clipboard
Zenghao Duan, Zhiyi Yin, Zhichao Shi, Liang Pang, Shaoling Jing, Zihe Huang, Jiayi Wu, Yu Yan, Jingcheng Deng, Huawei Shen, Xueqi Cheng
| Challenge: | Large language models (LLMs) exhibit exceptional performance but pose inherent risks of generating toxic content. |
| Approach: | They propose a method that removes toxic subspaces from FFN parameters . they propose to use a lightweight method to eliminate toxic subespaces . |
| Outcome: | The proposed method achieves SOTA detoxification while preserving general capabilities without large-scale retraining. |
Speculative Decoding for Multi-Sample Inference (2025.findings-emnlp)
Copied to clipboard
Yiwei Li, Jiayi Shi, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Yueqi Zhang, Ji Zhang, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Speculative decoding method exploits consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or external databases. |
| Approach: | They propose a speculative decoding method that exploits the consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or external databases. |
| Outcome: | The proposed method exploits the intrinsic consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or databases. |
Beyond Glass-Box Features: Uncertainty Quantification Enhanced Quality Estimation for Neural Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Quality Estimation (QE) is an essential role in applications of Machine Translation (MT). |
| Approach: | They propose to fuse uncertainty quantification into a pre-trained cross-lingual language model to predict the translation quality. |
| Outcome: | The proposed method achieves state-of-the-art on the datasets of WMT 2020 QE shared task. |
From Sub-Ability Diagnosis to Human-Aligned Generation: Bridging the Gap for Text Length Control via MarkerGen (2025.acl-long)
Copied to clipboard
Peiwen Yuan, Chuyi Tan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Existing methods to control text length are lacking in LCTG, posing a major limitation for practical applications. |
| Approach: | They propose a plug-and-play approach that decomposes LCTG sub-abilities with human patterns as reference and performs detailed error analysis. |
| Outcome: | The proposed method significantly improves LCTG across various settings, exhibiting outstanding effectiveness and generalizability. |
Mis-prompt: Benchmarking Large Language Models for Proactive Error Handling (2025.acl-long)
Copied to clipboard
| Challenge: | Current error-handling works are performed in a passive manner, with explicit error- handling instructions. |
| Approach: | They propose a new benchmark to analyze LLMs' performance on a mis-prompt benchmark and a dataset to promote further research. |
| Outcome: | The proposed benchmark shows that current LLMs show poor performance on proactive error handling, and that SFT improves on error handling instances. |
Revisiting Self-Consistency from Dynamic Distributional Alignment Perspective on Answer Aggregation (2025.findings-acl)
Copied to clipboard
Yiwei Li, Ji Zhang, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Jiayi Shi, Yueqi Zhang, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Existing studies on self-consistency show that it improves reasoning abilities by aggregating diverse stochastic samples. |
| Approach: | They propose a confidence-driven mechanism that dynamically calibrates temperature to align with high probability modes. |
| Outcome: | The proposed method outperforms fixed-diversity baselines on reasoning tasks and improves both average and best-case performance. |
Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing code generation benchmarks neglect flowchart-based code generation . existing benchmarks lack flowcharting-based evaluation, limiting the potential of large language models and minimizing human error. |
| Approach: | They propose to use flowcharts to evaluate existing LLMs' code generation capabilities. |
| Outcome: | The proposed benchmarks show that the supervised fine-tuning technique contributes greatly to the models’ performance. |
Concise Math Reasoning via Difficulty-Aware Distillation (2026.findings-acl)
Copied to clipboard
Yifan Wu, Jingze Shi, Bingheng Wu, Jiayi Zhang, Xiaotian Lin, Yizhang Zhu, Zhaoyang Yu, Bang Liu, Chenglin Wu, Nan Tang, Yuyu Luo
| Challenge: | Human experts tackle difficult math problems by identifying and executing a few pivotal steps rather than listing every intermediate thought. |
| Approach: | They propose a method for producing training data that mirrors concise human reasoning by rewriting a problem's solution to retain only the essential steps. |
| Outcome: | The proposed method outperforms models trained on 800k long CoT and cuts training and inference costs. |