Papers by Jiayi Pan
LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient (2026.acl-long)
Copied to clipboard
Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Using generic and efficient benchmark generators, human annotators are limited by inefficiency . current benchmark generator methods rely on seed signals, leading to long cycles and high costs . |
| Approach: | They propose a framework to evaluate LLMs as generic benchmark generators and integrate them as BenchMaker. |
| Outcome: | The proposed framework achieves comparable performance to human-annotated benchmarks on most metrics. |
DANLI: Deliberative Agent for Following Natural Language Instructions (2022.emnlp-main)
Copied to clipboard
Yichi Zhang, Jianing Yang, Jiayi Pan, Shane Storks, Nikhil Devraj, Ziqiao Ma, Keunwoo Yu, Yuwei Bao, Joyce Chai
| Challenge: | Recent work on embodied AI agents that can perform tasks by following human language instructions is limited by reactive methods, which are insufficient for long-horizon complex tasks. |
| Approach: | They propose a neuro-symbolic deliberative agent that, while following language instructions, proactively applies reasoning and planning based on its neural and symbolic representations acquired from past experience. |
| Outcome: | The proposed agent achieves greater than 70% improvement over reactive baselines on the challenging TEACh benchmark. |
InsBank: Evolving Instruction Subset for Ongoing Alignment (2025.findings-emnlp)
Copied to clipboard
Jiayi Shi, Yiwei Li, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Yueqi Zhang, Chuyi Tan, Boyuan Pan, Huan Ren, Yao Hu, Kan Li
| Challenge: | Recent studies emphasize that quality and diversity of instruction data are more crucial than quantity, highlighting the need to select diverse, high-quality subsets to reduce training costs. |
| Approach: | They propose to use a continuously updated repository to integrate the latest valuable instruction data with a progressive evolution framework to evolve InsBank over time. |
| Outcome: | The proposed framework outperforms baselines in InsBank evolution and extracts budget-specific subsets. |
Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation (2025.acl-long)
Copied to clipboard
Peiwen Yuan, Yueqi Zhang, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Existing efficient methods estimate performance of models on large benchmarks, but these methods rely on the assumption that target models have high prediction consistency with source models. |
| Approach: | They propose a method that conducts customized evaluation tailored to each target model. |
| Outcome: | The proposed method reduces the MAE of estimates by 31.4% on benchmarks across 300 models. |
World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | GOVA examines grounding and bootstrapping in open-world language learning. |
| Approach: | They propose a visually-grounded language model that uses grounding as an objective . they propose GOVA to investigate grounding and bootstrapping in open-world language learning . |
| Outcome: | The proposed model is faster and faster grounded than previous models, the authors show . they show that grounding helps the model to learn unseen words more rapidly and robustly . |
Speculative Decoding for Multi-Sample Inference (2025.findings-emnlp)
Copied to clipboard
Yiwei Li, Jiayi Shi, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Yueqi Zhang, Ji Zhang, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Speculative decoding method exploits consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or external databases. |
| Approach: | They propose a speculative decoding method that exploits the consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or external databases. |
| Outcome: | The proposed method exploits the intrinsic consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or databases. |
From Sub-Ability Diagnosis to Human-Aligned Generation: Bridging the Gap for Text Length Control via MarkerGen (2025.acl-long)
Copied to clipboard
Peiwen Yuan, Chuyi Tan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Existing methods to control text length are lacking in LCTG, posing a major limitation for practical applications. |
| Approach: | They propose a plug-and-play approach that decomposes LCTG sub-abilities with human patterns as reference and performs detailed error analysis. |
| Outcome: | The proposed method significantly improves LCTG across various settings, exhibiting outstanding effectiveness and generalizability. |
Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement Learning with Verifiable Rewards (RLVR) has shown significant promise for enhancing the reasoning capabilities of large language models (LLMs). |
| Approach: | They propose a model-free method that refines credit assignment by leveraging the model's internal uncertainty signals. |
| Outcome: | Extensive experiments on five mathematical reasoning benchmarks show that the proposed method outperforms strong RLVR baselines on multiple model scales, including 1.5B and 7B. |
Revisiting Self-Consistency from Dynamic Distributional Alignment Perspective on Answer Aggregation (2025.findings-acl)
Copied to clipboard
Yiwei Li, Ji Zhang, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Jiayi Shi, Yueqi Zhang, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Existing studies on self-consistency show that it improves reasoning abilities by aggregating diverse stochastic samples. |
| Approach: | They propose a confidence-driven mechanism that dynamically calibrates temperature to align with high probability modes. |
| Outcome: | The proposed method outperforms fixed-diversity baselines on reasoning tasks and improves both average and best-case performance. |
Reward Generalization in RLHF: A Topological Perspective (2025.findings-acl)
Copied to clipboard
Tianyi Alex Qiu, Fanzhi Zeng, Jiaming Ji, Dong Yan, Kaile Wang, Jiayi Zhou, Yang Han, Josef Dai, Xuehai Pan, Yaodong Yang
| Challenge: | Existing alignment methods share a common topology of information flow, but their alternatives have not been thoroughly explored. |
| Approach: | They propose a theory of reward generalization in reinforcement learning from human feedback . they propose induced Bayesian networks to model the impact of dataset topologies on reward generalisation . |
| Outcome: | The proposed method achieves an average win rate of 65% on three NLP tasks. |
Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Visual illusions are a phenomenon that is often seen in human perception but are not always faithful to the physical world. |
| Approach: | They build a dataset containing five types of visual illusions and formulate four tasks to examine visual illusion in state-of-the-art VLMs. |
| Outcome: | The proposed dataset reveals that larger models are closer to human perception and more susceptible to visual illusions. |