Papers by Chuyi Tan
LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient (2026.acl-long)
Copied to clipboard
Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Using generic and efficient benchmark generators, human annotators are limited by inefficiency . current benchmark generator methods rely on seed signals, leading to long cycles and high costs . |
| Approach: | They propose a framework to evaluate LLMs as generic benchmark generators and integrate them as BenchMaker. |
| Outcome: | The proposed framework achieves comparable performance to human-annotated benchmarks on most metrics. |
Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoning (2025.findings-naacl)
Copied to clipboard
Xinglin Wang, Shaoxiong Feng, Yiwei Li, Peiwen Yuan, Yueqi Zhang, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Existing decoding strategies for chain-of-thought reasoning do not exploit prior information about question difficulty. |
| Approach: | They propose a decoding strategy called self-consistency to improve reasoning performance by adjusting the number of samples based on the posterior distribution of a set of pre-samples. |
| Outcome: | The proposed method outperforms baseline methods on arithmetic, commonsense and symbolic reasoning tasks while achieving comparable performance. |
Focused Large Language Models are Stable Many-Shot Learners (2024.emnlp-main)
Copied to clipboard
Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Chuyi Tan, Boyuan Pan, Heda Wang, Yao Hu, Kan Li
| Challenge: | In-Context Learning (ICL) enables large language models to achieve rapid task adaptation by learning from demonstrations. |
| Approach: | They propose a training-free method that disperses model attention from the query . they propose 'focus' search strategy that uses model perplexity to ensure sufficient attention . |
| Outcome: | The proposed method achieves an average performance improvement of 5.2% over vanilla ICL and scales well with many-shot demonstrations. |
InsBank: Evolving Instruction Subset for Ongoing Alignment (2025.findings-emnlp)
Copied to clipboard
Jiayi Shi, Yiwei Li, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Yueqi Zhang, Chuyi Tan, Boyuan Pan, Huan Ren, Yao Hu, Kan Li
| Challenge: | Recent studies emphasize that quality and diversity of instruction data are more crucial than quantity, highlighting the need to select diverse, high-quality subsets to reduce training costs. |
| Approach: | They propose to use a continuously updated repository to integrate the latest valuable instruction data with a progressive evolution framework to evolve InsBank over time. |
| Outcome: | The proposed framework outperforms baselines in InsBank evolution and extracts budget-specific subsets. |
Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation (2025.acl-long)
Copied to clipboard
Peiwen Yuan, Yueqi Zhang, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Existing efficient methods estimate performance of models on large benchmarks, but these methods rely on the assumption that target models have high prediction consistency with source models. |
| Approach: | They propose a method that conducts customized evaluation tailored to each target model. |
| Outcome: | The proposed method reduces the MAE of estimates by 31.4% on benchmarks across 300 models. |
Speculative Decoding for Multi-Sample Inference (2025.findings-emnlp)
Copied to clipboard
Yiwei Li, Jiayi Shi, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Yueqi Zhang, Ji Zhang, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Speculative decoding method exploits consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or external databases. |
| Approach: | They propose a speculative decoding method that exploits the consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or external databases. |
| Outcome: | The proposed method exploits the intrinsic consensus of parallel reasoning paths to synthesize high-quality draft tokens without auxiliary models or databases. |
From Sub-Ability Diagnosis to Human-Aligned Generation: Bridging the Gap for Text Length Control via MarkerGen (2025.acl-long)
Copied to clipboard
Peiwen Yuan, Chuyi Tan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Existing methods to control text length are lacking in LCTG, posing a major limitation for practical applications. |
| Approach: | They propose a plug-and-play approach that decomposes LCTG sub-abilities with human patterns as reference and performs detailed error analysis. |
| Outcome: | The proposed method significantly improves LCTG across various settings, exhibiting outstanding effectiveness and generalizability. |
Revisiting Self-Consistency from Dynamic Distributional Alignment Perspective on Answer Aggregation (2025.findings-acl)
Copied to clipboard
Yiwei Li, Ji Zhang, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Jiayi Shi, Yueqi Zhang, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Existing studies on self-consistency show that it improves reasoning abilities by aggregating diverse stochastic samples. |
| Approach: | They propose a confidence-driven mechanism that dynamically calibrates temperature to align with high probability modes. |
| Outcome: | The proposed method outperforms fixed-diversity baselines on reasoning tasks and improves both average and best-case performance. |