Papers by Yunfan Shao
FastMCTS: A Simple Sampling Strategy for Data Synthesis (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for generating multi-step reasoning data rely on rejection sampling, which generates trajectories independently and suffers from inefficiency and imbalanced sampling across problems of varying difficulty levels. |
| Approach: | They propose a data synthesis strategy inspired by Monte Carlo Tree Search . it offers step-level evaluation signals and promotes balanced sampling . |
| Outcome: | Experiments show that FastMCTS generates 30% more correct reasoning paths than rejection sampling. |
Balanced Data Sampling for Language Model Training with Clustering (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are a fundamental part of the training process. |
| Approach: | They propose to use clustering to balance the text distribution of training data for better model training. |
| Outcome: | Extensive experiments validate the effectiveness of ClusterClip Sampling under various training datasets and large language models. |
UnitCoder: Scalable Code Synthesis from Pre-training Corpora (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel at code understanding and generation, yet code generation remains a challenge. |
| Approach: | They propose a model that supervises pre-training data quality through automatically generated unit tests while ensuring correctness via an iterative fix and refine flow. |
| Outcome: | The proposed model improves performance on a large dataset with high quality pre-training data. |
Character-LLM: A Trainable Agent for Role-Playing (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be used to simulate human behaviors . a recent study suggests that LLMs can be more effective at generating human behavior . |
| Approach: | They propose to use large language models to train agents with the profile, experience, and emotional states of a specific person instead of using limited prompts to instruct ChatGPT API. |
| Outcome: | The proposed model trains agents with the profile, experience, and emotional states of a specific person instead of using limited prompts to instruct ChatGPT API. |
Accelerating BERT Inference for Sequence Labeling via Early-Exit (2021.acl-long)
Copied to clipboard
| Challenge: | Existing early-exit mechanisms are designed for sequence-level tasks, rather than sequence labeling. |
| Approach: | They propose to extend sentence-level early-exit to accelerate inference of PTMs . they propose a token-level mechanism that allows partial tokens to exit early at different layers . |
| Outcome: | The proposed approach can save up to 66%75% inference cost with minimal performance degradation. |
PerturbScore: Connecting Discrete and Continuous Perturbations in NLP (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Natural language processing (NLP) applications are growing rapidly due to discrete nature of texts. |
| Approach: | They propose to connect discrete perturbations with continuous perturbations to help understand discrete ones in NLP models. |
| Outcome: | The proposed method surpasses methods used in discrete perturbation measuring and can be generalized to different datasets, perturbation methods. |
CoLAKE: Contextualized Language and Knowledge Embedding (2020.coling-main)
Copied to clipboard
| Challenge: | Existing models for integrating factual knowledge into pre-trained language models are shallow, static, and separately pre-train entities. |
| Approach: | They propose a method which integrates knowledge contexts from large-scale knowledge bases into a unified data structure. |
| Outcome: | The proposed model outperforms existing models on knowledge-driven tasks and knowledge probing tasks. |
Star-Transformer (N19-1)
Copied to clipboard
| Challenge: | Existing models with fully-connected attention connections are heavy and require large training data. |
| Approach: | They propose a lightweight alternative to the Transformer by sparsifying the fully-connected structure with a star-shaped topology. |
| Outcome: | The proposed model achieves significant performance improvements on 22 datasets on four tasks. |
Unified Active Retrieval for Retrieval Augmented Generation (2024.findings-emnlp)
Copied to clipboard
Qinyuan Cheng, Xiaonan Li, Shimin Li, Qin Zhu, Zhangyue Yin, Yunfan Shao, Linyang Li, Tianxiang Sun, Hang Yan, Xipeng Qiu
| Challenge: | Existing active retrieval methods struggle with handling various types of instructions. |
| Approach: | They propose a unified active retrieval framework for retrieval-augmented generation . they propose to combine four orthogonal criteria into plug-and-play classification tasks . |
| Outcome: | The proposed framework outperforms existing methods on four representative types of user instructions on four types of instructions. |
Case2Code: Scalable Synthetic Data for Code Generation (2025.coling-main)
Copied to clipboard
Yunfan Shao, Linyang Li, Yichuan Ma, Peiji Li, Demin Song, Qinyuan Cheng, Shimin Li, Xiaonan Li, Pengyu Wang, Qipeng Guo, Hang Yan, Xipeng Qiu, Xuanjing Huang, Dahua Lin
| Challenge: | Large Language Models (LLMs) have shown outstanding breakthroughs in code generation. |
| Approach: | They propose a case-to-code induction task that exploits the expressiveness and correctness of programs by incorporating LLMs into their training. |
| Outcome: | The proposed task improves distribution case-to-code induction and various coding generation tasks. |