Papers by Yunhua Zhou
Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law (2025.acl-long)
Copied to clipboard
Qiming Ge, Shuhao Xing, Songyang Gao, Yunhua Zhou, Yicheng Zou, Songyang Zhang, Zhi Chen, Hang Yan, Qi Zhang, Qipeng Guo, Kai Chen
| Challenge: | Large language models have demonstrated impressive performance across a wide range of tasks, but this achievement comes with the trade-off of significant computational demands. |
| Approach: | They propose a scaling law that decomposes the overall validation loss and assigns different importance weights to tokens to assess a specific meta-capability. |
| Outcome: | The proposed model can predict the loss trending of models across different levels of computation without a gap between validation loss and model's downstream capabilities. |
The Open-World Lottery Ticket Hypothesis for OOD Intent Classification (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods of Out-of-Domain intent classification lack confidence in In- and Out- of-domain intents. |
| Approach: | They propose to prune overparameterized models to provide better confidence . they extend the Lottery Ticket Hypothesis to open-world scenarios . |
| Outcome: | The proposed model can be calibrated to distinguish In- and Out-of-domain intents . the model can also improve on open-world scenarios . |
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling (2024.acl-long)
Copied to clipboard
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang, Xipeng Qiu
| Challenge: | Existing language models that use discrete representations for unified processing of various modalities are limited to text generation and do not include multimodal output. |
| Approach: | They propose a multimodal language model that utilizes discrete representations for unified processing of various modalities. |
| Outcome: | The proposed model can be trained stably without any alterations to existing models or training paradigms. |
Code Needs Comments: Enhancing Code LLMs with Comment Augmentation (2024.findings-acl)
Copied to clipboard
Demin Song, Honglin Guo, Yunhua Zhou, Shuhao Xing, Yudong Wang, Zifan Song, Wenwei Zhang, Qipeng Guo, Hang Yan, Xipeng Qiu, Dahua Lin
| Challenge: | Large Language Models (LLMs) require a deep understanding of programming languages and their correlation with natural languages (NLs). |
| Approach: | They propose a data augmentation method that generates comments for existing code and a filtering strategy that filters out code data poorly correlated with natural language. |
| Outcome: | The proposed method outperforms the model trained on the augmented data and the model further trained on data without augmentation on two widely-used programming skill benchmarks. |
Turn Waste into Worth: Rectifying Top-k Router of MoE (2024.emnlp-main)
Copied to clipboard
Zhiyuan Zeng, Qipeng Guo, Zhaoye Fei, Zhangyue Yin, Yunhua Zhou, Linyang Li, Tianxiang Sun, Hang Yan, Dahua Lin, Xipeng Qiu
| Challenge: | Top-k router suffers from redundancy computation and memory costs due to unbalanced routing . some experts are overflow, where exceeding tokens are dropped, while others are empty, which are padded with zeros, negatively impacting model performance. |
| Approach: | They propose a top-k router that is unbalanced and uses a multi-gPU system to handle dropped tokens and padding. |
| Outcome: | The proposed model surpasses the top-1 router by 4.7% in terms of performance . the top-k router suffers from redundancy computation and memory costs . |
Firewall Routing: Blocking Leads to Better Hybrid Inference for LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models have significantly enhanced performance across various NLP tasks . high computational costs and latency associated with deploying such models pose bottlenecks . |
| Approach: | They propose a dynamic hybrid inference framework that efficiently selects between a strong and a weak LLM based on the complexity of the query. |
| Outcome: | The proposed method outperforms existing routing strategies by up to 5.29% in APGR . large models often introduce higher latency, making them less suitable for real-time or resource-constrained applications. |
A Probabilistic Framework for Discovering New Intents (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for detecting unknown intents do not explore the intrinsic structure of unlabeled data. |
| Approach: | They propose a probabilistic framework for discovering intents where intent assignments are treated as latent variables. |
| Outcome: | The proposed framework can be used to discover intents with latent variables . it can be applied to three challenging real-world datasets . |
How to Set the Learning Rate for Large-Scale Pre-training? (2026.findings-acl)
Copied to clipboard
| Challenge: | Optimal configuration of the learning rate (LR) is a fundamental yet formidable challenge in large-scale pre-training. |
| Approach: | They propose a Fitting Paradigm and a Transfer Paradigme to investigate fit and transfer . they propose scalability and elucidate the reasons why module-wise parameter tuning underperforms . |
| Outcome: | The proposed model reduces the search complexity by reducing the search cost by lowering the search factor. |
Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities? (2025.acl-long)
Copied to clipboard
| Challenge: | Longer CoTs of o1-like models do not consistently enhance accuracy, causing performance degradation. |
| Approach: | They propose a method that combines parallel scaling strategies with CoT length characteristics to improve models’ test-time scalability. |
| Outcome: | The proposed method improves models’ test-time scalability compared to majority voting approaches. |
BBTv2: Towards a Gradient-Free Future with Large Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on parameter-efficient tuning (PET) only tunes a small portion of parameters while keeping most of the parameters of the LLM unchanged. |
| Approach: | They propose an improved version of Black-Box Tuning to tune PTMs through gradient descent . they prepend continuous prompts to every layer of the PTM and propose a divide-and-conquer gradient-free algorithm to optimize the prompts alternately. |
| Outcome: | The proposed method achieves comparable performance to full model tuning and state-of-the-art parameter-efficient methods under few-shot settings while maintaining much fewer tunable parameters. |
Two Birds One Stone: Dynamic Ensemble for OOD Intent Classification (2023.acl-long)
Copied to clipboard
| Challenge: | Out-of-domain (OOD) intent classification is an active field of natural language understanding . previous studies have suggested that PTMs would be "overthinking" the semantic features of the sample in the open-world scenario . |
| Approach: | They propose a method that allows the model to decide whether to make a decision on OOD classification early during inference. |
| Outcome: | The proposed method can improve inference speed and achieve significant performance improvements. |
Memorize Step by Step: Efficient Long-Context Prefilling with Incremental Memory and Decremental Chunk (2024.emnlp-main)
Copied to clipboard
Zhiyuan Zeng, Qipeng Guo, Xiaoran Liu, Zhangyue Yin, Wentao Shu, Mianqiu Huang, Bo Wang, Yunhua Zhou, Linlin Li, Qun Liu, Xipeng Qiu
| Challenge: | Existing methods to optimize LLM for long sequences for long documents are slow and consume memory. |
| Approach: | They propose a method that starts with a small memory size and gradually increases it . they propose Decremental Chunk based on Incremental Memory (IMDC) which reduces chunk size while increasing memory size . |
| Outcome: | The proposed method is faster (1.45x) and reduces GPU memory consumption by 23.3% compared to fixed-size memory. |
KNN-Contrastive Learning for Out-of-Domain Intent Classification (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for OOD intent classification are limited to regions with compact or simply-connected features, which assumes no OOD intentions reside. |
| Approach: | They propose a method that uses k-nearest neighbors to learn discriminative semantic features that are more conducive to OOD detection. |
| Outcome: | The proposed method improves OOD detection performance while requiring no restrictions on feature distribution. |
Towards Open Environment Intent Prediction (2023.findings-acl)
Copied to clipboard
| Challenge: | Out-of-Domain (OOD) Intent Classification and New Intent Discovering are two tasks in the Task-Oriented Dialogue System. |
| Approach: | They propose a task paradigm to extend Out-of-Domain (OOD) Intent Classification and New Intent Discovering tasks in the Task-Oriented Dialogue System. |
| Outcome: | The proposed scheme improves on existing OOD intent classification and discovery datasets. |
Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment (2026.acl-long)
Copied to clipboard
Yuming Yang, Mingyoung Lai, Wanxu Zhao, Xiaoran Fan, Zhiheng Xi, Mingqi Wu, Chiyue Huang, Jun Zhao, Haijun Lv, Jian Tong, Yunhua Zhou, Yicheng Zou, Qipeng Guo, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing methods assess suitability primarily through student likelihood, favoring trajectories that align closely with the student model’s current behavior but overlooking more informative ones. |
| Approach: | They propose a Rank–Surprisal Ratio metric that captures both alignment and informativeness to assess the suitability of a reasoning trajectory. |
| Outcome: | The proposed metric captures both alignment and informativeness to assess the suitability of a reasoning trajectory. |
UTC-IE: A Unified Token-pair Classification Architecture for Information Extraction (2023.acl-long)
Copied to clipboard
| Challenge: | Information Extraction (IE) tasks have been solved with different models because of their output structures. |
| Approach: | They propose a Unified Token-pair Classification architecture for Information Extraction that introduces Plusformer on top of the token-pear feature matrix. |
| Outcome: | The proposed approach outperforms task-specific and unified models on all tasks in 10 datasets and achieves better results on 2 joint IE datasets. |