Papers by Zhenpeng Su
Temporal Scaling Law for Large Language Models (2025.emnlp-main)
Copied to clipboard
Yizhe Xiong, Xiansheng Chen, Xin Ye, Hui Chen, Zijia Lin, Haoran Lian, Zhenpeng Su, Wei Huang, Jianwei Niu, Jungong Han, Guiguang Ding
| Challenge: | Existing studies have found that the test loss of LLMs scales as power-laws with model size, computational budget, and dataset size. |
| Approach: | They propose a concept of Temporal Scaling Law to study test loss of LLMs . they break down test loss into fine-grained token positions and develop a dynamic hyperbolic-law . |
| Outcome: | The proposed model predicts the test loss of LLMs as the training steps scale up. |
CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts (2025.naacl-long)
Copied to clipboard
Zhenpeng Su, Xing W, Zijia Lin, Yizhe Xiong, Minxuan Lv, Guangyuan Ma, Hui Chen, Songlin Hu, Guiguang Ding
| Challenge: | Large language models (LLMs) have been attracting much attention due to their impressive performance in all kinds of downstream tasks. |
| Approach: | They propose a mix-of-experts model that allows the model size to grow without raising training costs. |
| Outcome: | The proposed model outperforms existing models in perplexity and robustness tests. |
Document Hashing with Multi-Grained Prototype-Induced Hierarchical Generative Model (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing document hashing methods only consider flat semantics of documents, preserving hierarchical semantics. |
| Approach: | They propose a hierarchical generative model that can model and leverage hierarchic semantics . they introduce hierarchically-based prototypes into the model to construct a Hierarchical prior distribution . |
| Outcome: | The proposed model outperforms baseline methods on hierarchical and flat datasets. |
Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learning (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in large language models but treats all correct solutions equally, potentially reinforcing flawed traces that arrive at correct answers by chance. |
| Approach: | They propose a method that reweights rewards by a factor approximately proportional to Evidence Gain and assigns higher weights to high-quality traces without requiring costly computation. |
| Outcome: | Experiments on mathematical reasoning benchmarks show that Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in large language models but treats all correct solutions equally. |
CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for proximal policy optimization discard valuable gradient signals from low-probability tokens due to the clipping mechanism. |
| Approach: | They propose an algorithm that reintroduces gradients from clipped tokens in native PPO in a gentle and bounded manner. |
| Outcome: | The proposed algorithm outperforms strong baselines on reasoning benchmarks on different model scales. |
DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs (2025.emnlp-main)
Copied to clipboard
Minxuan Lv, Zhenpeng Su, Leiyu Pan, Yizhe Xiong, Zijia Lin, Hui Chen, Wei Zhou, Jungong Han, Guiguang Ding, Wenwu Ou, Di Zhang, Kun Gai, Songlin Hu
| Challenge: | Existing sparsification methods like pruning can lose model knowledge through parameter removal. |
| Approach: | They propose a novel approach that achieves sparsification by partitioning pre-trained FFN layers into computational blocks. |
| Outcome: | The proposed approach achieves superior performance across language modeling and downstream tasks under equivalent computational constraints. |
MiLe Loss: a New Loss for Mitigating the Bias of Learning Difficulties in Generative Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing generative language models neglect an inherent challenge in text corpus during training, i.e., the imbalance between frequent tokens and infrequent ones. |
| Approach: | They propose a function to mitigate the imbalance between frequent and infrequent tokens . authors propose 'MiLe Loss' function to assess learning difficulty of tokens during training . |
| Outcome: | Experiments show that models with proposed model can improve on downstream benchmarks. |
Dial-MAE: ConTextual Masked Auto-Encoder for Retrieval-based Dialogue Systems (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on dialogue response selection focus on post-training and fine-tuning for cross-encoders. |
| Approach: | They propose a post-training technique tailored for dense encoders in dialogue response selection . they propose 'Dialogue Contextual Masking Auto-Encoder' which compresses dialogue semantics into dense vectors . |
| Outcome: | The proposed technique achieves state-of-the-art on two commonly evaluated benchmarks. |
Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning (2026.findings-acl)
Copied to clipboard
Zhenpeng Su, Leiyu Pan, Minxuan Lv, Tiehua Mei, Zijia Lin, Yuntao Li, Wenping Hu, Ruiming Tang, Kun Gai, Guorui Zhou
| Challenge: | Large language model post-training often adopts an off-policy training paradigm . however, the off-poliicy training model introduces distribution shifts that push the policy beyond the trust region. |
| Approach: | They propose to use the entropy ratio as a global metric to measure the relative change in policy exploration throughout updates. |
| Outcome: | Experiments show that the proposed metric improves performance across multiple benchmarks. |