Papers by Yizhe Xiong
Temporal Scaling Law for Large Language Models (2025.emnlp-main)
Copied to clipboard
Yizhe Xiong, Xiansheng Chen, Xin Ye, Hui Chen, Zijia Lin, Haoran Lian, Zhenpeng Su, Wei Huang, Jianwei Niu, Jungong Han, Guiguang Ding
| Challenge: | Existing studies have found that the test loss of LLMs scales as power-laws with model size, computational budget, and dataset size. |
| Approach: | They propose a concept of Temporal Scaling Law to study test loss of LLMs . they break down test loss into fine-grained token positions and develop a dynamic hyperbolic-law . |
| Outcome: | The proposed model predicts the test loss of LLMs as the training steps scale up. |
CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts (2025.naacl-long)
Copied to clipboard
Zhenpeng Su, Xing W, Zijia Lin, Yizhe Xiong, Minxuan Lv, Guangyuan Ma, Hui Chen, Songlin Hu, Guiguang Ding
| Challenge: | Large language models (LLMs) have been attracting much attention due to their impressive performance in all kinds of downstream tasks. |
| Approach: | They propose a mix-of-experts model that allows the model size to grow without raising training costs. |
| Outcome: | The proposed model outperforms existing models in perplexity and robustness tests. |
Mitigating Hallucinations in Multi-modal Large Language Models via Image Token Attention-Guided Decoding (2025.naacl-long)
Copied to clipboard
| Challenge: | Multi-modal large language models (MLLMs) generate plausible but incorrect content, resulting in hallucinations . recent advances in MLLM technology have demonstrated their outstanding performance in a variety of visual tasks, such as object detection. |
| Approach: | They propose a plug-and-play method which leverages MLLMs’ internal representations to mitigate hallucinations by analyzing input and output tokens. |
| Outcome: | The proposed method exploits MLLMs’ internal representations to mitigate hallucinations. |
Bridging the Training-Inference Gap for Dense Phrase Retrieval (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for building dense retrievers are often misaligned and do not reflect retrieval scenario at inference time. |
| Approach: | They propose a way to validate dense retrievers using a small subset of the entire corpus. |
| Outcome: | The proposed model improves top-1 phrase retrieval accuracy by 2 3 points and top-20 passage retrieval by 2 4 points for open-domain question answering. |
Fast Quiet-STaR: Thinking Without Thought Tokens (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models have achieved impressive performance across a range of tasks, but further gains require more than scaling up model sizes or training data. |
| Approach: | They propose a method that gradually reduces the number of thought tokens . this method allows models to internalize more abstract reasoning processes . |
| Outcome: | The proposed framework preserves the benefits of token-level reasoning while reducing computational cost. |
Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models (2025.coling-main)
Copied to clipboard
Haoran Lian, Junmin Chen, Wei Huang, Yizhe Xiong, Wenping Hu, Guiguang Ding, Hui Chen, Jianwei Niu, Zijia Lin, Fuzheng Zhang, Di Zhang
| Challenge: | Recent studies show that Large language models struggle with handling long token sequences due to limited training context size. |
| Approach: | They propose a single-stage continual pretraining method to equip LLMs with long context modeling capabilities. |
| Outcome: | The proposed method outperforms existing methods on 4 language modeling benchmarks. |
DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs (2025.emnlp-main)
Copied to clipboard
Minxuan Lv, Zhenpeng Su, Leiyu Pan, Yizhe Xiong, Zijia Lin, Hui Chen, Wei Zhou, Jungong Han, Guiguang Ding, Wenwu Ou, Di Zhang, Kun Gai, Songlin Hu
| Challenge: | Existing sparsification methods like pruning can lose model knowledge through parameter removal. |
| Approach: | They propose a novel approach that achieves sparsification by partitioning pre-trained FFN layers into computational blocks. |
| Outcome: | The proposed approach achieves superior performance across language modeling and downstream tasks under equivalent computational constraints. |
Contrastive Multi-document Question Generation (2021.eacl-main)
Copied to clipboard
Woon Sang Cho, Yizhe Zhang, Sudha Rao, Asli Celikyilmaz, Chenyan Xiong, Jianfeng Gao, Mengdi Wang, Bill Dolan
| Challenge: | Multi-document question generation focuses on generating a question that covers the common aspect of multiple documents, but a naive model trained only using the targeted document set may generate too generic questions that cover a larger scope than delineated by the document set. |
| Approach: | They propose a contrastive learning strategy where given ‘positive’ and ‘negative’ sets of documents, generate a question that is closely related to the ‘positive' set but far away from the ‘negative' set. |
| Outcome: | The proposed model significantly outperforms several strong baselines, as measured by automatic metrics and human evaluation. |