Papers by Houyi Li
Multi-matrix Factorization Attention (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing variants for Multi-Head Attention (MHA) fail to maintain strong performance under stringent Key-Value cache (KV cache) constraints. |
| Approach: | They propose to use multi-matrix factorization attention and MFA-Key-reuse attention architectures to increase model capacity under tight KV cache constraints. |
| Outcome: | The proposed architecture outperforms existing methods while reducing KV cache usage by 56% and 93.7% in large-scale experiments. |
Scaling Laws for Code: A More Data-Hungry Regime (2026.acl-long)
Copied to clipboard
Xianzhen Luo, Wenzhen Zheng, Qingfu Zhu, Rongyi Zhang, Houyi Li, Siming Huang, YuanTao Fan, Wanxiang Che
| Challenge: | Code Large Language Models (LLMs) are revolutionizing software engineering, but scaling laws are primarily analyzed on Natural Language (NL). |
| Approach: | They fit Chinchilla law and Farsser law to test scaling laws for code . they find code is more data-hungry and requires higher data-to-parameter ratio . |
| Outcome: | The proposed scaling laws show that the more expressive Farsser law offers greater accuracy and scales with model size. |
OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models (2025.acl-long)
Copied to clipboard
Siming Huang, Tianhao Cheng, Jason Klein Liu, Weidi Xu, Jiaran Hao, Liuyihan Song, Yang Xu, Jian Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Xianzhen Luo, Qiufeng Wang, YuanTao Fan, Qingfu Zhu, Zhaoxiang Zhang, Yang Gao, Jie Fu, Qian Liu, Houyi Li, Ge Zhang, Yuan Qi, Xu Yinghui, Wei Chu, Zili Wang
| Challenge: | Code LLMs lack reproducible data pipelines and training protocols for reproducible advancements in code intelligence. |
| Approach: | They propose a top-tier code LLM that releases model weights and inference code . reproducible data pipelines, rigorous experimental ablation results and training protocols are included . |
| Outcome: | The proposed model achieves comparable performance to leading models and serves as an "open cookbook" reproducible training data, rigorous experimental ablation results, and detailed training protocols are also included in the model. |