Papers by Weile Li
ModRWKV: Transformer Multimodality in Linear Time (2025.emnlp-main)
Copied to clipboard
| Challenge: | Currently, multimodal studies are based on large language models with quadratic-complexity Transformer architectures. |
| Approach: | They propose a decoupled multimodal framework built upon the RWKV7 architecture as its LLM backbone and a lightweight architecture to achieve multi-source information fusion. |
| Outcome: | The proposed framework achieves multi-source information fusion through dynamically adaptable heterogeneous modality encoders. |
WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling (2026.findings-acl)
Copied to clipboard
Jiacheng Li, Jianchao Tan, Zhidong Yang, Pingwei Sun, Feiye Huo, Jiayu Qin, Xiangyu Zhang, Maoxin He, Guangming Tan, Weile Jia, Xunliang Cai, Tong Zhao
| Challenge: | Recent advances in training optimization for Transformer-based large language models lack systematic optimization of weight patterns during training. |
| Approach: | They propose a Weight Scaling method that rescales weights while preserving model outputs to improve model training efficiency and model quality. |
| Outcome: | The proposed method significantly improves convergence quality and loss reduction in LLMs with Grouped Query Attention architectures and LoRA fine-tuning tasks. |