Papers by Chi Wei
Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models (2025.acl-long)
Copied to clipboard
Xinlin Zhuang, Jiahui Peng, Ren Ma, Yinfan Wang, Tianyi Bai, Xingjian Wei, Qiu Jiantao, Chi Zhang, Ying Qian, Conghui He
| Challenge: | composition of pre-training datasets for large language models remains undisclosed . current methods for evaluating data quality are limited by single-dimensional evaluation or redundancy-focused strategies. |
| Approach: | They propose a multi-dimensional data selection method that integrates dimensions with existing quality metrics through learned optimal weightings. |
| Outcome: | The proposed method doubles convergence speed for 1.3B model models and improves downstream task performance by 3.23%. |
Cross-Lingual Phrase Retrieval (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual phrase retrieval learn word or sentence representations in word or sentences. |
| Approach: | They propose a cross-lingual phrase retrieval model that extracts phrase representations from unlabeled example sentences. |
| Outcome: | The proposed model outperforms state-of-the-art methods on a large-scale cross-lingual phrase retrieval dataset, showing it can perform in an unseen language pair during training. |
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2022.acl-long)
Copied to clipboard
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Bo Zheng, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, Furu Wei
| Challenge: | ELECTRA-style tasks are used to pretrain cross-lingual models for NLP tasks . masked language modeling tasks require massive computation resources, rendering such models quite expensive . |
| Approach: | They propose to use ELECTRA-style tasks to pre-train a cross-lingual language model . they propose to pretrain the model on multilingual and parallel corpora . |
| Outcome: | The proposed model outperforms baseline models on cross-lingual understanding tasks with much less computation cost. |
Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020.aacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have shown impressive results for cross-lingual transfer, but due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. |
| Approach: | They propose to transfer the knowledge from monolingual pretrained models to multilingual ones to improve zero-shot cross-lingual classification by using machine translation systems. |
| Outcome: | The proposed methods outperform vanilla multilingual fine-tuning on two cross-lingual classification benchmarks. |
mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs (2021.emnlp-main)
Copied to clipboard
Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Saksham Singhal, Xian-Ling Mao, Heyan Huang, Xia Song, Furu Wei
| Challenge: | Multilingual T5 pretrains a sequence-to-sequence model on monolingual texts, but it has shown promising results on many cross-lingual tasks. |
| Approach: | They propose a partially non-autoregressive objective for text-to-text pre-training and propose mT6 to improve cross-lingual transferability over multilingual T5. |
| Outcome: | The proposed model improves cross-lingual transferability over existing models. |
AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning (2025.emnlp-demos)
Copied to clipboard
Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, Xie Xie, Wei Zhou, Wang Xu, Zhou Su, Zhongwu Zhai, Xiaoming Liu, null Meiyudong, Jianming Xu, Hongyan Tian, Chongyi Wang, Chi Chen, Yuan Yao, Zhiyuan Liu, Maosong Sun
| Challenge: | Large language model agents have enabled GUI-based automation, but their deployment is limited by noisy data, poor generalization, and lack of support for non-English GUIs. |
| Approach: | They propose an 8B-parameter GUI agent built for robust and efficient on-device GUI interaction. |
| Outcome: | The proposed GUI agent achieves promising performance on five public benchmarks and proposed Chinese benchmark CAGUI. |
ChatMusician: Understanding and Generating Music Intrinsically with LLM (2024.findings-acl)
Copied to clipboard
Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, Liumeng Xue, Ziyang Ma, Qin Liu, Tianyu Zheng, Yizhi Li, Yinghao Ma, Yiming Liang, Xiaowei Chi, Ruibo Liu, Zili Wang, Chenghua Lin, Qifeng Liu, Tao Jiang, Wenhao Huang, Wenhu Chen, Jie Fu, Emmanouil Benetos, Gus Xia, Roger Dannenberg, Wei Xue, Shiyin Kang, Yike Guo
| Challenge: | Despite LLMs' impressive capabilities in musical knowledge, music reasoning remains an unsolved task. |
| Approach: | They propose an open-source large language model (LLM) that integrates intrinsic musical abilities into LLaMA2 and GPT-3.5. |
| Outcome: | The proposed model can understand and generate music with a pure text tokenizer without external multi-modal neural structures or tokenizers. |
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them (2023.findings-acl)
Copied to clipboard
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, Jason Wei
| Challenge: | Language models have already made good progress on this benchmark, with the best model outperforming average reported human-rater results on 65% of the BIG-Bench tasks. |
| Approach: | They propose to use chain-of-thought prompting to challenge language models on 23 challenging BIG-Bench tasks which they call BIG-Bench Hard. |
| Outcome: | The proposed language models outperform the average human-rater on 65% of the BIG-Bench tasks. |
Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data (2024.findings-acl)
Copied to clipboard
Yanda Li, Chi Zhang, Gang Yu, Wanqi Yang, Zhibin Wang, Bin Fu, Guosheng Lin, Chunhua Shen, Ling Chen, Yunchao Wei
| Challenge: | OpenAI's GPT-4 has demonstrated remarkable multimodal capabilities, but specific mechanics of GPT4 remain unknown. |
| Approach: | They propose a data collection methodology that synchronously synthesizes images and dialogues for visual instruction tuning. |
| Outcome: | The proposed method improves on ten commonly assessed models and provides greater flexibility compared to existing methods. |
Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts (2026.acl-long)
Copied to clipboard
| Challenge: | Reinforcement learning with verifiable rewards (RLVR) training with Mixture-of-Experts policies remains fragile and prone to reward collapse. |
| Approach: | They propose a router shift-based policy optimization method that computes a per-token router-shift ratio conditioned on the previously activated experts and applies stop-gradient and a lower-bound floor. |
| Outcome: | The proposed method achieves better performance and greater stability than previous methods. |
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning (2023.acl-long)
Copied to clipboard
Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
| Challenge: | XY-LENT: X-Y bitext enhanced Language ENcodings achieves state-of-the-art performance over 5 cross-lingual tasks within all model size bands. |
| Approach: | They propose a method for building multilingual representation models that are competitive with existing models and more parameter efficient. |
| Outcome: | The proposed model outperforms XLM-R XXL and is 5x and 6x smaller respectively. |
Consistency Regularization for Cross-Lingual Fine-Tuning (2021.acl-long)
Copied to clipboard
Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
| Challenge: | Experimental results show that consistency regularization improves cross-lingual fine-tuning . pre-trained cross-linguistic models can transfer task-specific supervision from one language to the other . |
| Approach: | They propose to improve cross-lingual fine-tuning with consistency regularization . they use example consistency regularized to penalize prediction sensitivity to four types of data augmentations . |
| Outcome: | The proposed method improves cross-lingual fine-tuning across tasks . it can be generalized to other target languages without additional training . |
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment (2021.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that denoising word alignment improves cross-lingual transferability . most applications and resources are still English-centric, making non-English users hard to access. |
| Approach: | They propose to denoise word alignment as a cross-lingual pre-training task . they first self-label word alignments for parallel sentences and then mask tokens . |
| Outcome: | The proposed model improves cross-lingual transferability on token-level tasks, especially on question answering, and structured prediction. |
Training a Better Chinese Spelling Correction Model via Prior-knowledge Guided Teacher (2024.findings-acl)
Copied to clipboard
| Challenge: | Chinese Spelling Correction models are prone to over-correct and poor generalization for error patterns outside the standard distribution. |
| Approach: | They propose a teacher network guided by prior knowledge for distillation learning of CSC models. |
| Outcome: | The proposed method significantly enhances the CSC model’s language modeling capabilities, crucial for minimizing over-correction. |
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2021.naacl-main)
Copied to clipboard
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
| Challenge: | Existing methods for learning cross-lingual representations are lacking in the field of NLP. |
| Approach: | They propose a framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. |
| Outcome: | The proposed approach improves cross-lingual transferability on benchmarks. |