Papers by Weiquan Mao
TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities (2023.acl-demo)
Copied to clipboard
Zhe Zhao, Yudong Li, Cheng Hou, Jing Zhao, Rong Tian, Weijie Liu, Yiren Chen, Ningyuan Sun, Haoyan Liu, Weiquan Mao, Han Guo, Weigang Gou, Taiqiang Wu, Tao Zhu, Wenhang Shi, Chen Chen, Shan Huang, Sihong Chen, Liqun Liu, Feifei Li, Xiaoshuai Chen, Xingwu Sun, Zhanhui Kang, Xiaoyong Du, Linlin Shen, Kimmo Yan
| Challenge: | Several pre-training models of different modalities are showing a rising trend of homogeneity in their model structures. |
| Approach: | They propose a toolkit that supports pre-training models of different modalities. |
| Outcome: | The proposed toolkit can match the performance of the original implementations on text, vision, and audio benchmarks. |
A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning (2022.coling-1)
Copied to clipboard
Kunbo Ding, Weijie Liu, Yuejian Fang, Weiquan Mao, Zhe Zhao, Tao Zhu, Haoyan Liu, Rong Tian, Yiren Chen
| Challenge: | Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries . however, its effect is limited by the gap between embedding clusters of different languages . |
| Approach: | They propose Embedding-Push, Attention-Pull, and Robust targets to transfer English embeddings to virtual multilingual embedders without semantic loss. |
| Outcome: | Experimental results show that the proposed method outperforms existing methods on cross-lingual tasks and can achieve a better multilingual alignment. |
SAMP: A Model Inference Toolkit of Post-Training Quantization for Text Processing via Self-Adaptive Mixed-Precision (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for quantization of models are too complicated and can cause performance damage. |
| Approach: | They propose a self-adaptive mixed-precision (SAMP) toolkit to automatically control quantization rate by a mixed-presence architecture to balance model accuracy and efficiency. |
| Outcome: | The proposed toolkit has a higher speedup than PyTorch and FasterTransformer while ensuring the required accuracy. |
Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity Matching (2022.findings-naacl)
Copied to clipboard
Kunbo Ding, Weijie Liu, Yuejian Fang, Zhe Zhao, Qi Ju, Xuefeng Yang, Rong Tian, Zhu Tao, Haoyan Liu, Han Guo, Xingyu Bai, Weiquan Mao, Yudong Li, Weigang Guo, Taiqiang Wu, Ningyuan Sun
| Challenge: | Existing studies have shown that cross-lingual knowledge distillation can improve the performance of pre-trained models for cross-linguistic similarity matching tasks. |
| Approach: | They propose a multi-stage distillation framework for constructing a small-size but high-performance cross-lingual model using contrastive learning, bottleneck, and parameter recurrent strategies. |
| Outcome: | The proposed model can compress the size of XLM-R and MiniLM by more than 50% while the performance is only reduced by about 1%. |
CSL: A Large-scale Chinese Scientific Literature Dataset (2022.coling-1)
Copied to clipboard
| Challenge: | Existing datasets centered around the English language restrict development of Chinese scientific NLP. |
| Approach: | They present a large-scale Chinese scientific literature dataset based on Chinese papers . they use semi-structured data as a natural annotation for many supervised NLP tasks . |
| Outcome: | The proposed dataset can serve as a Chinese corpus and perform many supervised tasks. |
Parameter-efficient Continual Learning Framework in Industrial Real-time Text Classification System (2022.naacl-industry)
Copied to clipboard
Tao Zhu, Zhe Zhao, Weijie Liu, Jiachi Liu, Yiren Chen, Weiquan Mao, Haoyan Liu, Kunbo Ding, Yudong Li, Xuefeng Yang
| Challenge: | Existing continual learning methods use data replay, parameter isolation and regularization to mitigate catastrophic forgetting. |
| Approach: | They propose a parameter-efficient continual learning framework that updates parameters offline and then trains using an online regularization method. |
| Outcome: | The proposed framework reduces catastrophic forgetting and saves the model with the changed parameters instead of all parameters. |
Topic Detection and Tracking with Time-Aware Document Embeddings (2024.lrec-main)
Copied to clipboard
| Challenge: | Topic Detection and Tracking (TDT) systems aim to cluster news articles into real-world events. |
| Approach: | They propose a neural method that fuses temporal and textual information into a single representation of news documents for event detection. |
| Outcome: | The proposed model outperforms baselines on two benchmark TDT data sets in English. |