Papers by Xinge Ma
Knowledge Distillation with Reptile Meta-Learning for Pretrained Language Model Compression (2022.coling-1)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) can transfer knowledge from the original model into a compact model to achieve model compression. |
| Approach: | They propose a knowledge distillation method with reptile meta-learning to facilitate the transfer of knowledge from the teacher to the student. |
| Outcome: | Extensive experiments on the GLUE benchmark show the proposed method performs better than previous methods. |
FedID: Federated Interactive Distillation for Large-Scale Pretraining Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | federated learning (FL) is widely studied in user-related natural language processing (NLP) but its performance is faded by confirmation bias. |
| Approach: | They propose a decentralized learning paradigm that uses labeled data to rectify local models . they propose federated interactive distillation (FedID) to alleviate communication overhead . |
| Outcome: | The proposed framework achieves the best results in homogeneous and heterogeneously federated scenarios. |