FedID: Federated Interactive Distillation for Large-Scale Pretraining Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | federated learning (FL) is widely studied in user-related natural language processing (NLP) but its performance is faded by confirmation bias. |
| Approach: | They propose a decentralized learning paradigm that uses labeled data to rectify local models . they propose federated interactive distillation (FedID) to alleviate communication overhead . |
| Outcome: | The proposed framework achieves the best results in homogeneous and heterogeneously federated scenarios. |
Similar Papers
FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks (2022.findings-naacl)
Copied to clipboard
Bill Yuchen Lin, Chaoyang He, Zihang Ze, Hulin Wang, Yufen Hua, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, Salman Avestimehr
| Challenge: | Increasing concerns and regulations about data privacy necessitate the study of privacy-preserving, decentralized learning methods for natural language processing tasks. |
| Approach: | They propose a framework for evaluating federated learning methods on four different tasks . they propose federation between Transformer-based language models and FL methods . |
| Outcome: | The proposed framework compares FL methods on four different tasks under non-IID partitioning strategies. |
Can Public Large Language Models Help Private Cross-device Federated Learning? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have shown that public data can be used to improve privacy-utility trade-offs for large and small language models. |
| Approach: | They propose to use large-scale public data to help differentially private FL training . they propose a distribution matching algorithm with theoretical grounding to sample public data close to private data distribution . |
| Outcome: | The proposed method is efficient and effective for training private models by taking advantage of public data. |
FedPETuning: When Federated Learning Meets the Parameter-Efficient Tuning Methods of Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing research on federated learning (FL) for pre-trained language models (PLMs) with increasing concerns about data privacy, enterprises or institutions are not allowed to collect data from end devices or local clients to a centralized server for fine-tuning PLMs. |
| Approach: | They investigate the parameter-efficient tuning of pre-trained language models (PLMs) and develop a federated benchmark for four representative PETuning methods . |
| Outcome: | The proposed method can defend against privacy attacks and maintain acceptable performance with reducing heavy resource consumption. |
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |
Practical Takes on Federated Learning with Pretrained Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | federated learning with pretrained language models for language tasks entails data privacy constraints when learning from diverse data domains. |
| Approach: | They propose to use pretrained language models to learn from diverse data domains . they elaborate hypotheses over the components in federated NLP architectures based on three tasks . |
| Outcome: | The proposed model can generalize by adapting to the different domains. |
FedED: Federated Learning via Ensemble Distillation for Medical Relation Extraction (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing relation extraction methods require centralizing training data from different medical platforms while holding the privacy-sensitive data puts patients' privacy at risk. |
| Approach: | They propose a federated relation extraction model that trains a central model without sharing or exchange of private local data. |
| Outcome: | The proposed model trains a central model without uploading local parameters, and it performs well on three publicly available datasets. |
FedCoT: Federated Chain-of-Thought Distillation for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have emerged as a transformative force in artificial intelligence, demonstrating exceptional proficiency across various tasks. |
| Approach: | They propose a federated framework for the Chain-of-Thought distillation of knowledge from LLMs to SLMs, while adhering to privacy requirements. |
| Outcome: | The proposed framework ensures secure knowledge transfer from an LLM on a high-powered server to an SLM on resource-constrained client while adhering to privacy requirements. |
Safely Learning with Private Data: A Federated Learning Framework for Large Language Model (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) use large amounts of public data and massive parameters, but private data is often stored in isolated data silos. |
| Approach: | They propose a Federated Learning framework for large language models which offloads most training parameters to the server while training embedding and output layers locally. |
| Outcome: | The proposed framework achieves comparable metrics to centralized chatGLM model on NLU and generation tasks. |
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on pre-trained language models focuses on reducing the size of these models into shallow ones. |
| Approach: | They propose a knowledge distillation technique that leverages teacher internal representations to reduce the size of pre-trained language models. |
| Outcome: | The proposed method outperforms previous methods in multilingual Named Entity Recognition (NER) it reduces the size of teacher models by 35x while retaining 95% of its F1 score. |
Empirical Studies of Institutional Federated Learning For Natural Language Processing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | federated learning is a promising ideology to unite isolated datasets for machine learning problems. |
| Approach: | They propose to use federated natural language processing networks to train a popular NLP model with applications in sentence intent classification. |
| Outcome: | The proposed model is sensitive to imbalanced data load and tested against a federated model under imbalanced datasets. |