Practical Takes on Federated Learning with Pretrained Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | federated learning with pretrained language models for language tasks entails data privacy constraints when learning from diverse data domains. |
| Approach: | They propose to use pretrained language models to learn from diverse data domains . they elaborate hypotheses over the components in federated NLP architectures based on three tasks . |
| Outcome: | The proposed model can generalize by adapting to the different domains. |
Similar Papers
Pretrained Models for Multilingual Federated Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Federated Learning (FL) is a machine learning technique that trains a model across multiple distributed clients holding local data samples, without ever storing client data in a central location. |
| Approach: | They propose to use pretrained models to study three multilingual language tasks . they also examine impact of non-IID text on FL in naturally occurring data . |
| Outcome: | The proposed methods perform better than centralized learning even when using non-IID partitioning. |
FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks (2022.findings-naacl)
Copied to clipboard
Bill Yuchen Lin, Chaoyang He, Zihang Ze, Hulin Wang, Yufen Hua, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, Salman Avestimehr
| Challenge: | Increasing concerns and regulations about data privacy necessitate the study of privacy-preserving, decentralized learning methods for natural language processing tasks. |
| Approach: | They propose a framework for evaluating federated learning methods on four different tasks . they propose federation between Transformer-based language models and FL methods . |
| Outcome: | The proposed framework compares FL methods on four different tasks under non-IID partitioning strategies. |
FedPerC: Federated Learning for Language Generation with Personal and Context Preference Embeddings (2023.findings-eacl)
Copied to clipboard
| Challenge: | federated learning is a decentralized learning paradigm that assumes no access to a large labeled dataset and instead leverages averaged parameter updates across all users of the system. |
| Approach: | They propose a method to personalize federated learning with personal embeddings and shared context embeddables. |
| Outcome: | The proposed approach achieves 50% improvement in test-time perplexity using 0.001% of the memory required by baseline approaches and greater sample- and compute-efficiency. |
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020.acl-main)
Copied to clipboard
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A. Smith
| Challenge: | Language models prerained on text from a wide variety of sources form the foundation of today’s NLP. |
| Approach: | They propose to tailor a pretrained model to the domain of a target task by using domain-adaptive pretraining in-domain. |
| Outcome: | The proposed model can be tailored to the domain of a target task and perform well under both high- and low-resource settings. |
On the Importance of Effectively Adapting Pretrained Language Models for Active Learning (2022.acl-short)
Copied to clipboard
| Challenge: | Recent active learning approaches in NLP use off-the-shelf pretrained language models (LMs) . a poor training strategy can be catastrophic for AL, authors argue . |
| Approach: | They propose to first adapt the pretrained LM to the target task and then use it for AL. |
| Outcome: | The proposed approach provides substantial data efficiency improvements compared to the standard fine-tuning approach. |
Combining Deep Generative Models and Multi-lingual Pretraining for Semi-supervised Document Classification (2021.eacl-main)
Copied to clipboard
| Challenge: | Semi-supervised learning and multilingual pretraining have been shown to be effective for task-specific labelled data shortages. |
| Approach: | They propose to combine semi-supervised deep generative models and multi-lingual pretraining to form a pipeline for document classification task. |
| Outcome: | The proposed method outperforms state-of-the-art models in low-resource settings across several languages and outperformed existing models in English. |
Empirical Studies of Institutional Federated Learning For Natural Language Processing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | federated learning is a promising ideology to unite isolated datasets for machine learning problems. |
| Approach: | They propose to use federated natural language processing networks to train a popular NLP model with applications in sentence intent classification. |
| Outcome: | The proposed model is sensitive to imbalanced data load and tested against a federated model under imbalanced datasets. |
The Trade-offs of Domain Adaptation for Neural Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Neural Language Models (LMs) trained on large generic training sets have been shown to be effective at adapting to smaller, specific target domains for language modeling and other downstream tasks. |
| Approach: | They propose a framework for a Neural Language Models (LM) to be presented in a common framework. |
| Outcome: | The proposed framework highlights similarities and subtle differences between adaptation techniques and the framework. |
Personalized Federated Learning for Text Classification with Gradient-Free Prompt Tuning (2024.findings-naacl)
Copied to clipboard
Rui Wang, Tong Yu, Ruiyi Zhang, Sungchul Kim, Ryan Rossi, Handong Zhao, Junda Wu, Subrata Mitra, Lina Yao, Ricardo Henao
| Challenge: | Pretrained language models (PLMs) are used for personalized federated learning . communication costs are high with large PLMs, and local training is expensive . |
| Approach: | They propose a framework for federated learning with pretrained language models . they propose 'discrete local search' and compression mechanism for local training . |
| Outcome: | The proposed framework achieves superior performance compared with baselines. |
How much pretraining data do language models need to learn syntax? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretraining methods are convenient, but expensive in terms of time and resources. |
| Approach: | They investigate the impact of pretraining data size on the syntactic capabilities of RoBERTa by using syntaktic structural probes to determine whether models pretrained on more data encode a higher amount of syntastic information. |
| Outcome: | The proposed models perform better on part-of-speech tagging, dependency parsing and paraphrase identification. |