KS-Lottery: Finding Certified Lottery Tickets for Multilingual Transfer in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that a small subset of parameters is highly effective in fine-tuning . prior work shows that there are a few additional parameters corresponding to an intrinsic dimension in a well-trained Large Language Model. |
| Approach: | They propose a method to identify a small subset of LLM parameters highly effective in multilingual fine-tuning. |
| Outcome: | The proposed method can find the certified winning tickets in the embedding layer, and fine-tuning on the found parameters is guaranteed to perform as well as full fine- tuning. |
Similar Papers
Finding the Dominant Winning Ticket in Pre-Trained Language Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on pre-trained language models show that they can fine-tune parameters but achieve good downstream performance. |
| Approach: | They find that a dominant winning ticket takes up 0.05% of the parameters and is transferable across different tasks. |
| Outcome: | The proposed model can achieve comparable performance with the full-parameter model, the authors show . the dominant winning ticket takes up 0.05% of the parameters, and the model is transferable across tasks, they show - the authors conclude . |
PLEX: Adaptive Parameter-Efficient Fine-Tuning for Code LLMs using Lottery-Tickets (2025.naacl-industry)
Copied to clipboard
| Challenge: | PLEX is a lottery-ticket based parameter-efficient fine-tuning method that adapts large language models to well-supported and underrepresented programming languages (PLs) in pretraining. |
| Approach: | They propose a lottery-ticket based parameter-efficient fine-tuning method that adapts large language models to well-supported and underrepresented programming languages (PLs) |
| Outcome: | The proposed method achieves state-of-the-art performance among PEFT methods while maintaining competitive results with reduced computational overhead. |
Robust Lottery Tickets for Pre-trained Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that pre-trained language models contain smaller matching subnetworks that are not robust to adversarial examples. |
| Approach: | They propose a method to find robust tickets hidden in pre-trained language models by learning binary weight masks and an adversarial loss objective to guide the search. |
| Outcome: | The proposed method improves on previous work on adversarial robustness evaluation. |
Multilingual Lottery Tickets to Pretrain Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual pretrained language models (mPLMs) have become the de-facto standard for multilingual tasks. |
| Approach: | They propose to search for a per-language subnetwork with comparable performance to the full model by scaling the model to reduce interference and then redistributing parameters to keep the parameters reduced. |
| Outcome: | The proposed model reduces the inference cost of models for each language while keeping the capacity per language more or less the same. |
Composable Sparse Fine-Tuning for Cross-Lingual Transfer (2022.acl-long)
Copied to clipboard
| Challenge: | Adapters and sparse fine-tuning have been developed to improve transfer learning . a number of approaches have been proposed to improve performance of fine-untuners . |
| Approach: | They propose a method that fine-tunes the entire set of parameters of a large pretrained model . they use adapters and sparse fine-uning to improve model efficiency . |
| Outcome: | The proposed method outperforms adapters in cross-lingual transfer benchmarks. |
Discovering Language-neutral Sub-networks in Multilingual Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that multilingual pre-trained language models transfer well on cross-lingual downstream tasks. |
| Approach: | They conceptualize language neutrality as a function of overlap between language-encoding sub-networks of multilingual models. |
| Outcome: | The proposed model performs well on cross-lingual tasks despite being pre-trained on multiple languages . |
Exploring Lottery Prompts for Pre-trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to optimize pre-trained language models are expensive and slow to scale. |
| Approach: | They propose to search for instance-level lottery prompts and generalize them to unseen data . they validate the assumption that for every instance, there is almost always a lottery prompt that induces the correct prediction from the PLM . |
| Outcome: | The proposed method can achieve comparable results with other gradient-free and optimization-free baselines. |
Prompts Can Play Lottery Tickets Well: Achieving Lifelong Information Extraction via Lottery Prompt Tuning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing research on information extraction tasks focuses on one specific task, but in real-world scenarios, new data of different IE tasks and domains come in a stream over time. |
| Approach: | They propose a parameter- and deployment-efficient prompt tuning method to evaluate the UIE system under a “lifelong learning” setting. |
| Outcome: | The proposed method is able to learn new tasks without forgetting old ones and expand knowledge and functionalities without retraining the whole system. |
How to Improve LLMs’ Performance on Specific Languages: A Perspective on LLM-Derived Language Similarity (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit uneven performance across languages. |
| Approach: | They propose to use a framework to quantify the similarity within each language pair through both the lenses of language-specific performance patterns and cross-lingual transferability. |
| Outcome: | The proposed approach outperforms traditional linguistic typology and cross-lingual transferability measures on multilingual LLMs. |
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization (2021.acl-long)
Copied to clipboard
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, Weizhu Chen
| Challenge: | 'lottery tickets' can be trained to match the performance of a full model . subnetwork training can also outperform random sampled subnetworks of the same size . |
| Approach: | They propose to train a subnetwork of 'lottery tickets' to match the full model's performance. |
| Outcome: | The proposed model outperforms subnetworks of the same size in a phase transition phenomenon . the proposed model improves single task fine-tuning by 0.9 points on BERT-base and 1.0 points on GLUE large . |