Pruning Multilingual Large Language Models for Multilingual Inference (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual large language models (MLLMs) demonstrate better zeroshot learning performance in non-English languages compared to large language model trained on English-dominant data. |
| Approach: | They propose a pruning approach to prune large language models using bilingual sentence pairs from English and other languages to enhance their performance in non-English language. |
| Outcome: | The proposed pruning strategy enhances the MLLMs’ performance in non-English language. |
Similar Papers
On the Limitations of Language-targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning (2026.tacl-1)
Copied to clipboard
| Challenge: | Recent advances in large language model pruning have shown high predictive performance in post-training settings. |
| Approach: | They conduct an empirical study on the performance and internal representation changes associated with pruning multilingual models for monolingual applications. |
| Outcome: | The proposed pruning methods retain perplexity and yield high signal-to-noise ratios, but not consistently improve downstream tasks. |
Everything you need to know about Multilingual LLMs: Towards fair, performant and reliable models for languages of the world (2023.acl-tutorials)
Copied to clipboard
| Challenge: | Responsible AI issues such as fairness, bias and toxicity will be discussed in this tutorial . |
| Approach: | This tutorial will describe various aspects of scaling up language technologies to many of the world’s languages by describing the latest research in Massively Multilingual Language Models (MMLMs). |
| Outcome: | This tutorial will cover various aspects of scaling up language technologies to many of the world's languages by describing the latest research in multilingual models. |
On the Calibration of Massively Multilingual Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Massively Multilingual Language Models (MMLMs) have gained popularity due to their effectiveness in cross-lingual transfer. |
| Approach: | They investigate how well calibrated MMLMs are with respect to confidence . they find that calibration methods like temperature scaling and label smoothing improve calibration . |
| Outcome: | The proposed models are able to generalize in languages unseen during fine-tuning, but they are not reliable across languages. |
Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency (2022.acl-long)
Copied to clipboard
| Challenge: | Structured pruning has been extensively studied on monolingual pre-trained models . but little attention has been paid to evaluating the effectiveness of structured pruning on multilingual models. |
| Approach: | They investigate settings, algorithms, and efficiency of structured pruning on multilingual models . authors propose a simple approach that allows training the model once and adapting to different model sizes at inference . |
| Outcome: | The proposed approach allows training the model once and adapting to different model sizes at inference. |
Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem? (2025.findings-acl)
Copied to clipboard
| Challenge: | Multimodal large language models have shown remarkable performance for cross-modal understanding and generation, yet suffer from severe inference costs. |
| Approach: | They propose to prune redundant tokens in MLLMs to reduce computation and storage costs. |
| Outcome: | The proposed method reduces the computational and storage costs of MLLMs by identifying redundant tokens and pruning them. |
Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean (2024.lrec-main)
Copied to clipboard
ChangSu Choi, Yongbin Jeong, Seoyoon Park, Inho Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, HyeJin Lee, Younggyun Hahm, Hansaem Kim, KyungTae Lim
| Challenge: | Large language models (LLMs) use pretraining to predict the subsequent word, but less-resourced languages are being overlooked. |
| Approach: | They propose to expand the MLLM vocabularies to enhance expressiveness and use bilingual data for pretraining to align the high- and less-resourced languages. |
| Outcome: | The proposed model outperforms existing models in qualitative analyses compared to Korean monolingual models. |
DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Structured pruning reduces model size but often causes uneven degradation across domains, leading to biased performance. |
| Approach: | They propose a method that dynamically adjusts the data distribution during training to restore balanced performance across heterogeneous and multi-tasking data. |
| Outcome: | Experiments in monolingual and multilingual settings show that the proposed method surpasses similarly sized models in pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning. |
M-Wanda: Improving One-Shot Pruning for Multilingual LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual LLM performance is often dependent on model size, resulting in performance loss. |
| Approach: | They propose a pruning method that models cross-lingual variation by incorporating language-aware activation statistics into its pruning criterion and dynamically adjusts layerwise sparsity based on cross-linguistic importance. |
| Outcome: | The proposed method improves performance at minimal additional costs while maintaining multilinguality. |
Mitigating Language-Level Performance Disparity in mPLMs via Teacher Language Selection and Cross-lingual Self-Distillation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large-scale multilingual pretrained language models (mPLMs) yield impressive performance on cross-language tasks, yet significant performance disparities exist across different languages within the same mPLm. |
| Approach: | They propose to leverage the learned knowledge from well-performing languages to guide under-performing ones within the same mPLM. |
| Outcome: | The proposed model shows that it can guide under-performing languages while minimizing language-level performance disparities across different mPLMs. |
mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus (2025.findings-acl)
Copied to clipboard
Matthieu Futeral, Armel Randy Zebaze, Pedro Ortiz Suarez, Julien Abadji, Rémi Lacroix, Cordelia Schmid, Rachel Bawden, Benoît Sagot
| Challenge: | Existing studies show that multimodal large language models can learn from text-image data. |
| Approach: | They propose to train multimodal large language models on large amounts of text-image data . they also show a boost in few-shot learning performance across various multilingual tasks . |
| Outcome: | The proposed dataset is not public and is only in English . it is the first large-scale multilingual and multimodal document corpus crawled from the web. |