Unraveling Babel: Exploring Multilingual Activation Patterns of LLMs and Their Applications (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on how large language models process multiple languages, but internal mechanisms of LLMs remain insufficiently explored. |
| Approach: | They propose to convert dense LLMs into fine-grained MoE architectures and analyze their activation patterns using expert activation frequency heatmaps. |
| Outcome: | The proposed method outperforms random expert pruning and exceeds models in some languages. |
Similar Papers
Unveiling Multimodal Processing: Exploring Activation Patterns in Multimodal LLMs for Interpretability and Efficiency (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models have remained opaque. |
| Approach: | They propose a method to convert dense MLLMs into fine-grained Mixture-of-Experts architectures. |
| Outcome: | The proposed method outperforms random expert pruning and sparse activation and model pruning. |
Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Recent studies suggest that large language models can transfer skills learned in one language to others, but internal mechanisms behind this ability remain unclear. |
| Approach: | They find that LLMs map semantically identical inputs from different languages into a common semantic latent space that allows for consistent processing across languages. |
| Outcome: | The findings highlight the structural evolution of multilingual models during training and scaling up. |
Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models (2024.acl-long)
Copied to clipboard
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, Ji-Rong Wen
| Challenge: | Despite the impressive multilingual capabilities demonstrated by LLMs, the understanding of how these abilities develop and function remains nascent. |
| Approach: | They propose a novel detection method to pinpoint language-specific neurons within LLMs by selectively activating or deactivating these neurons. |
| Outcome: | The proposed method can “steer” the output language of LLMs by selectively activating or deactivating language-specific neurons. |
On the Limitations of Language-targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning (2026.tacl-1)
Copied to clipboard
| Challenge: | Recent advances in large language model pruning have shown high predictive performance in post-training settings. |
| Approach: | They conduct an empirical study on the performance and internal representation changes associated with pruning multilingual models for monolingual applications. |
| Outcome: | The proposed pruning methods retain perplexity and yield high signal-to-noise ratios, but not consistently improve downstream tasks. |
Revealing the Parallel Multilingual Learning within Large Language Models (2024.emnlp-main)
Copied to clipboard
Yongyu Mu, Peinan Feng, Zhiquan Cao, Yuzhang Wu, Bei Li, Chenglong Wang, Tong Xiao, Kai Song, Tongran Liu, Chunliang Zhang, JingBo Zhu
| Challenge: | Large language models (LLMs) can handle multilingual and cross-lingual text within a single input; however, previous studies focusing on using English as the pivot language to enhance language understanding and reasoning focus on using multiple languages. |
| Approach: | They propose to use parallel multilingual input to enhance the model's comprehension of the input and to examine how multilingual processing affects prediction. |
| Outcome: | The proposed model can handle multilingual and cross-lingual text within a single input, but previous studies focused on using English as the pivot language to enhance language understanding and reasoning. |
Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit impressive abilities in various domains such as text generation, instruction following, and reasoning. |
| Approach: | They propose a method to decompose the activations of Large Language Models into a sparse linear combination of SAE features. |
| Outcome: | The proposed method shows that some features are strongly related to specific languages, while others are unaffected by ablating them. |
LLMs for Low Resource Languages in Multilingual, Multimodal and Dialectal Settings (2024.eacl-tutorials)
Copied to clipboard
| Challenge: | Recent advances in AI can be attributed to the remarkable performance of Large Language Models (LLMs) success of LLMs depends on specific training techniques, such as instruction tuning and prompting . |
| Approach: | They explore the capabilities of Large Language Models (LLMs) in various tasks and languages . they also examine their performance, fine-tuning, instructions tuning, and close vs. open models . |
| Outcome: | The proposed model can be used for speech and multimodal tasks across modalities, languages, and dialects. |
LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent multilingual models support limited number of human languages due to lack of training data for low resource languages. |
| Approach: | They propose a multilingual multilingual LLM that scales to 100 languages . they use a human feedback dataset and a data set to perform multilingual instruction tuning . |
| Outcome: | The proposed model outperforms its peers on five multilingual benchmarks. |
Language Adaptation of Large Language Models: An Empirical Study on LLaMA2 (2025.coling-main)
Copied to clipboard
| Challenge: | Popularity of Large Language Models (LLMs) has seen a skyrocketing increase in recent years. |
| Approach: | They present a systematic review of the language adaptation process for Large Language Models including vocabulary expansion, continued pre-training, and instruction fine-tuning. |
| Outcome: | The proposed model is based on empirical studies conducted on LLaMA2 and discussions on various settings affecting the model's capabilities. |
Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners (2024.emnlp-main)
Copied to clipboard
Shimao Zhang, Changjiang Gao, Wenhao Zhu, Jiajun Chen, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Shujian Huang
| Challenge: | Large Language Models (LLMs) have shown impressive language capabilities, but most of them have very unbalanced performance across different languages. |
| Approach: | They propose to use question translation data to enhance LLMs' multilingual capabilities by using mechanistic interpretability methods. |
| Outcome: | The proposed method improves multilingual alignment even with unannotated answers in English and a wide range of languages even with instruction-tuned LLMs. |