The Role of Outgoing Connection Heterogeneity in Feedforward Layers of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using a new fine-tuning loss, we show that inner neurons with diverse outgoing connections are more critical to model performance than those with uniform connections. |
| Approach: | They propose a new loss that reduces the outgoing connection entropy in feedforward layers and elucidates the role of outgoing connections in large language models. |
| Outcome: | The proposed method is significantly more effective than removing neurons randomly or based on their magnitude. |
Similar Papers
From Distributional to Overton Pluralism: Investigating Large Language Model Alignment (2025.naacl-long)
Copied to clipboard
| Challenge: | a large language model's (LLM) output distribution is changed by an alignment process . a recent study shows that aligned models surface information that cannot be recovered from base models without fine-tuning. |
| Approach: | They analyze two aspects of the alignment process that change output distributions . they find alignment suppresses irrelevant and unhelpful content . |
| Outcome: | The proposed model can be imitated without fine-tuning by using in-context examples and lower-resolution semantic hints about response content. |
Unveiling the Generalization Power of Fine-Tuned Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional multitasking abilities, but the comprehensive effects of fine-tuning on the LLMs’ generalization ability are not fully understood. |
| Approach: | They conduct extensive experiments across five distinct language tasks on different datasets to investigate whether fine-tuning affects the generalization ability intrinsic to LLMs. |
| Outcome: | The proposed model can generalize to different domains and tasks by integrating the in-context learning strategy during fine-tuning on generation tasks. |
The Effect of Language Diversity When Fine-Tuning Large Language Models for Translation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Prior research on language diversity in LLM fine-tuning has reported benefits while others find no benefits. |
| Approach: | They find that expanding language diversity during fine-tuning improves translation quality . they also show that increased language diversity creates more language-agnostic representations . |
| Outcome: | The proposed model improves translation quality for unsupervised and supervised pairs . the results plateau or decrease beyond a certain diversity threshold. |
Disentangling Linguistic Relatedness from Task Alignment in Cross-Lingual Transfer (2026.acl-srw)
Copied to clipboard
| Challenge: | Large language models (LLMs) have advanced natural language processing, yet their benefits remain concentrated in English and a small number of high-resource languages. |
| Approach: | They fine-tuned large language models (4B–671B parameters) on Arabic and evaluated zero-shot reading comprehension on Semitic languages and non-Semitic controls. |
| Outcome: | The results show that models with weak baselines improve across all languages, whereas strong-baseline models show only marginal gains regardless of language family. |
Let’s Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are composed of neurons that exhibit diverse behaviors and roles. |
| Approach: | They propose a novel approach that refines the granularity of parameter training down to the individual neuron, enabling a more parameter-efficient fine-tuning model. |
| Outcome: | The proposed approach exceeds the performance of full-parameter fine-tuning and PEFT and provides insights into the analysis of neurons. |
Understanding the effects of language-specific class imbalance in multilingual fine-tuning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to fine-tune large language models have been developed to reduce the amount of resources needed to perform classification tasks. |
| Approach: | They modify traditional class weighing approach to reduce imbalance by calculating class weights separately for each language. |
| Outcome: | The proposed model improves performance and reduces the promotion of uninformative features. |
Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies have shown that contextual language models display outlier dimensions . this is true for monolingual and multilingual models, but little work has been done on multilingual contexts . |
| Approach: | They investigate outlier dimensions and their relationship to anisotropy in multilingual contexts . they focus on cross-lingual semantic similarity tasks . |
| Outcome: | The proposed model improves on cross-lingual semantic similarity tasks. |
Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models (2025.acl-long)
Copied to clipboard
Zhisong Zhang, Yan Wang, Xinting Huang, Tianqing Fang, Hongming Zhang, Chenlong Deng, Shuaiyi Li, Dong Yu
| Challenge: | Large language models have demonstrated remarkable performance across a wide range of language tasks due to their remarkable ability in context modeling. |
| Approach: | They propose to use parallel context encoding to reduce attention entropy by incorporating attention sinks and selective mechanisms to reduce irregular attention . they also propose to incorporate attention sink mechanisms into the parallel encoded context to reduce the irregular attention. |
| Outcome: | The proposed methods lower irregular attention entropy and narrow performance gaps. |
Data and Parameter Scaling Laws for Neural Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that supervised neural machine translation models scale like a power law with the amount of training data and number of non-embedding parameters in the model. |
| Approach: | They show that cross-entropy loss of supervised neural machine translation models scales like a power law with the amount of training data and number of non-embedding parameters in the model. |
| Outcome: | The proposed model can predict BLEU and ROI of labeling data in low-resource language pairs. |
Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Effective cross-lingual transfer is hindered by performance gaps and the scarcity of fine-tuning data in many languages. |
| Approach: | They propose a middle-layer alignment objective integrated into task-specific training to improve cross-lingual transfer across languages. |
| Outcome: | The proposed method improves cross-lingual transfer to lower-resource languages and can be merged with existing modules without full re-training. |