LLM Braces: Straightening Out LLM Predictions with Relevant Sub-Updates (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies reveal that much of the knowledge in a Transformer-based Large Language Model (LLM) is encoded in its feed-forward (FFN) layers, where each FNN layer can be interpreted as the summation of sub-updates, each corresponding to a weighted column vector from the FFN’s value parameter matrix. |
| Approach: | They propose a method that computes relevance scores associated with value vectors in FFN layers and leverages these scores to dynamically adjust the contribution of sub-updates. |
| Outcome: | The proposed framework outperforms baseline approaches in fine-tuning and zero-shot settings while requiring significantly fewer tunable parameters. |
Similar Papers
Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities (2025.naacl-long)
Copied to clipboard
Chung-En Sun, Xiaodong Liu, Weiwei Yang, Tsui-Wei Weng, Hao Cheng, Aidan San, Michel Galley, Jianfeng Gao
| Challenge: | Recent research shows that Large Language Models (LLMs) are vulnerable to automated jailbreak attacks. |
| Approach: | They propose a framework that crafts adversarial LLMs with enhanced jailbreak ability. |
| Outcome: | ADV-LLM significantly reduces the computational cost of generating adversarial suffixes while achieving nearly 100% ASR on various open-source LLMs. |
FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping (2024.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive Large Language Models (LLMs) are omnipresent but typically come with a substantial model size. |
| Approach: | They propose a novel fine-grained skip strategy for autoregressive large language models . they observe the saturation of computationally expensive feed-forward blocks of LLMs . |
| Outcome: | The proposed method can skip 25-30% of FFN blocks with marginal change in performance on knowledge-intensive generation tasks. |
Efficient Layer-wise LLM Fine-tuning for Revision Intention Prediction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models have shown extraordinary success across text generation tasks . however, their potential for simple yet essential text classification remains underexplored . |
| Approach: | a plug-and-play layer-wise parameter-efficient fine-tuning framework is proposed . it fine- tunes a subset of important LLM layers while freezing redundant ones . |
| Outcome: | a plug-and-play framework fine-tunes a subset of important LLM layers while freezing redundant layers. |
CogSteer: Cognition-Inspired Selective Layer Intervention for Efficiently Steering Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) achieve excellent performance through pretraining on extensive data. |
| Approach: | They propose an efficient selective layer intervention based on parameter-efficient fine-tuning methods to select the optimal steering layer to modulate LLM semantics. |
| Outcome: | The proposed approach is based on a model-agnostic framework and is safe to deploy. |
Improving Cross-Domain Low-Resource Text Generation through LLM Post-Editing: A Programmer-Interpreter Approach (2024.findings-eacl)
Copied to clipboard
| Challenge: | Large pre-trained language models such as GPT-3.5 and GPT-4 have gained significant attention in natural language research due to limited computational resources or inaccessible parameters. |
| Approach: | They propose a neural programmer-interpreter approach that preserves the domain generalization ability of LLMs while editing their output. |
| Outcome: | The proposed framework significantly improves GPT-3.5’s performance in logical form-to-text conversion and low-resource machine translation, surpassing other state-of-the-art (SOTA) LLM post-editing methods in cross-domain settings. |
MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning (2025.naacl-long)
Copied to clipboard
| Challenge: | Efficient finetuning of large language models (LLMs) aims to adapt the LLMs with reduced computational and memory costs. |
| Approach: | They propose a simple yet effective method that initializes low-rank matrices with Gaussian distribution and zero values while keeping the original weight matrics frozen. |
| Outcome: | The proposed approach only updates the minor components of the weight matrix while keeping the principal singular components frozen. |
GRAMMAR-LLM: Grammar-Constrained Natural Language Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to fine-tuning and prompting are insufficient to ensure compliance with predefined taxonomies, syntactic structures, or domain-specific rules. |
| Approach: | They propose a framework that integrates formal grammatical constraints into the decoding process to enforce syntactic correctness in linear time while maintaining expressiveness in grammar rule definition. |
| Outcome: | The proposed framework enforces syntactic correctness in linear time while maintaining expressiveness in grammar rule definition. |
Hit the Sweet Spot! Span-Level Ensemble for Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | a recent study focused on sample-level and token-level ensembles, which hinder dynamic correction and enhancement of outputs during the generation process. |
| Approach: | They propose a span-level ensemble method that balances real-time adjustments and accurate ensemble decisions. |
| Outcome: | The proposed method improves performance across language generation tasks significantly. |
Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference (2024.findings-eacl)
Copied to clipboard
| Challenge: | Large language models excel at understanding and generating human-like text, but their widespread deployment can be prohibitively expensive. |
| Approach: | They propose a method that makes large language models dynamic without Pre-Training . they use modularity in networks and sort sub-models based on computation/accuracy in a nested manner. |
| Outcome: | The proposed method can make large language models dynamic without pre-training and replace standard fine-tuning with sorted fine- tuning. |
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models (2023.emnlp-main)
Copied to clipboard
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Lee
| Challenge: | Large language models (LLMs) have shown unprecedented performance across various tasks. |
| Approach: | They propose an easy-to-use framework that integrates adapters into LLMs . they evaluate adapters on 14 datasets from two different reasoning tasks . |
| Outcome: | The proposed framework can be used to fine-tune open-access language models with task-specific data and instruction data. |