Papers with LLMBraces
LLM Braces: Straightening Out LLM Predictions with Relevant Sub-Updates (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies reveal that much of the knowledge in a Transformer-based Large Language Model (LLM) is encoded in its feed-forward (FFN) layers, where each FNN layer can be interpreted as the summation of sub-updates, each corresponding to a weighted column vector from the FFN’s value parameter matrix. |
| Approach: | They propose a method that computes relevance scores associated with value vectors in FFN layers and leverages these scores to dynamically adjust the contribution of sub-updates. |
| Outcome: | The proposed framework outperforms baseline approaches in fine-tuning and zero-shot settings while requiring significantly fewer tunable parameters. |