State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for State Space Models (2025.acl-short)
Copied to clipboard
| Challenge: | State Space Models (SSMs) have emerged as efficient alternatives to Transformers, but their application to SSMs remains unexplored. |
| Approach: | They propose a state-based PEFT method that adjusts state directly instead of using external prompts. |
| Outcome: | The proposed method is based on state-offset tuning, which directly affects state at every timestep. |
Similar Papers
On Pruning State-Space LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposed state-space models as an efficient alternative to transformers. |
| Approach: | They propose to prune state-space models (SSMs) to reduce computation costs by using unstructured pruning methods. |
| Outcome: | The proposed pruning methods show that they can be pruned to reduce their computation costs. |
PARA: Parameter-Efficient Fine-tuning with Prompt-Aware Representation Adjustment (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning excel in the context of single-backbone multi-tenant applications. |
| Approach: | They propose to integrate a lightweight vector generator within each Transformer layer to improve prompt-aware representation adjustment. |
| Outcome: | The proposed method surpasses current benchmarks in terms of performance despite having a similar number of adjustable parameters. |
Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large multi-modal models (LMMs) are revolutionizing the way machines interact with the world, unlocking new possibilities across multi-dimensional applications. |
| Approach: | They propose a parameter-efficient fine-tuning strategy that combines both . they find that parameter tuning methods distort the feature representation space . |
| Outcome: | The proposed strategy preserves representation space while limiting performance on downstream tasks. |
SSMLoRA: Enhancing Low-Rank Adaptation with State Space Model (2025.naacl-long)
Copied to clipboard
| Challenge: | Fine-tuning requires substantial computational resources and is prone to overfitting when applied to small datasets. |
| Approach: | They propose a parameter-efficient fine-tuning method that integrates a State Space Model (SSM) to interconnect low-rank matrices. |
| Outcome: | The proposed method achieves comparable performance to LoRA on the general language understanding evaluation (GLUE) benchmark while using only half the parameters. |
CodeSSM: Towards State Space Models for Code Understanding (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing transformers have limitations, such as quadratic complexity and high inference costs. |
| Approach: | They propose a state space model that is trained on code corpora to assess its effectiveness. |
| Outcome: | The proposed model reduces memory usage by up to 64% compared to transformers at a context length of 2048. |
Birdie: Advancing State Space Language Modeling with Dynamic Mixtures of Training Objectives (2024.emnlp-main)
Copied to clipboard
| Challenge: | Efficient state space models struggle with tasks requiring in-context retrieval, such as text copying and associative recall, limiting their usefulness in practical settings. |
| Approach: | They propose a training procedure that improves the performance of SSMs on retrieval-intensive tasks such as phone book lookup, long paragraph question-answering, and infilling tasks. |
| Outcome: | The proposed training procedure improves performance on retrieval-intensive tasks that challenge current SSMs, such as phone book lookup, long paragraph question-answering, and infilling tasks. |
Incorporating Exponential Smoothing into MLP: a Simple but Effective Sequence Model (2024.findings-naacl)
Copied to clipboard
| Challenge: | Structured State Space models (SSMs) have been used for long-range sequence learning but are limited in their complexity and computational and memory requirements. |
| Approach: | They propose to incorporate a simple SSM into an element-wise MLP to reduce inductive bias. |
| Outcome: | The proposed model achieves comparable results to existing models on the LRA benchmark. |
Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning (2023.acl-short)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning only optimizes a few task-specific parameters with frozen pre-trained model. |
| Approach: | They propose to optimize a prefix vector inserted into Transformer layers to optimize the prefix . they propose to use a gate mechanism to adjust the prefixed to each layer . |
| Outcome: | The proposed approach improves on the SuperGLUE and NER datasets. |
From Bottom to Top: Extending the Potential of Parameter Efficient Fine-Tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to fine-tune large language models primarily focus on the interaction between different layers, ignoring the fact that different layers store different information. |
| Approach: | They propose a Parameter Efficient Fine-Tuning method which freeze pre-trained parameters and fine-tunes only a few task-specific parameters. |
| Outcome: | The proposed methods reduce parameter count to nearly half by omitting fine-tuning in the middle layers. |
Memory-Efficient Fine-Tuning of Transformers via Token Selection (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning require caching of intermediate activations to update weights during the backward pass. |
| Approach: | They develop a method to reduce memory usage in fine-tuning of transformers by backpropagating through just a subset of input tokens. |
| Outcome: | The proposed method reduces memory usage and memory footprint on large transformer models . it can be easily combined with existing methods like LoRA, reducing memory cost . |