| Challenge: | Prefix Learning is an empirically efficient and effective method for language models . but the theoretical understandings are limited on the performance of such methods . |
| Approach: | They propose a method that can train an ultra-long prefix in a stylized setting using the Neural Tangent Kernel framework. |
| Outcome: | The proposed method can achieve superior performance on vision, natural language, and math data. |
Similar Papers
Prefix Propagation: Parameter-Efficient Tuning for Long Sequences (2023.acl-short)
Copied to clipboard
| Challenge: | Prefix-tuning prepends trainable tokens to sequences while freezing the rest of the model’s parameters. |
| Approach: | They propose a method that prefixes on previous hidden states to improve model performance. |
| Outcome: | The proposed architecture outperforms prefix-tuning on long-document tasks while using 50% fewer parameters. |
Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning (2023.acl-short)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning only optimizes a few task-specific parameters with frozen pre-trained model. |
| Approach: | They propose to optimize a prefix vector inserted into Transformer layers to optimize the prefix . they propose to use a gate mechanism to adjust the prefixed to each layer . |
| Outcome: | The proposed approach improves on the SuperGLUE and NER datasets. |
Inducer-tuning: Connecting Prefix-tuning and Adapter-tuning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prefix-tuning is an essential paradigm of parameter-efficient transfer learning . fine-tuned models require separate copies of model parameters for each task . |
| Approach: | They propose to understand and further develop prefix-tuning through the kernel lens . they propose a new variant of prefix tuning that shares the exact mechanism as prefix tun . |
| Outcome: | The proposed method improves prefix-tuning performance by training only a small portion of parameters. |
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021.acl-long)
Copied to clipboard
| Challenge: | Fine-tuning is the prevalent paradigm for using large pretrained language models for downstream tasks, but it requires updating and storing all the parameters of the LM. |
| Approach: | They propose a lightweight alternative to fine-tuning for natural language generation tasks that optimizes a sequence of continuous vectors, which they call the prefix. |
| Outcome: | The proposed approach outperforms fine-tuning in the full data setting and extrapolates better to examples with topics that are unseen during training. |
Selective Prefix Tuning for Pre-trained Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models are time-consuming and memory-inefficient. |
| Approach: | They propose a method that inserts learnable vectors into each Transformer layer . they propose SL to encourage diversity in prefix tokens . |
| Outcome: | Extensive experiments validate the effectiveness of Prefix Tuning in sentence and token classification tasks. |
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (P19-1)
Copied to clipboard
| Challenge: | Term memory networks (RNNs) are difficult to optimize due to gradient vanishing and explosion. |
| Approach: | They propose a neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. |
| Outcome: | The proposed method improves state-of-the-art performance on short and long sequences and generates coherent, novel text articles with thousands of tokens. |
From Bottom to Top: Extending the Potential of Parameter Efficient Fine-Tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to fine-tune large language models primarily focus on the interaction between different layers, ignoring the fact that different layers store different information. |
| Approach: | They propose a Parameter Efficient Fine-Tuning method which freeze pre-trained parameters and fine-tunes only a few task-specific parameters. |
| Outcome: | The proposed methods reduce parameter count to nearly half by omitting fine-tuning in the middle layers. |
On the Adaptation of Unlimiformer for Decoder-Only Transformers (2024.lrec-main)
Copied to clipboard
| Challenge: | despite efforts in the community, most common models have a context length of 4k or less. |
| Approach: | They propose to adapt a vector-retrieval augmentation method to decoder-only transformers . they also expand the experimental setup on summarization to include a new task and an instruction-tuned model . |
| Outcome: | The proposed model performs on par with a model with 2x the context length. |
Exploring Fine-Tuning for In-Context Retrieval and Efficient KV-Caching in Long-Context Language Models (2026.eacl-short)
Copied to clipboard
| Challenge: | Long-Context Language Models (LCLMs) can encode entire document collections, offering a strong alternative to retrieval-augmented generation (RAG). |
| Approach: | They propose to use LCLMs to encode documents with context windows of millions of tokens to improve their performance. |
| Outcome: | The proposed training strategies improve long-context performance and their robustness under compression techniques. |
Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large multi-modal models (LMMs) are revolutionizing the way machines interact with the world, unlocking new possibilities across multi-dimensional applications. |
| Approach: | They propose a parameter-efficient fine-tuning strategy that combines both . they find that parameter tuning methods distort the feature representation space . |
| Outcome: | The proposed strategy preserves representation space while limiting performance on downstream tasks. |