| Challenge: | Existing prefix parsers are typically tied to particular parsing algorithms. |
| Approach: | They propose a prefix grammar transformation that reduces prefix parsing to ordinary parsers . they propose enabling prediction of the next token by computing the next-token weight vector . |
| Outcome: | The proposed method reduces prefix parsing to ordinary parsers without modification . the transformed grammar is only a small factor larger than the input . |
Similar Papers
A Fast Algorithm for Computing Prefix Probabilities (2023.acl-short)
Copied to clipboard
| Challenge: | Probabilistic context-free grammars are an important formalism in NLP . |
| Approach: | They propose to run a probabilistic context-free grammar in O(n3|N|3 + |N|4), where n is the input length and |N is the number of non-terminals in the grammar. |
| Outcome: | The proposed algorithm runs in O(n3|N|3 + |N|4), where n is the input length and |N | is the number of non-terminals in the grammar. |
Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning (2023.acl-short)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning only optimizes a few task-specific parameters with frozen pre-trained model. |
| Approach: | They propose to optimize a prefix vector inserted into Transformer layers to optimize the prefix . they propose to use a gate mechanism to adjust the prefixed to each layer . |
| Outcome: | The proposed approach improves on the SuperGLUE and NER datasets. |
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021.acl-long)
Copied to clipboard
| Challenge: | Fine-tuning is the prevalent paradigm for using large pretrained language models for downstream tasks, but it requires updating and storing all the parameters of the LM. |
| Approach: | They propose a lightweight alternative to fine-tuning for natural language generation tasks that optimizes a sequence of continuous vectors, which they call the prefix. |
| Outcome: | The proposed approach outperforms fine-tuning in the full data setting and extrapolates better to examples with topics that are unseen during training. |
Prefix Propagation: Parameter-Efficient Tuning for Long Sequences (2023.acl-short)
Copied to clipboard
| Challenge: | Prefix-tuning prepends trainable tokens to sequences while freezing the rest of the model’s parameters. |
| Approach: | They propose a method that prefixes on previous hidden states to improve model performance. |
| Outcome: | The proposed architecture outperforms prefix-tuning on long-document tasks while using 50% fewer parameters. |
On Parsing as Tagging (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to reduce constituency parsing to tagging are based on linearization, learning, and decoding . linearization of the derivation tree is the most critical factor in achieving accurate parsers as taggers . |
| Approach: | They propose a pipeline with three steps for reducing constituency parsing to tagging . they find that linearization and learning are critical factors for accurate parsers . |
| Outcome: | The proposed pipelines are linearized, learning, and decoded, and have three steps to achieve accurate parsing as taggers. |
Inducer-tuning: Connecting Prefix-tuning and Adapter-tuning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prefix-tuning is an essential paradigm of parameter-efficient transfer learning . fine-tuned models require separate copies of model parameters for each task . |
| Approach: | They propose to understand and further develop prefix-tuning through the kernel lens . they propose a new variant of prefix tuning that shares the exact mechanism as prefix tun . |
| Outcome: | The proposed method improves prefix-tuning performance by training only a small portion of parameters. |
Prefix Lexicalization of Synchronous CFGs using Synchronous TAG (P18-1)
Copied to clipboard
| Challenge: | epsilon-free, chain-free synchronous context-free grammars can be converted into weakly equivalent synchronous tree-adjoining grammars (STAGs) this transformation doubles the grammar’s rank and cubes its size, but in practice the size increase is only quadratic. |
| Approach: | They extend Greibach normal form from CFGs to SCFGs and prove new formal properties about SCFG, a formalism with many applications in natural language processing. |
| Outcome: | The proposed grammars achieve asymptotic and empirical speed improvements on a machine translation task. |
Selective Prefix Tuning for Pre-trained Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models are time-consuming and memory-inefficient. |
| Approach: | They propose a method that inserts learnable vectors into each Transformer layer . they propose SL to encourage diversity in prefix tokens . |
| Outcome: | Extensive experiments validate the effectiveness of Prefix Tuning in sentence and token classification tasks. |
Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefix (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a large pre-trained language model can cause computational burdens in inference time due to multiple forward passes. |
| Approach: | They propose a method to learn fixed text representations with source tasks . they learn a task-specific prefix for each source task independently and combine them . |
| Outcome: | The proposed method improves generalizability of representations with source tasks. |
PIP: Parse-Instructed Prefix for Syntactically Controlled Paraphrase Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing fine-tuning methods for this task are costly and require updating the parameters of the entire model to adapt to the newly included syntax information. |
| Approach: | They propose a method to instruct model’s encoder prefix to capture syntax-related knowledge by direct initiation and indirect optimization. |
| Outcome: | The proposed methods are 10 times more efficient and learnable than existing methods. |