Papers with PT
How Does In-Context Learning Help Prompt Tuning? (2024.findings-eacl)
Copied to clipboard
| Challenge: | a growing number of parameter-efficient adaptation methods are needed to fine-tune large language models. |
| Approach: | They propose a method that combines prompt tuning and in-context learning to improve prompt tuning by concatenating a natural language demonstration with learned prompt embeddings. |
| Outcome: | The proposed method outperforms prompt tuning and prompt tuning on five language generation tasks. |
A Multilingual Wikified Data Set of Educational Material (L18-1)
Copied to clipboard
Iris Hendrickx, Eirini Takoulidou, Thanasis Naskos, Katia Lida Kermanidis, Vilelmini Sosoni, Hugo de Vos, Maria Stasimioti, Menno van Zaanen, Panayota Georgakopoulou, Valia Kordoni, Maja Popovic, Markus Egg, Antal van den Bosch
| Challenge: | a crowdsourcing effort to annotate and link parallel texts has been unsuccessful . a data set of parallel texts in eleven languages is presented . |
| Approach: | They present a wikified data set of English sentences linked to Wikipedia pages . they use crowdsourcing to annotate the texts and perform crowdsourcing for complex annotations . |
| Outcome: | The proposed data set is valuable as it constitutes a rich resource . it includes annotated data of English sentences linked to translations in eleven languages . |
Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas (2025.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that LLMs can't tailor outputs to users with uncommon preferences . despite the success of persona inference, we may need debiasing and abstention. |
| Approach: | They propose to use preference data to infer needs and interests of users who prefer either output . they argue that training on preference data augmented with PI boosts personalization . |
| Outcome: | The proposed method can be used to improve personalization with less privacy concerns. |
Efficient and Effective Prompt Tuning via Prompt Decomposition and Compressed Outer Product (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models overlook intrinsic semantic associations between soft prompt tokens, leading to high discreteness and limited interactions. |
| Approach: | They propose a low-parameters Prompt Tuning method which leverages prompt decomposition and compressed outer product to facilitate multiple interactions among prompt tokens. |
| Outcome: | Experiments on six architectures and eight datasets show that the proposed method outperforms state-of-the-art methods in performance and efficiency. |
On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that PT and BT are nicely complementary to each other. |
| Approach: | They introduce two probing tasks for PT and BT respectively and investigate their complementarity. |
| Outcome: | The proposed methods establish state-of-the-art on the WMT16 English-Romanian and English-Russian benchmarks. |
On Transferability of Prompt Tuning for Natural Language Processing (2022.naacl-main)
Copied to clipboard
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, Jie Zhou
| Challenge: | Pre-trained language models (PLMs) can achieve comparable performance to full-parameter fine-tuning by tuning a few soft prompts, but require much more training time than fine-timing. |
| Approach: | They empirically investigate the transferability of soft prompts across different downstream tasks and PLMs to determine what decides prompt transferability. |
| Outcome: | The proposed method can achieve comparable performance to full-parameter fine-tuning by tuning a few soft prompts, but requires much more training time than fine-timing. |
On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translation (2022.coling-1)
Copied to clipboard
| Challenge: | Pre-Training (PT) of text representations has been successfully applied to low-resource Neural Machine Translation (NMT) however, it often fails to achieve notable gains on resource-rich NMT on par with its Random-Initialization (RI) counterpart. |
| Approach: | They propose to combine pre-training and random-initialization techniques to achieve significant improvements in NMT. |
| Outcome: | The proposed model fusion algorithm can achieve significant improvements on two resource-rich translation benchmarks. |
Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prompt Tuning has been successful as a parameter-efficient method of conditioning large-scale pre-trained language models to perform downstream tasks. |
| Approach: | They propose to use a vector-quantized input-contextualized prompt as an extension to the soft prompt tuning framework to learn contextualization of soft prompt tokens. |
| Outcome: | The proposed prompt outperforms soft prompt tuning by an average margin of 1.19% on various language understanding tasks like SuperGLUE, QA, Relation classification, NER and NLI. |
Extracting Shopping Interest-Related Product Types from the Web (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing e-commerce products are limited in their ability to assist customers in interest-oriented shopping. |
| Approach: | They propose to extract PTs from Web pages containing hand-crafted PT recommendations for SIs . they propose to use tree-transformer encoders for node classification to improve inter-node dependency modeling . |
| Outcome: | The proposed model outperforms the best baseline model by 2.37 F1 points on a WebPT dataset. |
FPT: Improving Prompt Tuning Efficiency via Progressive Training (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent prompt tuning (PT) has gained increasing attention as a parameter-efficient way of tuning pre-trained language models (PLMs). |
| Approach: | They propose a prompt tuning algorithm that uses a small-scale partial PLM and progressively expands its depth and width until the full-model size. |
| Outcome: | The proposed method could save over 30% of training computations while achieving comparable performance. |
Evaluating Pre-training Objectives for Low-Resource Translation into Morphologically Rich Languages (2022.lrec-1)
Copied to clipboard
| Challenge: | a lack of parallel data is a major limitation for Neural Machine Translation systems, especially for morphologically rich languages. |
| Approach: | They propose to leverage target monolingual data to overcome the lack of parallel data . they introduce a new technique called PT-Inflect to train NMT systems . |
| Outcome: | The proposed techniques outperform NMT systems trained on parallel data on four typologically diverse target languages. |
CIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Training (2023.findings-acl)
Copied to clipboard
| Challenge: | Speech-to-text training and language model distillation are used to bridge the representations between speech and text. |
| Approach: | They propose a pre-training paradigm that integrates speech and text into a single frame-to-token alignment. |
| Outcome: | The proposed paradigm outperforms the state-of-the-art model on intent classification and slot filling tasks. |
Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning? (2023.acl-long)
Copied to clipboard
| Challenge: | Prompt tuning (PT) based on frozen pre-trained language models has shown remarkable performance in few-shot learning . however, it relies heavily on good initialization of the prompt embeddings. |
| Approach: | They propose to use meta prompt tuning to improve cross-task generalization by learning to initialize prompt embeddings from other relevant tasks. |
| Outcome: | The proposed method outperforms PT on classification tasks, but not multi-task learning. |
The Threat of PROMPTS in Large Language Models: A System and User Prompt Perspective (2025.findings-acl)
Copied to clipboard
| Challenge: | Prompts are essential for guiding model output and influencing content generation. |
| Approach: | They propose to attack models with prompt leakage and prompt jailbreak attacks . they summarize the experimental setups of these methods and explore the relationship between prompt threats and prompt injection attacks. |
| Outcome: | The proposed methods summarize the experimental setups and examine the relationship between prompt threats and prompt injection attacks. |
A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training (2023.acl-long)
Copied to clipboard
| Challenge: | Modern Natural Language Generation models come with massive computational and storage requirements. |
| Approach: | They propose a method that applies word-level knowledge distillation to multiple PTs generated by both teacher and student. |
| Outcome: | The proposed techniques can be used to compress natural language models while preserving their performance. |