Papers with prompt-tuning
Building a Personalized Dialogue System with Prompt-Tuning (2022.naacl-srw)
Copied to clipboard
| Challenge: | We build a dialogue system that can respond based on a given character setting (persona) this method is not suitable because the more persona information is added, the longer the input text becomes. |
| Approach: | They propose to use prompt-tuning to build a dialogue system that responds based on a persona . they conduct automatic and manual evaluations on English and Japanese . |
| Outcome: | The proposed method can build a dialogue system with more natural responses with less computational resources than fine-tuning. |
Long-Form Speech Translation through Segmentation with Finite-State Decoding Constraints on Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a challenge in speech translation is that plenty of spoken content is long-form, but short units are necessary for obtaining high-quality translations. |
| Approach: | They propose a large language model to split long ASR transcripts into segments that can be independently translated to maximize translation quality. |
| Outcome: | The proposed model improves the average BLEU by 2.9 points for English–German, English–Spanish, and English–Arabic TED talk translation in 9 sets. |
Contrastive Demonstration Tuning for Pre-trained Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies focus on searching discrete or continuous prompts or optimized verbalizers, yet the demonstration examples are crucial for an excellent final performance of prompt-tuning. |
| Approach: | They propose a pluggable, extensible, and efficient approach to prompt tuning that is free of demonstration sampling. |
| Outcome: | The proposed approach can be pluggable, extensible, and efficient on 16 datasets. |
Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning (2023.acl-short)
Copied to clipboard
Mustafa Ozdayi, Charith Peris, Jack FitzGerald, Christophe Dupuy, Jimit Majmudar, Haidar Khan, Rahil Parikh, Rahul Gupta
| Challenge: | Large Language Models memorize significant portions of training data, which poses privacy risk. |
| Approach: | They propose a prompt-tuning approach to control the extraction rates of memorized content in large language models. |
| Outcome: | The proposed techniques yield 9.3% increase in extraction rate compared to baseline model . the proposed defense achieves 97.7% reduction with a perplexity increase of 16.9% . |
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies suggest that pre-trained language models have gained rich knowledge during pre-training. |
| Approach: | They propose to tune pre-trained language models with task-specific prompts to improve and stabilize prompttuning. |
| Outcome: | Extensive experiments on zero and few-shot text classification tasks show that prompt-tuning improves and stabilizes prompttun-ing. |
AlignRE: An Encoding and Semantic Alignment Approach for Zero-Shot Relation Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing prototype-based methods for ZSRE ignore abundant side information and suffer from a significant encoding gap between prototypes and sentences. |
| Approach: | They propose a framework to encode schema alignment to enhance prototype-based ZSRE methods. |
| Outcome: | The proposed method outperforms existing methods on FewRel and Wiki-ZSL datasets and exhibits substantially faster performance and reduces the need for extensive manual labor in prototype construction. |
PCBERT: Parent and Child BERT for Chinese Few-shot NER (2022.coling-1)
Copied to clipboard
| Challenge: | Existing approaches to improve model performance on few-shot or zero-shot datasets are not effective for Chinese few- shot NER. |
| Approach: | They propose a prompt-based Parent and Child BERT for Chinese few-shot NER to train an annotating model on high-resource datasets and then discover more implicit labels on low-resourced datasets. |
| Outcome: | The proposed model can be used on Weibo and other Chinese NER datasets and it is shown to be effective in few-shot learning. |
Making Pretrained Language Models Good Long-tailed Learners (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prompt-tuning has shown appealing performance in few-shot classification . however, it is less promising in long-tailed classification due to long tail . |
| Approach: | They propose to use prompt-tuning to make pretrained language models at least good long-tailed learners by bridging the gap between prompt- and commonly used finetun. |
| Outcome: | The proposed method makes pretrained language models at least good long-tailed learners, bridging the gap between prompt-tuning and finetunation. |
DisCup: Discriminator Cooperative Unlikelihood Prompt-tuning for Controllable Text Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing prompt tuning approaches for attribute-controllable text generation are difficult to implement due to the lack of interpretability of deep neural networks. |
| Approach: | They propose a new approach that incorporates attribute knowledge of discriminator to optimize prompt tuning by steering a frozen CLM to produce attribute-specific texts. |
| Outcome: | The proposed approach can achieve state-of-the-art control performance while maintaining high-quality text generation. |
CEPT: A Contrast-Enhanced Prompt-Tuning Framework for Emotion Recognition in Conversation (2024.lrec-main)
Copied to clipboard
| Challenge: | Emotion recognition in conversation research suffers from data imbalance and the presence of similar linguistic expressions for different emotions. |
| Approach: | They propose a Contrast-Enhanced Prompt-Tuning framework that transforms an ERC task into a Masked Language Modeling task and generates the emotion for each utterance in the conversation. |
| Outcome: | The proposed framework outperforms the state-of-the-art methods on all three benchmark datasets and excels in recognizing minority emotions. |
Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained multilingual language models show significant performance gains for zero-shot cross-lingual model transfer on a wide range of natural language understanding (NLU) tasks. |
| Approach: | They do cross-lingual evaluation using prompt tuning and compare it with fine-tuning . prompt tuning achieves much better cross-linguistic transfer than fine- tuning . |
| Outcome: | The results show that prompt tuning achieves better cross-lingual transfer than fine-tuning across datasets, with only 0.1% to 0.3% tuned parameters. |
Bi-Directional Iterative Prompt-Tuning for Event Argument Extraction (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing prompt-tuning methods for event argument extraction lack entity information . eAE is a key step of event extraction, but it requires a pre-trained language model to extract event arguments. |
| Approach: | They propose a prompt-tuning method that takes advantage of entity information and pre-trained language models. |
| Outcome: | The proposed method outperforms the state-of-the-art prompt-tuning methods on an english dataset. |
Think Twice: Perspective-Taking Improves Large Language Models’ Theory-of-Mind Capabilities (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances to LLMs’ reasoning capabilities from simple yet effective prompting techniques such as Chain-of-Thought have seen limited applicability to ToM. |
| Approach: | They propose a two-stage prompting framework inspired by Simulation Theory's notion of perspective-taking to elicit Theory-of-Mind capabilities in Large Language Models. |
| Outcome: | The proposed framework shows that it is much more effective than existing prompts. |
Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue System (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work on end-to-end dialogue models with pre-trained dialogue corpora shows promising performance in the conversational system. |
| Approach: | They propose an end-to-end TOD system with task-optimized adapters which learn independently per task adding only small number of parameters after fixed layers of pre-trained network. |
| Outcome: | The proposed system achieves state-of-the-art performance on the MultiWOZ benchmark compared to existing models. |
Event-enhanced Retrieval in Real-time Search (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing embedding-based retrieval models face the "semantic drift" problem . a low adoption rate of retrieval results is evident in real-time search scenarios . |
| Approach: | They propose an embedding-based retrieval approach that enhances real-time retrieval performance by adding contrastive learning to the dual-encoder model. |
| Outcome: | The proposed approach improves the dual-encoder model of traditional EBR. |
PRO-CS : An Instance-Based Prompt Composition Technique for Code-Switched Tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Code-switching (CS) data is ubiquitous in today’s globalized world, but the dearth of annotated datasets in code-switch tasks poses a significant challenge for transfer learning in limited-resource setups. |
| Approach: | They propose a prompt composition technique that outperforms prompt-tuning and fine-tuned prompt-based prompt composition techniques for CS tasks that combine language and task knowledge. |
| Outcome: | The proposed approach outperforms prompt-tuning and fine-tuned approaches on 10 datasets across 4 languages and achieves competitive results in low-resource cross-lingual and cross-task setting. |
IntCoOp: Interpretability-Aware Vision-Language Prompt Tuning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing prompt-tuning frameworks lack interpretability, limiting their ability to understand compositional nature of images. |
| Approach: | They propose a prompt-tuning method that integrates compositional attributes into manual prompts to enhance image-text alignment scores. |
| Outcome: | The proposed method improves CoOp performance by 7.35% across 10 diverse datasets. |
Prompt Tuning for Few-shot Relation Extraction via Modeling Global and Local Graphs (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent studies show that prompt-tuning is effective for few-shot relation extraction tasks. |
| Approach: | They propose to incorporate the knowledge in relation labels into prompt-tuning by inserting prompt templates into the input. |
| Outcome: | The proposed method improves on four datasets under low-resource conditions. |
SynPrompt: Syntax-aware Enhanced Prompt Engineering for Aspect-based Sentiment Analysis (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods of prompt-tuning for Aspect-based Sentiment Analysis (ABSA) are crude and simple. |
| Approach: | They propose a Syntax-aware Enhanced Prompt method which mines syntactic information related to aspect words from the syntaktic dependency tree. |
| Outcome: | The proposed method exploits the syntactic knowledge embedded in PLMs and achieves favorable results on three benchmark datasets. |
Target-Adaptive Consistency Enhanced Prompt-Tuning for Multi-Domain Stance Detection (2024.lrec-main)
Copied to clipboard
| Challenge: | Stance detection is a fundamental task in natural language processing, but it is challenging due to diverse expressions and topics related to the targets from multiple domains. |
| Approach: | They propose a prompt-tuning method that incorporates target knowledge and prior knowledge to construct target-adaptive verbalizers for diverse domains. |
| Outcome: | The proposed method outperforms the state-of-the-art methods on nine stance detection datasets from multiple domains. |