Modular and Parameter-Efficient Multimodal Fusion with Prompting (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent research has made impressive progress in large-scale multimodal pre-training. |
| Approach: | They propose to use prompt vectors to align multimodal modalities by pretraining text inputs with prompts or embedding vectors. |
| Outcome: | The proposed method achieves comparable performance to several other multimodal fusion methods in low-resource settings. |
Similar Papers
Prompt Tuning for Unified Multimodal Pretrained Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt tuning has demonstrated success in natural language pretraining and even vision pretraining. |
| Approach: | They propose to apply prompt tuning to a unified sequence-to-sequence pretrained model by adding a sequence of learnable embeddings to each layer and finetuning the pretrained models on downstream tasks. |
| Outcome: | The proposed method outperforms other parameter-efficient tuning methods on multimodal models and is robust against adversarial attacks. |
promptolution: A Unified, Modular Framework for Prompt Optimization (2026.eacl-demo)
Copied to clipboard
| Challenge: | Existing implementations of prompt optimization are tied to unmaintained, isolated codebases or require invasive integration into application frameworks. |
| Approach: | They propose a unified, modular open-source framework that integrates multiple contemporary discrete prompt optimizers within a single extensible system for both practitioners and researchers. |
| Outcome: | The proposed framework integrates multiple discrete prompt optimizers, supports systematic and reproducible benchmarking, and returns framework-agnostic prompt strings, enabling seamless integration into existing LLM pipelines while remaining agnosite to the underlying model implementation. |
M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning (2024.emnlp-main)
Copied to clipboard
Taowen Wang, Yiyang Liu, James Liang, Junhan Zhao, Yiming Cui, Yuning Mao, Shaoliang Nie, Jiahao Liu, Fuli Feng, Zenglin Xu, Cheng Han, Lifu Huang, Qifan Wang, Dongfang Liu
| Challenge: | Multimodal Large Language Models (MLLMs) exhibit remarkable performance across a wide range of domains. |
| Approach: | They propose a multimodal prompt tuning approach for efficient instruction tuning of MLLMs. |
| Outcome: | The proposed approach shows superior performance on multimodal evaluation datasets compared to state-of-the-art methods. |
Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Prompt tuning is a parameter-efficient approach to adapting pre-trained language models to downstream tasks. |
| Approach: | They propose to combine pre-trained modules with pre-trains to boost prompt tuning for few-shot learning. |
| Outcome: | The proposed model outperforms prompt tuning, full model tuning, and prior prompt pre-training methods in few-shot learning settings. |
The Power of Scale for Parameter-Efficient Prompt Tuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Unlike discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples. |
| Approach: | They propose a mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. |
| Outcome: | The proposed method outperforms fewshot learning using GPT-3 and matches the quality of model tuning as models exceed billions of parameters. |
Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis often fail due to equipment failure, data corruption, privacy issues and the like. |
| Approach: | They propose a multimodal Transformer framework using prompt learning to address the issue of missing modalities. |
| Outcome: | The proposed framework outperforms existing methods significantly across evaluation metrics. |
Modular and Parameter-Efficient Fine-Tuning for NLP Models (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | State-of-the-art language models in NLP perform best when fine-tuned even on small datasets. |
| Approach: | They provide an overview of parameter-efficient fine-tuning methods and highlight similarities and differences . they highlight benefits and usage scenarios of a neglected property of parameter efficient models . |
| Outcome: | This paper provides an overview of parameter-efficient fine-tuning methods . it highlights similarities and differences by presenting them in a unified view . |
APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng, Lifu Huang, Xiaojun Quan, Zenglin Xu, Dongfang Liu
| Challenge: | Existing prompt tuning methods only introduce prompts at the input layer, limiting performance and leaving large room for improvement. |
| Approach: | They propose a method that involves tuning a small set of soft prompts for pre-trained language models. |
| Outcome: | The proposed method outperforms state-of-the-art methods with pre-trained models on the SuperGLUE benchmark. |
SMoP: Towards Efficient and Effective Prompt Tuning with Sparse Mixture-of-Prompts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prompt tuning has emerged as a successful parameter-efficient alternative to the full fine-tuning of language models. |
| Approach: | They propose a prompt tuning method that utilizes short soft prompts for efficient training and inference while maintaining performance gains typically induced by longer soft prompt. |
| Outcome: | The proposed method outperforms baseline methods while preserving memory usage. |
Prompt Compression for Large Language Models: A Survey (2025.naacl-long)
Copied to clipboard
| Challenge: | Current methods for improving LLM efficiency focus on optimizing the model itself, while prompt-centric methods focus on lowering the complexity of input. |
| Approach: | They propose to use prompt compression to optimize the compression encoder and combine hard and soft prompt methods to improve the efficiency of LLMs. |
| Outcome: | The proposed methods are categorized into hard prompt methods and soft prompt methods. |