Enabling Natural Zero-Shot Prompting on Encoder Models via Statement-Tuning (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit remarkable capabilities in zero-shot and few-shot settings, but they struggle with extending to few- shot and zero- shot settings due to their architectural design. |
| Approach: | They propose a technique that models discriminative tasks as a set of finite statements and trains an encoder model to discriminate between the potential statements to determine the label. |
| Outcome: | The proposed method achieves competitive performance compared to state-of-the-art LLMs with significantly fewer parameters. |
Similar Papers
Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models (2025.findings-acl)
Copied to clipboard
Ahmed Elshabrawy, Thanh-Nhi Nguyen, Yeeun Kang, Lihan Feng, Annant Jain, Faadil Abdullah Shaikh, Jonibek Mansurov, Mohamed Fazli Mohamed Imam, Jesus-German Ortiz-Barajas, Rendi Chevi, Alham Fikri Aji
| Challenge: | Large Language Models excel in zero-shot and few-shot tasks, but their architecture makes them difficult to use. |
| Approach: | They adapt Large Language Models (LLMs) for zero-shot generalization using Statement Tuning . they find encoders can achieve zero- shot cross-lingual generalization . |
| Outcome: | The proposed model generalizes well across languages while being more efficient. |
Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibration (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research demonstrates that multilingual encoder models are capable of zero-shot cross-lingual learning by using cloze-style prompts. |
| Approach: | They propose to reformulate input examples into cloze-style prompts to perform zero-shot multilingual tasks or linguistic probing by predicting label words at the masked token position. |
| Outcome: | The proposed method performs zero-shot multilingual tasks without updating parameters. |
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Instruction tuning is emerging in NLP, but has not been explored for dialogue-related tasks. |
| Approach: | They propose an instruction tuning framework for dialogue that leverages natural language instructions with language models to induce zero-shot generalization on unseen tasks. |
| Outcome: | The proposed framework enables good zero-shot performance on unseen datasets and tasks such as dialogue evaluation and intent detection. |
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large pre-trained language models (LMs) have a surprising ability to perform zero-shot learning. |
| Approach: | They propose to fine-tune pre-trained language models to optimize the zero-shot learning objective by aggregating 43 existing datasets and annotating 441 label descriptions in a question-answering format. |
| Outcome: | The proposed model outperforms a same-sized QA model and the previous SOTA zero-shot learning system on unseen tasks. |
Prompt-free and Efficient Few-shot Learning with Language Models (2022.acl-long)
Copied to clipboard
Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, Majid Yazdani
| Challenge: | Existing methods for few-shot fine-tuning of pretrained language models require carefully engineered prompts and verbalizers to convert inputs into a cloze-format that the PLM can score. |
| Approach: | They propose a method for few-shot fine-tuning of pretrained language models that uses task-specific adapters instead of manually engineered prompts and verbalizers. |
| Outcome: | The proposed method outperforms existing state-of-the-art methods on a wide range of few shot NLP tasks. |
Making Pre-trained Language Models Better Few-shot Learners (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that the GPT-3 model can perform few-shots on language understanding tasks with a natural-language prompt and a few task demonstrations. |
| Approach: | They propose a technique for fine-tuning language models using a few examples . they propose LM-BFF, which uses prompt-based fine-uning and a pipeline for automating prompt generation . |
| Outcome: | The proposed approach outperforms standard fine-tuning procedures on a range of NLP tasks. |
Formulating Few-shot Fine-tuning Towards Language Model Pre-training: A Pilot Study on Named Entity Recognition (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models are limited . we propose a few-shot fine-uning framework for NER . |
| Approach: | They propose a few-shot fine-tuning framework for named entity recognition (NER) they propose three new types of tokens, "is-entity", "which-type" and "bracket" |
| Outcome: | The proposed framework improves on pre-trained language models on several benchmark datasets. |
Prompt Consistency for Zero-Shot Task Generalization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has shown that pre-trained language models can perform zero-shot generalization to new tasks without annotated examples. |
| Approach: | They propose to regularize prompt consistency to encourage consistent predictions over a diverse set of prompts. |
| Outcome: | The proposed approach outperforms the state-of-the-art zero-shot learner, T0, on 9 out of 11 datasets across 4 NLP tasks by 10.6 absolute points in terms of accuracy. |
Towards Unified Prompt Tuning for Few-shot Text Classification (2022.findings-emnlp)
Copied to clipboard
Jianing Wang, Chengyu Wang, Fuli Luo, Chuanqi Tan, Minghui Qiu, Fei Yang, Qiuhui Shi, Songfang Huang, Ming Gao
| Challenge: | Prompt-based fine-tuning has boosted performance of Pre-trained Language Models (PLMs) on few-shot text classification, but PLMs are unfamiliar with prompt-style expressions during pre-training, which limits the few- shot learning performance on downstream tasks. |
| Approach: | They propose a framework for prompt-based fine-tuning that captures prompting semantics from non-target NLP datasets and propose 'Prompt-Options-Verbalizer' for joint prompt learning across different NLP tasks. |
| Outcome: | Experiments show that the proposed framework outperforms state-of-the-art prompt-based fine-tuning frameworks on few-shot text classification tasks. |
Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that instruction tuning is effective in instruction learning for unseen tasks, but it relies on a large amount of human-annotated samples, which restricts its generalization. |
| Approach: | They propose an instruction tuning technique which fine-tunes a pre-trained language model on a massive collection of tasks described via human-craft instructions and then tests its generalization ability on unseen tasks. |
| Outcome: | The proposed method improves IT performance versus labeled data and training tasks by constructing pseudo-labeled data from unlabele . data is used to build a model that can learn from human instructions for zero-shot generalization on unseen tasks. |