Papers by Timo Schick
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2023.acl-long)
Copied to clipboard
| Challenge: | Instruction tuning enables pretrained language models to perform new tasks from inference-time natural language descriptions without human supervision. |
| Approach: | They propose to use a dataset of natural language instructions to generate large datasets with no human supervision. |
| Outcome: | The proposed dataset outperforms open-source models on various benchmarks, and is cost-effective. |
LongForm: Effective Instruction Tuning with Reverse Instructions (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on instruction tuning relies on expensive human annotation and crowd-sourced datasets with alignment issues. |
| Approach: | They propose a method to generate instructions via LLMs from human-written corpus examples using reverse instructions. |
| Outcome: | The proposed method outperforms larger language models without instruction tuning on tasks such as story/recipe generation and long-form question answering. |
Few-Shot Text Generation with Natural Language Instructions (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to text generation combine task descriptions and examples with supervised learning. |
| Approach: | They propose a method for text generation that is based on pattern-exploiting training. |
| Outcome: | The proposed approach improves on several summarization and headline generation datasets. |
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP (2021.tacl-1)
Copied to clipboard
| Challenge: | Pretrained language models pick up and reproduce undesirable biases when trained on large, unfiltered crawls from the Internet. |
| Approach: | They propose a decoding algorithm that, given only a textual description of the undesired behavior, reduces the probability of a language model producing problematic text. |
| Outcome: | The proposed approach reduces the probability of a language model producing problematic text by giving only a textual description of the undesired behavior. |
Active Learning Principles for In-Context Learning with Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning has significantly enhanced predictive performance in few-shot learning settings. |
| Approach: | They propose to use pool-based Active Learning to identify the most informative demonstrations for few-shot learning over a single iteration to identify best demonstrations. |
| Outcome: | The proposed model outperforms all other methods, including random sampling, in the analysis of 24 classification and multi-choice tasks. |
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing approaches to learning from examples are limited due to the vast number of languages, domains and tasks. |
| Approach: | They propose a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task. |
| Outcome: | The proposed approach outperforms supervised training and strong semi-supervised approaches in low-resource settings by a large margin. |
Task-aware Retrieval with Instructions (2023.findings-acl)
Copied to clipboard
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, Wen-tau Yih
| Challenge: | Existing models that learn intents from labeled data are complicated and require a vast number of annotated examples to train a model. |
| Approach: | They propose a general-purpose task-aware retrieval system with instructions that can adapt to a new task without any parameter updates. |
| Outcome: | The proposed system outperforms two benchmarks on a set of domains and tasks on X2-Retrieval. |
CoDA21: Evaluating Language Understanding Capabilities of NLP Models With Context-Definition Alignment (2022.acl-short)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) have achieved superhuman performance on many benchmarks, creating a need for harder tasks. |
| Approach: | They propose a benchmark that measures natural language understanding (NLU) abilities of pretrained language models. |
| Outcome: | The proposed benchmark measures the ability of pretrained language models to perform on many tasks. |
Generating Datasets with Pretrained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to obtain high-quality sentence embeddings from pretrained language models require labeled data or finetuned on large set of labeles. |
| Approach: | They propose to use generative abilities of large and high-performing PLMs to generate entire datasets of labeled text pairs from scratch and fine tune much smaller and more efficient models. |
| Outcome: | The proposed approach outperforms baselines on several semantic textual similarity datasets. |
MEAL: Stable and Active Learning for Few-Shot Prompting (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for few-shot classification have high variance across different sets of few shots and finetuning runs. |
| Approach: | They propose novel ensembling methods that significantly reduce run variability and introduce a new active learning criterion for *data selection*. |
| Outcome: | The proposed method significantly reduces run variability and improves performance on five tasks. |
Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts (N19-1)
Copied to clipboard
| Challenge: | Mimicking has been proposed as a solution to learning high-quality embeddings for rare words because of sparse context information. |
| Approach: | They propose a method to reproduce embeddings of frequent words from their surface form and then use it to compute embedds for rare words. |
| Outcome: | The proposed model outperforms previous work on rare and medium-frequency words. |
Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to few-shot text classification require domain expertise and an understanding of the language model's abilities to define the mapping between words and labels. |
| Approach: | They propose a method that converts textual inputs to cloze questions that contain some form of task description and processes them with a pretrained language model to map the predicted words to labels. |
| Outcome: | The proposed approach performs almost as well as hand-crafted label-to-word mappings for a number of tasks with small amounts of training data. |
BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to improve word embeddings for rare words are limited to uncontextualized word embeds. |
| Approach: | They propose a powerful architecture that can infer high-quality embeddings for rare words . they use the surface form and contexts of a word to interact in a deep architecture . |
| Outcome: | The proposed architecture can infer high-quality embeddings for rare words that are suitable as input representations for deep language models. |
Leveraging QA Datasets to Improve Generative Data Augmentation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in NLP have substantially improved the capability of pretrained language models to generate high-quality text. |
| Approach: | They propose to reformulate data generation as context generation for a given question-answer (QA) pair and leverage QA datasets for training context generators. |
| Outcome: | The proposed approach improves performance for few-shot and zero-shot classification datasets on multiple classification dataset. |
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2021.naacl-main)
Copied to clipboard
| Challenge: | Pretraining ever-larger language models on massive corpora requires enormous amounts of compute. |
| Approach: | They propose to convert textual inputs into cloze questions that contain a task description . they also exploit unlabeled data to improve their performance . |
| Outcome: | The proposed model outperforms GPT-3 with PET/iPET with cloze questions and unlabeled data. |
True Few-Shot Learning with Prompts—A Real-World Perspective (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent work has cast doubt on the effectiveness of prompt-based approaches at few-shot learning in a “true” few- shot setting. |
| Approach: | They propose a method that combines textual instructions with example-based finetuning to give prompt-based learning a powerful method for few-shot text classification. |
| Outcome: | The proposed method performs well in a few-shot setting without a dev set and is able to handle multiple prompts. |