Papers by Sebastian Riedel
ProoFVer: Natural Logic Theorem Proving for Fact Verification (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent fact verification systems rely on neural network classifiers for veracity prediction, which lack explainability. |
| Approach: | They propose a model that generates natural logic-based inferences as proofs using lexical mutations between spans in the claim and the evidence retrieved. |
| Outcome: | The proposed model has highest label accuracy and second best score in the FEVER leaderboard. |
A Few More Examples May Be Worth Billions of Parameters (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on few-shot learning for natural language tasks explores the dynamics of scaling up either the number of model parameters or labeled examples while controlling for the other variable by setting it to a constant. |
| Approach: | They explore the dynamics of scaling up the number of model parameters versus the number labeled examples across a wide variety of tasks. |
| Outcome: | The results show that scaling parameters yields performance improvements, while adding examples does not. |
Joint Verification and Reranking for Open Fact Checking Over Tables (2021.acl-long)
Copied to clipboard
Michael Sejr Schlichtkrull, Vladimir Karpukhin, Barlas Oguz, Mike Lewis, Wen-tau Yih, Sebastian Riedel
| Challenge: | Existing research into structured data has focused on textual data and the closed-domain setting is not reflective of real-world fact checking tasks. |
| Approach: | They propose a joint reranking-and-verification model which fuses evidence documents in the verification component and a heuristic retrieval baseline. |
| Outcome: | The proposed model achieves comparable performance to the closed-domain state-of-the-art on the TabFact dataset and significantly improves over a heuristic retrieval baseline. |
Interpretation of Natural Language Rules in Conversational Machine Reading (D18-1)
Copied to clipboard
Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, Sebastian Riedel
| Challenge: | Existing work on question answering problems requires the reading of text because it contains a recipe to derive an answer together with the reader’s background knowledge. |
| Approach: | They formalise a task and develop a crowd-sourcing strategy to collect 37k task instances based on real-world rules and crowd-generated questions and scenarios. |
| Outcome: | The proposed task is based on 37k task instances based in real-world rules and crowd-generated questions and scenarios. |
Training Adaptive Computation for Open-Domain Question Answering with Computational Constraints (2021.acl-short)
Copied to clipboard
| Challenge: | Adaptive Computation (AC) has been shown to be effective in improving the efficiency of Open-Domain Question Answering systems. |
| Approach: | They propose an AC method that can be applied to an existing ODQA model and can be trained efficiently on a single GPU. |
| Outcome: | The proposed method improves upon a state-of-the-art model on two datasets and is more accurate than previous AC methods due to the stronger base ODQA model. |
PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them (2021.tacl-1)
Copied to clipboard
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, Sebastian Riedel
| Challenge: | Open-domain Question Answering models that directly leverage question-answer (QA) pairs show promise in terms of speed and memory compared with conventional models which retrieve and read from text corpora. |
| Approach: | They propose a question-answer (QA)-pair retriever to facilitate improved QA-patch models by introducing Probably Asked Questions (PAQ) they propose QA pair retriever, RePAQ, which preempts and caches test questions, enabling it to match the accuracy of recent retrieve-and-read models, whilst being significantly faster. |
| Outcome: | The proposed model outperforms baseline models by 5% but trails RePAQ by 15% . it can be configured for size (under 500MB) or speed (over 1K questions per second) while retaining high accuracy. |
Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts? (2025.findings-acl)
Copied to clipboard
| Challenge: | Latent multi-hop reasoning is a problem in Large Language Models that can develop shortcuts by encountering the head entity and answer entity in training sequences. |
| Approach: | They propose desiderata for shortcut-free evaluation of latent multi-hop reasoning ability . they exclude test queries where head and answer entities might have co-appeared . |
| Outcome: | The proposed model can latently recall and compose single-hop facts without shortcuts, but only for certain types of queries. |
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity (2022.acl-long)
Copied to clipboard
| Challenge: | Large pretrained language models can generate text classification results that match fully supervised models. |
| Approach: | They propose to use a few sample training to determine which permutations are performant . they use generative language models to construct an artificial development set . |
| Outcome: | The proposed model outperforms fully-supervised models in eleven text classification tasks. |
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | a new approach to generate adversarial data is needed to improve question answering models . crowdworkers can fool a model only 8.8% of the time, compared to 17.6% for a trained model without synthetic data. |
| Approach: | They develop a pipeline that generates questions and then filters or labels them to improve quality. |
| Outcome: | The proposed approach improves state-of-the-art on a human-written adversarial dataset by 3.7F1 and improves model generalisation on nine of the twelve MRQA datasets. |
Dynabench: Rethinking Benchmarking in NLP (2021.naacl-main)
Copied to clipboard
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, Adina Williams
| Challenge: | Dynabench is an open-source platform for dynamic dataset creation and model benchmarking. |
| Approach: | They propose an open-source platform for dynamic dataset creation and model benchmarking. |
| Outcome: | The proposed platform can be used to create models that fail on simple challenges and falter in real-world scenarios. |
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data (2020.acl-main)
Copied to clipboard
| Challenge: | Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language understanding tasks. |
| Approach: | They propose a pretrained language model that jointly learns representations for NL sentences and (semi-)structured tables. |
| Outcome: | The proposed model performs best on the weakly-supervised semantic parsing benchmark WikiTableQuestions while performing competitively on the text-to-SQL dataset Spider. |
MLQA: Evaluating Cross-lingual Extractive Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | Question answering (QA) models have shown rapid progress enabled by the availability of large, high-quality benchmark datasets. |
| Approach: | They present a multi-way aligned extractive QA evaluation benchmark in 7 languages . they evaluate state-of-the-art cross-lingual models and machine-translation-based baselines . |
| Outcome: | The proposed model is based on MLQA, which has over 12K instances in english and 5K in each other language. |
LOFT: Scalable and More Realistic Long-Context Evaluation (2025.findings-naacl)
Copied to clipboard
Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Séb Arnold, Vincent Perot, Siddharth Dalmia, Hexiang Hu, Xudong Lin, Panupong Pasupat, Aida Amini, Jeremy R. Cole, Sebastian Riedel, Iftekhar Naim, Ming-Wei Chang, Kelvin Guu
| Challenge: | Long-context language models (LCLMs) can be used to perform tasks traditionally reliant on external tools like retrieval systems or databases. |
| Approach: | They propose a benchmark to evaluate LCLMs' performance on in-context retrieval and reasoning tasks using a set of tokens. |
| Outcome: | The proposed model outperforms state-of-the-art retrieval and RAG systems on in-context retrieval tasks while still requiring prompting strategies. |
Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension (2020.tacl-1)
Copied to clipboard
| Challenge: | Innovations in annotation methodologies have been a catalyst for Reading Comprehension (RC) datasets and models. |
| Approach: | They propose to use a model-in-the-annotation-loop approach to train adversarial models in three different settings to explore reproducibility of the adversarial effect, transfer from data collected with varying model- in-the loop strengths, and generalization to data collected without a modeling model. |
| Outcome: | The proposed approach achieves 39.9F1 on questions it cannot answer when trained on SQUAD, but lower than when trained using RoBERTa itself (41.0F1). |
Language Models as Knowledge Bases? (D19-1)
Copied to clipboard
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller
| Challenge: | Recent advances in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. |
| Approach: | They present a method for pretraining language models on large textual corpora . they find that they can store relational knowledge and answer queries structured as "fill-in-the-blank" queries. |
| Outcome: | The proposed language models can recall factual knowledge without fine-tuning without fine tuning . the proposed models can answer queries structured as "fill-in-the-blank" cloze statements . |
Jack the Reader – A Machine Reading Framework (P18-4)
Copied to clipboard
Dirk Weissenborn, Pasquale Minervini, Isabelle Augenstein, Johannes Welbl, Tim Rocktäschel, Matko Bošnjak, Jeff Mitchell, Thomas Demeester, Tim Dettmers, Pontus Stenetorp, Sebastian Riedel
| Challenge: | Many Machine Reading and Natural Language Understanding tasks require reading supporting text in order to answer questions. |
| Approach: | They propose a framework for Machine Reading that allows for quick prototyping by component reuse and evaluation of new models on existing datasets. |
| Outcome: | The proposed framework supports question answering, natural language inference and link prediction tasks. |
How Decoding Strategies Affect the Verifiability of Generated Text (2020.findings-emnlp)
Copied to clipboard
Luca Massarelli, Fabio Petroni, Aleksandra Piktus, Myle Ott, Tim Rocktäschel, Vassilis Plachouras, Fabrizio Silvestri, Sebastian Riedel
| Challenge: | Recent advances in pre-trained language models have generated text of an increasingly high quality. |
| Approach: | They propose a decoding strategy that produces less repetitive and more verifiable text. |
| Outcome: | The proposed method produces less repetitive and more verifiable text than previously used decoding strategies. |
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work on multilingual pre-trained models has focused on pre-training transformers on concatenated corpora of a large number of languages. |
| Approach: | They propose a language-specific module approach that allows for more languages to be trained post-hoc. |
| Outcome: | The proposed model can be pre-trained on multiple languages with no drop in performance . |
Generating Fact Checking Briefs (2020.emnlp-main)
Copied to clipboard
Angela Fan, Aleksandra Piktus, Fabio Petroni, Guillaume Wenzek, Marzieh Saeidi, Andreas Vlachos, Antoine Bordes, Sebastian Riedel
| Challenge: | Existing work has framed fact checking as classification, often supported by a claim as input. |
| Approach: | They propose to use natural language briefs to increase the accuracy of fact checking . they show that QABriefer increases the accuracy by 10% while QABries reduce time . |
| Outcome: | The proposed model increases the accuracy of crowdworkers by 10% while reducing the time required by 20%. |
EDIN: An End-to-end Benchmark and Pipeline for Unknown Entity Discovery and Indexing (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on Entity Linking assumes that the knowledge base is complete and all mentions can be linked. |
| Approach: | They propose a temporally segmented Unknown Entity Discovery and Indexing (EDIN) benchmark where unknown entities have to be integrated into existing entity linking systems. |
| Outcome: | The proposed system detects, clusters, and indexes mentions of unknown entities in context. |
Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers (P18-1)
Copied to clipboard
| Challenge: | Numeracy is the ability to understand and work with numbers. |
| Approach: | They propose a neural architecture that uses a continuous probability density function to model numerals from an open vocabulary using hierarchical models. |
| Outcome: | The proposed model reduces errors by 18% and 54% on clinical and scientific datasets compared to the second best model for each dataset . |
Evaluating Rewards for Question Generation Models (N19-1)
Copied to clipboard
| Challenge: | Recent approaches to question generation have used modifications to a Seq2Seq architecture inspired by advances in machine translation. |
| Approach: | They propose to use a Seq2Seq architecture to train models to generate one-step-ahead predictions, but at test time, the model is asked to generate a whole sequence, causing errors to propagate through the generation process. |
| Outcome: | The proposed model is trained to generate a plausible question, conditioned on an input document and answer span within that document. |
Multilingual Autoregressive Entity Linking (2022.tacl-1)
Copied to clipboard
Nicola De Cao, Ledell Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Plekhanov, Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, Fabio Petroni
| Challenge: | mGENRE is a sequence-to-sequence system for multilingual entity linking . mGenRE is used to solve language-specific mentions to a multilingual Knowledge Base . |
| Approach: | They propose a sequence-to-sequence system for multilingual entity linking . they match language-specific mentions against a multilingual Knowledge Base (KB) mGENRE is a sequential system that predicts the name of the target entity token-by-token . |
| Outcome: | The proposed system improves on three popular MEL benchmarks and shows improvements in accuracy. |
How Well Can Reasoning Models Identify and Recover from Unhelpful Thoughts? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent reasoning models show the ability to reflect, backtrack, and self-validate their reasoning . |
| Approach: | They investigate how well models identify and recover from unhelpful thoughts . they show that models are effective at identifying most unhelpfully thoughts - but struggle to recover from the same thoughts when they are injected into their thinking process . |
| Outcome: | The proposed model can identify unhelpful thoughts but struggle to recover from them . larger models struggle more than smaller models to recover short irrelevant thoughts . |
Unsupervised Question Answering by Cloze Translation (P19-1)
Copied to clipboard
| Challenge: | Existing QA datasets only available for limited domains and languages. |
| Approach: | They propose to generate context, question and answer triples in an unsupervised manner and synthesize extractive QA training data automatically. |
| Outcome: | The proposed approach outperforms existing QA models on a common EQA benchmark dataset. |
Task-aware Retrieval with Instructions (2023.findings-acl)
Copied to clipboard
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, Wen-tau Yih
| Challenge: | Existing models that learn intents from labeled data are complicated and require a vast number of annotated examples to train a model. |
| Approach: | They propose a general-purpose task-aware retrieval system with instructions that can adapt to a new task without any parameter updates. |
| Outcome: | The proposed system outperforms two benchmarks on a set of domains and tasks on X2-Retrieval. |
Boosted Dense Retriever (2022.naacl-main)
Copied to clipboard
| Challenge: | DrBoost is a dense retrieval ensemble that is trained in stages to correct retrieval mistakes . it produces representations which are 4x more compact, while delivering comparable retrieval results. |
| Approach: | They propose a dense retrieval ensemble inspired by boosting that is trained in stages . they produce representations which are 4x more compact, while delivering comparable retrieval results . |
| Outcome: | The proposed model performs surprisingly well under approximate search with coarse quantization, reducing latency and bandwidth needs by another 4x. |
Open Vocabulary Extreme Classification Using Generative Models (2022.findings-acl)
Copied to clipboard
Daniel Simig, Fabio Petroni, Pouya Yanki, Kashyap Popat, Christina Du, Sebastian Riedel, Majid Yazdani
| Challenge: | Extreme multi-label classification (XMC) aims at tagging content with subset of labels from an extremely large label set. |
| Approach: | They propose a model that predicts a set of labels outside of the known vocabulary by using a loss-dependent loss-based loss-free model. |
| Outcome: | The proposed model can predict labels outside the known vocabulary while performing on par with state-of-the-art solutions for known labels. |
Database reasoning over text (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models cannot handle database queries such as “List/Count all female athletes who were born in 20th century”. |
| Approach: | They propose a modular architecture to answer database-style queries over multiple spans from text and aggregate them at scale. |
| Outcome: | The proposed architecture scales to databases containing thousands of facts whereas current models are limited by how many facts can be encoded. |
An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods rely on parametric models that store knowledge in parameters or retrieval-augmented models that have access to external knowledge sources. |
| Approach: | They propose a parametric parametric model that stores knowledge in its parameters or a retrieval-augmented model that has access to external knowledge sources. |
| Outcome: | The proposed method runs substantially faster across the board and produces more accurate results on WoW and ELI5. |
Scalable Zero-shot Entity Linking with Dense Entity Retrieval (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for entity linking use manually curated mention tables and incoming Wikipedia link popularity. |
| Approach: | They propose a BERT-based entity linking model with a bi-encoder that embeds the mention context and the entity descriptions and then re-ranked the candidate with . they also evaluate the accuracy-speed trade-off inherent to large pre-trained models. |
| Outcome: | The proposed model is state-of-the-art on recent zero-shot benchmarks and established non-zero-shot evaluations. |
Zero-Shot Transfer Learning for Event Extraction (P18-1)
Copied to clipboard
| Challenge: | Existing supervised event extraction methods rely on manual annotations and features specific to each event type. |
| Approach: | They propose a framework that maps event mentions to a specific type in an event ontology . they use existing annotations to extract event types from unstructured text data . |
| Outcome: | The proposed framework can be applied to new unseen event types without manual annotations. |
KILT: a Benchmark for Knowledge Intensive Language Tasks (2021.naacl-main)
Copied to clipboard
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, Sebastian Riedel
| Challenge: | Existing models for knowledge-intensive language tasks require access to large, external knowledge sources. |
| Approach: | They propose a benchmark for knowledge-intensive language tasks (KILT) they test a shared dense vector index coupled with a seq2seq model to generate disambiguated text. |
| Outcome: | The proposed model outperforms tailor-made approaches on fact checking, open-domain question answering and dialog by generating disambiguated text. |
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2022.naacl-main)
Copied to clipboard
| Challenge: | Dynamic Adversarial Data Collection (DADC) is a time-consuming and costly approach . DADC is based on training data collected from adversarial and out-of-domain settings . |
| Approach: | They propose a dynamic data collection approach that uses generator-in-the-loop models to provide real-time suggestions that annotators can approve, modify, or reject. |
| Outcome: | The proposed model is more robust in adversarial and out-of-domain settings and harder for humans to fool. |
Behavior Analysis of NLI Models: Uncovering the Influence of Three Factors on Robustness (N18-1)
Copied to clipboard
| Challenge: | Currently, state-of-the-art models achieve impressive test set performance in the form of accuracy scores. |
| Approach: | They examine the models' robustness to semantically-valid alterations to the input data by identifying three factors and comparing their impact on three SNLI models. |
| Outcome: | The proposed models show that they can generalise to new in-domain instances while also showing that they suffer from insensitivity to small but semantically significant alterations. |
Do Large Language Models Latently Perform Multi-Hop Reasoning? (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) latently perform multi-hop reasoning with complex prompts such as "The mother of the singer of ‘Superstition’ is". |
| Approach: | They examine whether Large Language Models (LLMs) latently perform multi-hop reasoning with complex prompts such as "The mother of the singer of ‘Superstition’ is". |
| Outcome: | The proposed model can latently perform multi-hop reasoning with complex prompts such as "The mother of the singer of ‘Superstition’ is". |
Domain-matched Pre-training Tasks for Dense Retrieval (2022.findings-naacl)
Copied to clipboard
Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Scott Yih, Sonal Gupta, Yashar Mehdad
| Challenge: | Existing approaches to improve performance of pre-training tasks are needed. |
| Approach: | They propose to pre-train large bi-encoder models on a recently released set of 65 millionsynthetically generated questions and 200 million post-comment pairs from a preexisting reddit conversation dataset. |
| Outcome: | The proposed model can be pre-trained on a set of 65 millionsynthetically generated questions and 200 million post-comment pairs from a preexisting dataset of Reddit conversations. |
Undersensitivity in Neural Reading Comprehension (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models generalise well to in-distribution test sets, yet perform poorly on adversarially selected data. |
| Approach: | They propose an adversarial attack which searches among semantic variations of the question for which a model erroneously predicts the same answer, and with even higher probability. |
| Outcome: | The proposed attack reduces the vulnerability of models trained on SQuAD2.0 and NewsQA, and outperforms a conventional model by as much as 10.9% F1. |
Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets (2021.eacl-main)
Copied to clipboard
| Challenge: | a recent study examines the ability of Open-Domain Question Answering models to produce answers to factoid questions . a large number of models have been used to study the performance of open-domain QA datasets . |
| Approach: | They evaluate open-domain question answering models to see what they can generalize . they find that all models perform substantially worse on questions that cannot be memorized from train sets . |
| Outcome: | The proposed model outperforms a closed-book QA model on the open-domain datasets. |
Wronging a Right: Generating Better Errors to Improve Grammatical Error Detection (D18-1)
Copied to clipboard
| Challenge: | grammatical error correction is a labor-intensive task that requires large amounts of training data. |
| Approach: | They propose to use a human-annotated corpus of human-generated grammatical errors to generate a synthetic model. |
| Outcome: | The proposed method outperforms the current state of the art in grammatical error correction . human annotators achieve 39.39 F1 scores, suggesting the model generates mostly human-like instances . |
Avoiding the Hypothesis-Only Bias in Natural Language Inference via Ensemble Adversarial Training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural models pick up on annotation artefacts and spurious correlations, resulting in learning sentences that suffer from the same biases. |
| Approach: | They propose to tackle this problem by using adversarial training to reduce the bias in sentence representations by using an ensemble of adversaries. |
| Outcome: | The proposed approach produces more robust models outperforming previous de-biasing efforts when generalised to 12 other NLI datasets. |
AxCell: Automatic Extraction of Results from Machine Learning Papers (2020.emnlp-main)
Copied to clipboard
Marcin Kardas, Piotr Czapla, Pontus Stenetorp, Sebastian Ruder, Sebastian Riedel, Ross Taylor, Robert Stojnic
| Challenge: | AxCell is an automatic machine learning pipeline for extracting results from papers . it uses a table segmentation subtask to learn relevant structural knowledge that aids extraction. |
| Approach: | They propose to use a table segmentation subtask to learn relevant structural knowledge that aids extraction. |
| Outcome: | The proposed approach improves state of the art for results extraction and can be used for semi-automated results extraction in production. |
Don’t Read Too Much Into It: Adaptive Computation for Open-Domain Question Answering (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to Open-Domain Question Answering assume all passages are of equal importance and allocate computation to them. |
| Approach: | They propose to use adaptive computation to control the computational budget allocated for the passages to be read. |
| Outcome: | The proposed approach reduces computational cost by 4.3x over strong static and adaptive methods while retaining 95% performance of the full model. |
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning. |
| Approach: | They propose to fine tune masked language models with training examples and task descriptions to reduce prompt engineering by using null prompts. |
| Outcome: | The proposed prompts can be used to improve few-shot learning by finetuning only the bias terms while updating only 0.1% of the parameters. |
Challenges in Generalization in Open Domain Question Answering (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work on Open Domain Question Answering has shown that there is a large discrepancy in model performance between novel test questions and those that largely overlap with training questions. |
| Approach: | They introduce and annotate questions according to three categories that measure training set overlap, compositional generalization, and novel-entity generalization. |
| Outcome: | The proposed models perform better on established datasets and lower on comp-gen/novel-entity questions than on the full test set. |
Strings from the Library of Babel: Random Sampling as a Strong Baseline for Prompt Optimisation (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent prompt optimisation approaches use the generative nature of language models to produce prompts, even rivaling the performance of human-curated prompts. |
| Approach: | They propose to randomly sample tokens from the model vocabulary as "separators" they show that random separators are competitive baselines, having less than a 1% difference compared to previous self-optimisation methods. |
| Outcome: | The proposed method outperforms human-curated prompts in nine text classification tasks and eight language models and has a 40% chance of performing better than human-generated separators. |