Papers by Douwe Kiela

Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021.emnlp-main)

Challenge:	Dynamic adversarial data collection (DADC) can be used to build models that are robust across a wide range of test inputs.
Approach:	They propose to run Dynamic adversarial data collection over many rounds to maximize its training-time benefits.
Outcome:	The proposed model makes 26% fewer errors on the premise paragraphs compared to models trained on non-adversarial examples.

Copied to clipboard

Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, Douwe Kiela

Semantic Text Processing

Challenge:	masked language models (MLMs) pre-train to model higher-order word co-occurrence statistics . authors suggest that such models have learned to represent syntactic structures prevalent in classical NLP pipelines . purely distributional information largely explains the success of pre-training, authors say .
Approach:	They propose to pre-train masked language models on sentences with random shuffled word order and show they still achieve high accuracy after fine-tuning on many downstream tasks.
Outcome:	The proposed model performs well according to parametric syntactic probes . the authors argue that the model is not all that different from earlier distributional models .

Dynamic Meta-Embeddings for Improved Sentence Representations (D18-1)

Copied to clipboard

Douwe Kiela, Changhan Wang, Kyunghyun Cho

Representation Learning

Challenge:	A sprawling literature has emerged about what word embeddings are most useful for which tasks . word embed-ding is a technique that can be used to learn word-level meaning representations for a variety of tasks.
Approach:	They propose a method for supervised learning of embedding ensembles that leads to state-of-the-art performance on a variety of tasks.
Outcome:	The proposed method leads to state-of-the-art performance on a variety of tasks.

Learning Visually Grounded Sentence Representations (N18-1)

Copied to clipboard

Douwe Kiela, Alexis Conneau, Allan Jabri, Maximilian Nickel

Multimodality

Challenge:	Unsupervised sentence representation models suffer from the grounding problem because of lack of association between symbols and external information.
Approach:	They train a sentence encoder to predict image features of a caption and use them as sentence representations.
Outcome:	The proposed model improves on word embeddings and word representations on standard benchmarks.

Nearest Neighbor Normalization Improves Multimodal Retrieval (2024.emnlp-main)

Copied to clipboard

Neil Chowdhury, Franklin Wang, Sumedh Shenoy, Douwe Kiela, Sarah Schwettmann, Tristan Thrush

Multimodality

Challenge:	Recent training-free methods suggest that accuracy can be improved without fine-tuning.
Approach:	They propose a method for correcting errors in trained contrastive image-text retrieval models with no additional training, called Nearest Neighbor Normalization.
Outcome:	The proposed method improves retrieval metrics for all contrastive models and datasets and does not require training on the reference database.

Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation (2021.emnlp-main)

Copied to clipboard

Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, Douwe Kiela

Dynabench: Rethinking Benchmarking in NLP (2021.naacl-main)

Challenge:	a new approach to generate adversarial data is needed to improve question answering models . crowdworkers can fool a model only 8.8% of the time, compared to 17.6% for a trained model without synthetic data.
Approach:	They develop a pipeline that generates questions and then filters or labels them to improve quality.
Outcome:	The proposed approach improves state-of-the-art on a human-written adversarial dataset by 3.7F1 and improves model generalisation on nine of the twelve MRQA datasets.

Copied to clipboard

Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, Adina Williams

Challenge:	Dynabench is an open-source platform for dynamic dataset creation and model benchmarking.
Approach:	They propose an open-source platform for dynamic dataset creation and model benchmarking.
Outcome:	The proposed platform can be used to create models that fail on simple challenges and falter in real-world scenarios.

Adversarial NLI: A New Benchmark for Natural Language Understanding (2020.acl-main)

Copied to clipboard

Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, Douwe Kiela

Inferring Concept Hierarchies from Text Corpora via Hyperbolic Embeddings (P19-1)

Challenge:	a new large-scale NLI benchmark dataset is presented to test models on a variety of popular NLIs.
Approach:	They propose a large-scale NLI benchmark dataset that is iteratively compared with a human-and-model-in-the-loop procedure.
Outcome:	The proposed method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate.

Copied to clipboard

Matthew Le, Stephen Roller, Laetitia Papaxanthos, Douwe Kiela, Maximilian Nickel

Representation Learning

Challenge:	Using hyperbolic embeddings, we can infer concept hierarchies from distributional contexts while also being able to predict missing “is-a”-relationships and correct wrong extractions.
Approach:	They propose a method combining hyperbolic embeddings and Hearst patterns to set appropriate constraints for inferring “is-a” relationships from large text corpora and improve taxonomic consistency.
Outcome:	The proposed method achieves state-of-the-art performance on a variety of hypernymy benchmarks.

What’s Hidden in a One-layer Randomly Weighted Transformer? (2021.emnlp-main)

Copied to clipboard

Sheng Shen, Zhewei Yao, Douwe Kiela, Kurt Keutzer, Michael Mahoney

Finding Generalizable Evidence by Learning to Convince Q&A Models (D19-1)

Challenge:	Existing subnetworks of one-layer randomly weighted neural networks can achieve impressive performance without changing initializations.
Approach:	They find subnetworks within one-layer randomly weighted neural networks that can achieve impressive performance without ever modifying the initializations.
Outcome:	The proposed subnetworks match 98%/92% of the performance of a trained Transformersmall/base on IWSLT14/WMT14.

Copied to clipboard

Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, Kyunghyun Cho

Question Answering

Challenge:	a system that finds the strongest supporting evidence for a given answer is proposed . a study using passage-based question-answering (QA) shows that agents select evidence that generalizes .
Approach:	They propose a system that finds the strongest supporting evidence for a given answer . they use passage-based question-answering (QA) as a testbed to train evidence agents .
Outcome:	The proposed system improves QA in a robust manner by using agent-selected evidence.

What makes a good conversation? How controllable attributes affect human judgments (N19-1)

Copied to clipboard

Abigail See, Stephen Roller, Douwe Kiela, Jason Weston

Retrieval Augmentation Reduces Hallucination in Conversation (2021.findings-emnlp)

Challenge:	Existing work on dialogue models for conversational quality is incompletely understanding the relationship between quality and individual attributes.
Approach:	They propose to use conditional training and weighted decoding to control four attributes for chit-chat dialogue: repetition, specificity, response-relatedness and question-asking.
Outcome:	The proposed methods improve human quality judgments by controlling combinations of these variables.

Copied to clipboard

Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, Jason Weston

Cross-Modal Retrieval Augmentation for Multi-Modal Classification (2021.findings-emnlp)

Challenge:	State-of-the-art dialogue models suffer from factual incorrectness and hallucination of knowledge.
Approach:	They propose to use neural-retrieval-in-the-loop architectures to optimize knowledge-grounded dialogue by retrieving, ranking, and encoder-decoders.
Outcome:	The proposed architectures exhibit open-domain conversational capabilities and generalize effectively to scenarios not within the training data.

Copied to clipboard

Shir Gur, Natalia Neverova, Chris Stauffer, Ser-Nam Lim, Douwe Kiela, Austin Reiter

Information Retrieval

Challenge:	Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing.
Approach:	They propose a retrieval-augmented multi-modal transformer architecture for embedding images and captions in the same space.
Outcome:	The proposed approach improves visual question answering over strong baselines and hot-swapping indices.

Learning to Speak and Act in a Fantasy Text Adventure Game (D19-1)

Copied to clipboard

Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, Jason Weston

I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling (2021.acl-long)

Challenge:	Existing studies on grounded dialogue use only statistical regularities of text data, without explicit understanding of the world that the text describes.
Approach:	They propose a large-scale crowdsourced text adventure game as a research platform for studying grounded dialogue.
Outcome:	The proposed game allows agents to perceive, emote, and act whilst conducting dialogue with other agents.

Copied to clipboard

Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, Jason Weston

Gradient-based Adversarial Attacks against Text Transformers (2021.emnlp-main)

Challenge:	Recent advances on neural approaches to natural language processing have triggered a resurgent interest on building intelligent open-domain chatbots.
Approach:	They propose a dialoguE COntradiction DEtection task and a conversational dataset . they show that their best contradiction detection model correlates well with human judgments .
Outcome:	The proposed model is more robust and generalizes well on analysis and out-of-distribution dialogues than standard (unstructured) Transformer models that explicitly hinge on utterance structures are more robust, the study shows .

Copied to clipboard

Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, Douwe Kiela

DynaSent: A Dynamic Benchmark for Sentiment Analysis (2021.acl-long)

Challenge:	Existing methods for obtaining adversarial examples are difficult with text data.
Approach:	They propose a gradient-based adversarial attack against transformer models that searches for a distribution of adversarials parameterized by a continuous-valued matrix.
Outcome:	The proposed attack outperforms existing methods on a variety of natural language tasks with matching imperceptibility.

Copied to clipboard

Christopher Potts, Zhengxuan Wu, Atticus Geiger, Douwe Kiela

Sentiment Analysis

Challenge:	Sentiment analysis is an early success story for NLP, in both a technical and an industrial sense.
Approach:	They propose to combine naturally occurring sentences with sentences created using the open-source Dynabench Platform, which facilities human-and-model-in-the-loop dataset creation.
Outcome:	The proposed model is more coherent than comparable models and motivates training models from scratch over successive fine-tuning.

LMUNIT: Fine-grained Evaluation with Natural Language Unit Tests (2025.findings-emnlp)

Copied to clipboard

Jon Saad-Falcon, Rajan Pathe Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, Shikib Mehri

Personalizing Dialogue Agents: I have a dog, do you have pets too? (P18-1)

Challenge:	Using natural language unit tests, language models are costly and noisy, and automated metrics provide only coarse, difficult-to-interpret signals.
Approach:	They propose a paradigm that decomposes response quality into explicit, testable criteria and a unified scoring model, LMUnit, which combines multi-objective training across preferences, direct ratings, and natural language rationales.
Outcome:	The proposed paradigm significantly improves inter-annotator agreement and enables more effective LLM development workflows.

Copied to clipboard

Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, Jason Weston

Dialogue Systems & Conversational Agents

Challenge:	chit-chat models lack specificity, do not display a consistent personality and are often not very captivating.
Approach:	They propose to train chit-chat models to condition on profile information and profile information about the interlocutors.
Outcome:	The proposed model can predict profile information about the interlocutors based on the data . the proposed model is able to generate meaningful responses in a chit-chat setting .

Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2022.naacl-main)

Copied to clipboard

Max Bartolo, Tristan Thrush, Sebastian Riedel, Pontus Stenetorp, Robin Jia, Douwe Kiela

Reservoir Transformers (2021.acl-long)

Challenge:	Dynamic Adversarial Data Collection (DADC) is a time-consuming and costly approach . DADC is based on training data collected from adversarial and out-of-domain settings .
Approach:	They propose a dynamic data collection approach that uses generator-in-the-loop models to provide real-time suggestions that annotators can approve, modify, or reject.
Outcome:	The proposed model is more robust in adversarial and out-of-domain settings and harder for humans to fool.

Copied to clipboard

Sheng Shen, Alexei Baevski, Ari Morcos, Kurt Keutzer, Michael Auli, Douwe Kiela

Seeded self-play for language learning (D19-64)

Challenge:	Using random initialization, we show that some transformers obtain impressive performance even when some of the layers are frozen.
Approach:	They propose to freeze transformer layers and use them to improve performance . they find that the transformers obtain impressive performance even when some of the layers are randomly initialized and never updated.
Outcome:	The proposed model improves on translation and language modelling tasks even when some layers are frozen.

Copied to clipboard

Abhinav Gupta, Ryan Lowe, Jakob Foerster, Douwe Kiela, Joelle Pineau

Emergent Linguistic Phenomena in Multi-Agent Communication Games (D19-1)

Challenge:	Current methods for learning human language are too data inefficient to learn it in this way.
Approach:	They propose to train a meta-learning agent in simulation to interact with populations of pre-trained agents, each with their own distinct communication protocol.
Outcome:	The proposed algorithm minimizes the number of on-policy interactions while learning human language while minimizing the number on-political interactions.

Copied to clipboard

Laura Harding Graesser, Kyunghyun Cho, Douwe Kiela

Semantic Text Processing

Challenge:	a recent study examines the behavior of linguistic agents in a community-level setting . a linguistic continuum emerges where neighboring languages are more mutually intelligible than farther removed ones .
Approach:	They propose a multi-agent communication framework for studying linguistic phenomena at the community level.
Outcome:	The proposed framework can reproduce complex linguistic behavior observed in natural language . it can be used to study interactions between perceptually-enabled agents .

Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection (2021.acl-long)

Copied to clipboard

Bertie Vidgen, Tristan Thrush, Zeerak Waseem, Douwe Kiela

Responsible & Trustworthy NLP

Challenge:	Detecting online hate speech has proven difficult and concerns raised about performance, robustness, generalisability and fairness of stateof-the-art models.
Approach:	They propose a human-and-model-in-the-loop process for dynamically generating datasets and training better performing hate detection models.
Outcome:	The proposed model improves on a dataset of 40,000 hateful entries . the model is harder for annotators to trick and better on HateCheck .

Countering Language Drift via Visual Grounding (D19-1)

Copied to clipboard

Jason Lee, Kyunghyun Cho, Douwe Kiela

Visual Data in NLP

Challenge:	Emergent multi-agent communication protocols are different from natural language . a long-standing goal of artificial intelligence research is to develop agents that can cooperate with other agents .
Approach:	They propose to use syntactic and semantic constraints to improve communication . they propose to combine these constraints with auxiliary training constraints to reduce language drift .
Outcome:	a new study shows that pre-trained agents retain English syntax while learning to convey intended meaning . the proposed training constraints can be used to mitigate language drift .

Unsupervised Question Decomposition for Question Answering (2020.emnlp-main)

Copied to clipboard

Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, Douwe Kiela

Low-Resource NLP

Challenge:	Existing QA systems struggle to answer complex questions because information is scattered in different places.
Approach:	They propose an unsupervised algorithm that decomposes hard questions into simpler sub-questions . they propose an algorithm that can be used to generate a final answer from millions of questions .
Outcome:	The proposed algorithm decomposes hard questions into simpler sub-questions that existing QA systems can answer.

Investigating Multi-source Active Learning for Natural Language Inference (2023.eacl-main)

Copied to clipboard

Ard Snijders, Douwe Kiela, Katerina Margatina

Responsible & Trustworthy NLP

Challenge:	Recent studies often assume that training and test data are drawn from the same distribution.
Approach:	They propose to apply active learning to unlabelled data pools to test for learning and generalisation.
Outcome:	The proposed strategies outperform random selection and outperformed hard-to-learn data on the task of natural language inference.

I am a Strange Dataset: Metalinguistic Tests for Language Models (2024.acl-long)

Copied to clipboard

Tristan Thrush, Jared Moore, Miguel Monares, Christopher Potts, Douwe Kiela

Semantic Text Processing

Challenge:	Existing datasets for metalinguistic self-reference are limited by the number of subtasks.
Approach:	They propose a dataset that aims to address metalinguistic self-reference in large language models.
Outcome:	The proposed dataset is hand-crafted by experts and validated by non-expert annotators.

Hearst Patterns Revisited: Automatic Hypernym Detection from Large Text Corpora (P18-2)

Copied to clipboard

Stephen Roller, Douwe Kiela, Maximilian Nickel

Responsible & Trustworthy NLP

Challenge:	a well-known problem of Hearst-like patterns is their extreme sparsity.
Approach:	They propose to use pattern-based and distributional methods to perform unsupervised hypernym detection.
Outcome:	The proposed method outperforms distributional methods on hypernymy tasks.

Perturbation Augmentation for Fairer NLP (2022.emnlp-main)

Copied to clipboard

Rebecca Qian, Candace Ross, Jude Fernandes, Eric Michael Smith, Douwe Kiela, Adina Williams

SentEval: An Evaluation Toolkit for Universal Sentence Representations (L18-1)

Challenge:	Unwanted and often harmful social biases are becoming more salient in NLP research.
Approach:	They propose to train a neural perturbation model that rewrites demographic references in text to make them more fair.
Outcome:	The proposed model outperforms heuristic alternatives on a large dataset of human annotated text perturbations.

Copied to clipboard

Alexis Conneau, Douwe Kiela

Representation Learning

Challenge:	a toolkit for evaluating the quality of universal sentence representations is available for download and preprocessing . word embeddings are not trained to perform well on one specific task, but their value lies in their transferability . evaluation of general-purpose word and sentence embeddables has been problematic .
Approach:	They propose a toolkit to evaluate the quality of universal sentence representations.
Outcome:	The proposed toolkit includes scripts to download and preprocess datasets and an easy interface to evaluate sentence encoders.

On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study (2021.acl-long)

Copied to clipboard

Divyansh Kaushik, Douwe Kiela, Zachary C. Lipton, Wen-tau Yih

Anchor Points: Benchmarking Models with Much Fewer Examples (2024.eacl-long)

Challenge:	Existing studies have shown that adversarial data collection (ADC) models perform better on other adversarially collected data but are liable under plausible domain shifts.
Approach:	They conduct a large-scale controlled study on question answering by assigning workers at random to compose questions either adversarially (with a model in the loop) or in the standard fashion (without a modeling).
Outcome:	The proposed model performs better on other adversarial datasets but worse on diverse collection of out-of-domain evaluation sets.

Copied to clipboard

Rajan Vivek, Kawin Ethayarajh, Diyi Yang, Douwe Kiela