Papers by Phil Blunsom
Augmenting Multi-Turn Text-to-SQL Datasets with Self-Play (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Numerous architectures and pretraining methods have been proposed for context-dependent text-to-SQL, but the size of the datasets used has been limited due to the high cost of annotating multi-turn dialogue and SQL pairs. |
| Approach: | They propose to augment training datasets using self-play which leverages contextual information to synthesize new interactions to adapt the model to new databases. |
| Outcome: | The proposed model improves accuracy on SParC and CoSQL, two widely used cross-domain text-to-SQl datasets. |
WikiCREM: A Large Unsupervised Corpus for Coreference Resolution (D19-1)
Copied to clipboard
| Challenge: | Large-scale training sets for pronoun resolution are scarce, since manually labelling data is costly. |
| Approach: | They propose a language-model-based approach to solve pronoun disambiguation problems using a WikiCREM dataset. |
| Outcome: | The proposed model outperforms state-of-the-art approaches on 6 out of 7 datasets. |
Scalable Syntax-Aware Language Models Using Knowledge Distillation (P19-1)
Copied to clipboard
| Challenge: | Prior work has shown that syntactic neural language models learn from small amounts of training data more effectively than sequential models. |
| Approach: | They propose a knowledge distillation technique that transfers knowledge from a syntactic language model trained on a small corpus to an LSTM language model and enables it to develop a more structurally sensitive representation of the larger training data. |
| Outcome: | The proposed method improves on baseline syntactic evaluations on LSTMs with a higher level of accuracy than previous methods. |
Relational Memory-Augmented Language Models (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing language models rely on word correlation and are difficult to interpret . existing models often lack explicit representations for such information . |
| Approach: | They propose a memory-augmented approach to condition autoregressive language models on knowledge graphs. |
| Outcome: | The proposed model improves perplexity and bits per character in an autoregressive language model . it is complementary to token-based memory and enables causal interventions . |
A Systematic Investigation of Commonsense Knowledge in Large Language Models (2022.emnlp-main)
Copied to clipboard
Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoffmann, Cyprien de Masson d’Autume, Phil Blunsom, Aida Nematzadeh
| Challenge: | Recent large language models (LMs) have shown impressive performance on many NLP tasks under the zero-shot and few-shot setup. |
| Approach: | They conduct a systematic and rigorous zero-shot and few-shot commonsense evaluation of large pre-trained language models to better understand their ability to capture commonsensical knowledge. |
| Outcome: | The proposed model can exploit surface cues and annotation artefacts without task-specific supervision and is insufficient to achieve human-level commonsense performance. |
Better Document-Level Machine Translation with Bayes’ Rule (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing document translation models are based on autoregressive language models, but they are not able to be learned from monolingual documents. |
| Approach: | They propose to use Bayes' rule to create document translation models that can be learned from only parallel sentences and monolingual documents. |
| Outcome: | The proposed model outperforms existing document translation approaches and is based on a novel left-to-right beam-search algorithm. |
A Generative Framework for Simultaneous Machine Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches use a fixed number of source words to translate or learn dynamic policies for the number of sources by reinforcement learning. |
| Approach: | They propose a generative framework that uses a latent variable to model read or translate actions at every time step and integrates out to consider all possible translation policies. |
| Outcome: | The proposed framework achieves the best BLEU scores on benchmark datasets. |
Learning Robust and Multilingual Speech Representations (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Unsupervised speech representation learning has shown success at finding representations that correlate with phonetic structures and improve downstream speech recognition performance. |
| Approach: | They evaluate unsupervised speech representation learning representations by looking at their robustness to domain shifts and their ability to improve recognition performance in many languages. |
| Outcome: | The proposed representations improve the recognition performance in 25 phonetically diverse languages and are robust to domain shifts. |
Counterfactual Data Augmentation for Neural Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Neural machine translation models often rely on large-scale parallel corpora for training, exhibiting degraded performance on low-resource languages. |
| Approach: | They propose a method that interprets language models and phrasal alignment causally and generates augmented parallel translation corpora by sampling new source phrases from a masked language model. |
| Outcome: | The proposed method improves translation, backtranslation and translation robustness on IWSLT’15 English Vietnamese, WMT’17 English - German, and WMT'18 English – Turkish. |
Syntactic Structure Distillation Pretraining for Bidirectional Encoders (2020.tacl-1)
Copied to clipboard
Adhiguna Kuncoro, Lingpeng Kong, Daniel Fried, Dani Yogatama, Laura Rimell, Chris Dyer, Phil Blunsom
| Challenge: | Textual representation learners trained on large amounts of data have been successful on downstream tasks. |
| Approach: | They propose a knowledge distillation strategy for injecting syntactic biases into BERT pretraining by distilling the approximate marginal distribution over words in context from the syntaktic LM. |
| Outcome: | The proposed method reduces relative error by 2–21% on a diverse set of structured prediction tasks. |
Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale (2022.tacl-1)
Copied to clipboard
| Challenge: | a novel class of Transformer language models that combine expressive power, scalability, and strong performance of Transformers and recursive syntactic compositions. |
| Approach: | They introduce Transformer Grammars, a class of Transformer language models that combine expressive power and recursive syntactic compositions. |
| Outcome: | The proposed model outperforms strong baselines on sentence-level language modeling perplexity and syntax-sensitive language evaluation metrics. |
Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization (2023.findings-eacl)
Copied to clipboard
Aishwarya Agrawal, Ivana Kajic, Emanuele Bugliarello, Elnaz Davoodi, Anita Gergely, Phil Blunsom, Aida Nematzadeh
| Challenge: | Visual question answering (VQA) is a task of answering open-ended questions about images. |
| Approach: | They evaluate two vision-and-language (V&L) models under different settings . they find they tend to learn to solve the benchmark rather than the skills required by VQA . |
| Outcome: | The proposed models exhibit poor generalization under out-of-distribution settings. |
Pretraining the Noisy Channel Model for Task-Oriented Dialogue (2021.tacl-1)
Copied to clipboard
| Challenge: | Current research on task-oriented dialogue models suffers from the explaining-away effect, manifested in models that favor short and generic responses. |
| Approach: | They propose to factorize the dialogue task into two models, the distribution of the context given the response, and the prior for the response itself, using Bayes' theorem. |
| Outcome: | The proposed model mitigates the explaining-away effect and allows the principled incorporation of large pretrained models for the response prior. |
Learning to Discover, Ground and Use Words with Segmental Neural Language Models (P19-1)
Copied to clipboard
| Challenge: | Existing models of word learning do not account for the long-range dependencies manifest in language and that are easily captured by recurrent neural networks. |
| Approach: | They propose a segmental neural language model that unifies word discovery, learning how words fit together to form sentences, and by conditioning the model on visual context, how words’ meanings ground in representations of nonlinguistic modalities. |
| Outcome: | The proposed model learns predictive distributions better than character LSTM models, discovers words competitively with nonparametric Bayesian word segmentation models, and improves on both. |
LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better (P18-1)
Copied to clipboard
| Challenge: | a recent study found that language models fail to learn long-range syntax sensitive dependencies. |
| Approach: | They propose to use a subject-verb agreement diagnostic to determine whether language models can learn long-range syntax sensitive dependencies. |
| Outcome: | The proposed model outperforms left-corner and bottom-up variants in learning non-local dependencies. |
Improving Reward Models with Synthetic Critiques (2025.findings-naacl)
Copied to clipboard
| Challenge: | a recent study shows that reward models overfit on superficial features, hindering generalization performance . prevailing approach to training preference-based reward models presents several challenges . |
| Approach: | They propose a method that uses synthetic natural language critiques to provide additional feedback to large language models. |
| Outcome: | The proposed approach improves performance and data efficiency of RMs initialized from different pretrained models, reducing the reliance on costly human annotations. |
Revisiting the Compositional Generalization Abilities of Neural Sequence Models (2022.acl-short)
Copied to clipboard
| Challenge: | Existing studies have suggested that standard seq-to-seq models lack the ability to generalize compositionally. |
| Approach: | They propose to use one-shot primitive generalization as introduced by the popular SCAN benchmark to modify the training distribution in simple and intuitive ways to achieve near-perfect generalization performance. |
| Outcome: | The proposed model achieves near-perfect generalization performance despite a lack of training data . |
Neural Syntactic Generative Models with Exact Marginalization (N18-1)
Copied to clipboard
| Challenge: | Recent models have added structure to recurrent neural networks at the cost of giving up exact inference, or using soft structure instead of latent variables. |
| Approach: | They propose a syntactic generative model with exact marginalization that supports dependency parsing and language modeling. |
| Outcome: | The proposed models achieve state-of-the-art for supervised dependency parsing and language modeling. |
Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions (2023.acl-long)
Copied to clipboard
| Challenge: | Recent studies have found that Transformers struggle to model several formal languages when compared to recurrent models. |
| Approach: | They conduct an extensive empirical study on Boolean functions to demonstrate that Transformers are relatively more biased towards functions of low sensitivity . they also show that Transformer's generalize near perfectly even in the presence of noisy labels whereas recurrent models overfit and achieve poor generalization accuracy. |
| Outcome: | The results show that Transformers generalize near perfectly even in noisy Boolean functions whereas recurrent models overfit and achieve poor generalization accuracy. |
Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations (2020.acl-main)
Copied to clipboard
| Challenge: | a promising research direction consists of designing neural models capable of generating natural language explanations for their predictions. |
| Approach: | They propose a framework for sanity checking models against inconsistent explanations . they apply the framework to a state-of-the-art neural natural language inference model . |
| Outcome: | The proposed framework can generate inconsistent explanations on a state-of-the-art model . it also addresses the problem of adversarial attacks with full target sequences . |
Learning to Segment Actions from Observation and Narration (2020.acl-main)
Copied to clipboard
| Challenge: | a generative segmental model of task structure is applied to video training . despite its simplicity, the model performs well in unsupervised and weakly-supervised settings . |
| Approach: | They propose a generative segmental model of task structure guided by narration to video segmentation . |
| Outcome: | The proposed model performs well in unsupervised and weakly-supervised training . it allows us to vary the sources of supervision used in training despite its simplicity . |
On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research (2023.acl-long)
Copied to clipboard
Made Nindyatama Nityasya, Haryo Wibowo, Alham Fikri Aji, Genta Winata, Radityo Eko Prasojo, Phil Blunsom, Adhiguna Kuncoro
| Challenge: | Despite rapid recent progress, current research practices conflate different sources of model improvement without conducting proper ablation studies and principled comparisons . authors conclude with recommendations for how to encourage and incentivize this line of work . |
| Approach: | They critique current research practices in the field of language model pre-training . they examine the success of language models pre-trained on large amounts of data . |
| Outcome: | The proposed models can achieve competitive or better performance than BERT under comparable conditions. |