Papers by Chris Dyer
Diverse Pretrained Context Encodings Improve Document Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models for sentence-level sequence-to-sequence translations do not use extra-sentential information. |
| Approach: | They propose a sentence-level sequence-to-sequence transformer with multiple pre-trained context signals. |
| Outcome: | The proposed model outperforms existing models on Chinese-English and English-German tasks. |
An Empirical Investigation of Global and Local Normalization for Recurrent Neural Sequence Models Using a Continuous Relaxation to Beam Search (N19-1)
Copied to clipboard
| Challenge: | Neural encoder-decoder models have been successful at a variety of NLP tasks, including machine translation, parsing, and dialog generation. |
| Approach: | They propose a method for search-aware training via a continuous relaxation of beam search to enable global normalization. |
| Outcome: | The proposed approach is able to train globally normalized recurrent sequence models through simple backpropagation. |
Better Chinese Sentence Segmentation with Reinforcement Learning (2021.findings-acl)
Copied to clipboard
| Challenge: | Chinese-English machine translation systems use ambiguous sentence boundaries, but English and Chinese use different orthographic conventions to designate sentence boundaries. |
| Approach: | They propose a segmentation policy that splits Chinese texts into segments that can be independently translated to maximise translation quality. |
| Outcome: | The proposed method improves the baseline BLEU score on the Chinese-English news translation task by +0.3 BLUE overall and the score on input segments that contain more than 60 words by +3 BL EU. |
Syntactic Scaffolds for Semantic Structures (D18-1)
Copied to clipboard
| Challenge: | Syntactic scaffolds avoid expensive syntactical processing at runtime . many systems have used syntastic dependency or phrase-based parsers as preprocessing for semantic analysis. |
| Approach: | They propose a multitask learning approach that uses a syntactic treebank to integrate syntaktic information into semantic tasks. |
| Outcome: | The proposed method improves on PropBank semantics, frame semantics and coreference resolution tasks. |
Compound Probabilistic Context-Free Grammars for Grammar Induction (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to grammar induction have resorted to manually-engineered features and auxiliary objectives to induce the desired structures. |
| Approach: | They propose a formalization of the grammar induction problem that models sentences as being generated by a compound probabilistic context free grammar. |
| Outcome: | Experiments on English and Chinese show that the proposed approach is more efficient than other methods. |
Scalable Syntax-Aware Language Models Using Knowledge Distillation (P19-1)
Copied to clipboard
| Challenge: | Prior work has shown that syntactic neural language models learn from small amounts of training data more effectively than sequential models. |
| Approach: | They propose a knowledge distillation technique that transfers knowledge from a syntactic language model trained on a small corpus to an LSTM language model and enables it to develop a more structurally sensitive representation of the larger training data. |
| Outcome: | The proposed method improves on baseline syntactic evaluations on LSTMs with a higher level of accuracy than previous methods. |
Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation (D19-1)
Copied to clipboard
Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Krishnamurthy Dvijotham, Pushmeet Kohli
| Challenge: | Recent work has exposed the vulnerabilities of neural NLP models, e.g. with small, semantically invariant input alterations. |
| Approach: | They propose to model text classification under synonym replacements or character flip perturbations and then use a formal model verification method to verify its robustness. |
| Outcome: | The proposed models show little difference in terms of nominal accuracy, but have much improved verified accuracy under perturbations and come with an efficiently computable formal guarantee on worst case adversaries. |
Better Document-Level Machine Translation with Bayes’ Rule (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing document translation models are based on autoregressive language models, but they are not able to be learned from monolingual documents. |
| Approach: | They propose to use Bayes' rule to create document translation models that can be learned from only parallel sentences and monolingual documents. |
| Outcome: | The proposed model outperforms existing document translation approaches and is based on a novel left-to-right beam-search algorithm. |
Game-theoretic Vocabulary Selection via the Shapley Value and Banzhaf Index (2021.naacl-main)
Copied to clipboard
| Challenge: | Using the full vocabulary results in less explainable and memory intensive models. |
| Approach: | They propose a vocabulary selection method that views words as members of a team trying to maximize the model's performance. |
| Outcome: | The proposed method outperforms baseline models on multiple tasks and datasets. |
Learning Robust and Multilingual Speech Representations (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Unsupervised speech representation learning has shown success at finding representations that correlate with phonetic structures and improve downstream speech recognition performance. |
| Approach: | They evaluate unsupervised speech representation learning representations by looking at their robustness to domain shifts and their ability to improve recognition performance in many languages. |
| Outcome: | The proposed representations improve the recognition performance in 25 phonetically diverse languages and are robust to domain shifts. |
Syntactic Structure Distillation Pretraining for Bidirectional Encoders (2020.tacl-1)
Copied to clipboard
Adhiguna Kuncoro, Lingpeng Kong, Daniel Fried, Dani Yogatama, Laura Rimell, Chris Dyer, Phil Blunsom
| Challenge: | Textual representation learners trained on large amounts of data have been successful on downstream tasks. |
| Approach: | They propose a knowledge distillation strategy for injecting syntactic biases into BERT pretraining by distilling the approximate marginal distribution over words in context from the syntaktic LM. |
| Outcome: | The proposed method reduces relative error by 2–21% on a diverse set of structured prediction tasks. |
Text Genre and Training Data Size in Human-like Parsing (D19-1)
Copied to clipboard
| Challenge: | Using domain-specific training, NLP systems work better, but only when the training examples come from the same textual genre. |
| Approach: | They relate the states of a neural phrase-structure parser to electrophysiological measures from human participants. |
| Outcome: | The proposed model is well-matched to the training data from human participants, but only when the training examples come from the same genre. |
Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale (2022.tacl-1)
Copied to clipboard
| Challenge: | a novel class of Transformer language models that combine expressive power, scalability, and strong performance of Transformers and recursive syntactic compositions. |
| Approach: | They introduce Transformer Grammars, a class of Transformer language models that combine expressive power and recursive syntactic compositions. |
| Outcome: | The proposed model outperforms strong baselines on sentence-level language modeling perplexity and syntax-sensitive language evaluation metrics. |
Using Morphological Knowledge in Open-Vocabulary Neural Language Models (N18-1)
Copied to clipboard
| Challenge: | Existing models that generate words from a fixed vocabulary are linguistically nave . authors present an open-vocabulary language model that incorporates morphological knowledge into a neural framework . |
| Approach: | They propose a model that incorporates morphological knowledge into a neural model by generating words as a sequence of characters, generating full word forms and combining them with a hand-written morphology analyzer. |
| Outcome: | The proposed model outperforms character-based models on Finnish, Turkish, and Russian on three languages. |
Learning to Discover, Ground and Use Words with Segmental Neural Language Models (P19-1)
Copied to clipboard
| Challenge: | Existing models of word learning do not account for the long-range dependencies manifest in language and that are easily captured by recurrent neural networks. |
| Approach: | They propose a segmental neural language model that unifies word discovery, learning how words fit together to form sentences, and by conditioning the model on visual context, how words’ meanings ground in representations of nonlinguistic modalities. |
| Outcome: | The proposed model learns predictive distributions better than character LSTM models, discovers words competitively with nonparametric Bayesian word segmentation models, and improves on both. |
Finding syntax in human encephalography with beam search (P18-1)
Copied to clipboard
| Challenge: | RNNGs are generative models of (tree , string ) pairs that evaluate derivational choices . a non-syntactic neural language model yields no reliable effects . |
| Approach: | They propose to combine a probabilistic generative grammar with a parsing procedure that uses it to manage syntactic derivations as it advances from one word to the next. |
| Outcome: | The proposed model derives two amplitude effects when used against human encephalography data. |
LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better (P18-1)
Copied to clipboard
| Challenge: | a recent study found that language models fail to learn long-range syntax sensitive dependencies. |
| Approach: | They propose to use a subject-verb agreement diagnostic to determine whether language models can learn long-range syntax sensitive dependencies. |
| Outcome: | The proposed model outperforms left-corner and bottom-up variants in learning non-local dependencies. |
A Probabilistic Generative Model for Typographical Analysis of Early Modern Printing (2020.acl-main)
Copied to clipboard
| Challenge: | Scholars often need to go beyond textual analysis for establishing provenance of historical documents. |
| Approach: | They propose a deep and interpretable probabilistic generative model to analyze glyph shapes in printed Early Modern documents by generating a latent vector responsible for inking variations, jitter, noise and other unforeseen phenomena. |
| Outcome: | The proposed model outperforms interpretable clustering baselines and overly-flexible deep generative models on the task of completely unsupervised discovery of typefaces in mixed-fonts documents. |
Learning to Segment Actions from Observation and Narration (2020.acl-main)
Copied to clipboard
| Challenge: | a generative segmental model of task structure is applied to video training . despite its simplicity, the model performs well in unsupervised and weakly-supervised settings . |
| Approach: | They propose a generative segmental model of task structure guided by narration to video segmentation . |
| Outcome: | The proposed model performs well in unsupervised and weakly-supervised training . it allows us to vary the sources of supervision used in training despite its simplicity . |
Unsupervised Recurrent Neural Network Grammars (N19-1)
Copied to clipboard
| Challenge: | RNNGs model syntax and structure by incrementally generating a syntax tree and sentence in a top-down, left-to-right order. |
| Approach: | They explore unsupervised learning of recurrent neural network grammars for language modeling and grammar induction. |
| Outcome: | The proposed model outperforms standard sequential language models and improves parsing performance. |