Papers by Wilker Aziz
Deep Generative Model for Joint Alignment and Word Representation (N18-1)
Copied to clipboard
| Challenge: | EmbedAlign model embeds words in their complete observed context and learns by marginalisation of latent lexical alignments. |
| Approach: | They exploit translation as a distributional context and embed words as posterior probability densities, rather than point estimates, which allows them to compare words in context using a measure of overlap between distributions. |
| Outcome: | The proposed model performs on a range of lexical semantics tasks and achieves competitive results on benchmarks including natural language inference, paraphrasing, and text similarity. |
Stop Measuring Calibration When Humans Disagree (2022.emnlp-main)
Copied to clipboard
| Challenge: | Calibration is a popular framework to evaluate whether a classifier knows when it does not know - i.e., predictive probabilities are a good indication of how likely a prediction is to be correct. |
| Approach: | They propose to measure calibration to human majority given inherent disagreements on tasks where humans inherently disagree about which class applies. |
| Outcome: | The proposed measures capture key statistical properties of human judgements including class frequency, ranking and entropy. |
Interpreting Predictive Probabilities: Model Confidence or Human Label Variation? (2024.eacl-short)
Copied to clipboard
| Challenge: | In modern NLP, neural networks are the de-facto standard to predict complex probability measures from available context. |
| Approach: | They propose to use a single predictive distribution to evaluate models with disentangled representations of uncertainty about predictions and uncertainty about human labels. |
| Outcome: | The proposed models are crucial for trustworthy and fair NLP systems, but exploiting a single distribution is limiting. |
Effective Estimation of Deep Generative Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing techniques for parameterisation of probabilistic models by deep neural networks are difficult to use in language modelling due to posterior collapse. |
| Approach: | They propose to use variational auto-encoder to estimate probabilistic models of language by deep neural networks. |
| Outcome: | The proposed model performs reasonably well given enough resources, but a favourite can be named based on convenience. |
Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve beam search quality are inadequate in many ways . a new approximation to the beam search curse has been proposed . |
| Approach: | They propose an approximation to minimum Bayes risk decoding that would solve the beam search curse. |
| Outcome: | The proposed approximation has no equivalent to the beam search curse. |
A Stochastic Decoder for Neural Machine Translation (P18-1)
Copied to clipboard
| Challenge: | Neural machine translation models do not account for local lexical and syntactic variation in parallel corpora. |
| Approach: | They propose a deep generative model of machine translation which incorporates a chain of latent variables to account for local lexical and syntactic variation in parallel corpora. |
| Outcome: | The proposed model consistently improves over strong baselines on several different language pairs. |
Predict the Next Word: <Humans exhibit uncertainty in this task and language models _____> (2024.eacl-short)
Copied to clipboard
| Challenge: | Language models (LMs) are statistical models trained to assign probability to human-generated text. |
| Approach: | They evaluate language models' ability to reproduce variability that humans exhibit in the ‘next word prediction’ task. |
| Outcome: | The language models are trained to assign probability to human-generated text . they exhibit low calibration to human uncertainty, and advise against it . |
Variational Inference and Deep Generative Models (P18-5)
Copied to clipboard
| Challenge: | Unsupervised and semi-supervised learning has been addressed scarcely in NLP . this tutorial provides an introduction to variational inference followed by an example-driven discussion of how to use variational methods for training DGMs. |
| Approach: | This tutorial provides an introduction to variational inference followed by an example-driven discussion of how to use variational methods for training DGMs. |
| Outcome: | This tutorial provides an introduction to variational inference followed by an example-driven discussion of how to use variational methods for training DGMs. |
Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies have revealed a number of pathologies of neural machine translation systems. |
| Approach: | They propose to use maximum a posteriori decoding to identify the highest-scoring translation, i.e. the mode problem, to validate the model and its training algorithm. |
| Outcome: | The proposed model reproduces the statistical data well, but the beam search strays from the statistics. |
Explanation Regularisation through the Lens of Attributions (2025.coling-main)
Copied to clipboard
| Challenge: | Explanation regularisation (ER) is a method to guide text classifiers to form their predictions relying on tokens that humans consider plausible. |
| Approach: | They introduce an auxiliary explanation loss to measure how well an input attribution technique's output agrees with human-annotated rationales. |
| Outcome: | The proposed model improves classification performance in out-of-domain (OOD) settings by relying on tokens humans consider plausible. |
Editing Factual Knowledge in Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | KnowledgeEditor can be used to edit factual knowledge stored in Language Models without the need for expensive retraining or fine-tuning. |
| Approach: | They propose a method which edits factual knowledge implicitly stored in Language Models and uses it to fix 'bugs' and 'obvious errors' they train a hyper-network with constrained optimization to modify a fact without affecting the rest of the knowledge; the hyper-netzwork is then used to predict the weight update at test time. |
| Outcome: | The proposed method can be used to edit factual knowledge without retraining or fine-tuning and can fix 'bugs' or unexpected predictions without the need for expensive re-training or meta-learning. |
Question Answering by Reasoning Across Documents with Graph Convolutional Networks (N19-1)
Copied to clipboard
| Challenge: | Recent research in reading comprehension has focused on answering questions based on individual documents or even single paragraphs. |
| Approach: | They propose a neural model which integrates and reasons relying on information spread within documents and across multiple documents. |
| Outcome: | The proposed model achieves state-of-the-art on a multi-document question answering dataset, WikiHop. |
Highly Parallel Autoregressive Entity Linking with Discriminative Correction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to EL have been shown to be effective for both Entity Disambiguation and Entity Linking, but they suffer from high computational cost due to a complex (deep) decoder and the need for training on a large amount of data. |
| Approach: | They propose a method that parallelizes autoregressive linking across all potential mentions and relies on a shallow and efficient decoder. |
| Outcome: | The proposed method outperforms state-of-the-art approaches on the English dataset AIDA-CoNLL and is >70 times faster and more accurate than the previous generative method. |
Latent Variable Model for Multi-modal Translation (P19-1)
Copied to clipboard
| Challenge: | Libovick and Helcl (2017) show improvements due to imposing a constraint on the KL term to promote models with non-negligible mutual information between inputs and latent variable and training on additional target-language image descriptions. |
| Approach: | They propose to model interaction between visual and textual features for multi-modal neural machine translation (MMT) using a latent variable model. |
| Outcome: | The proposed model improves over baselines including a multi-task learning approach and a conditional variational auto-encoder approach. |
What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability (2023.emnlp-main)
Copied to clipboard
| Challenge: | In Natural Language Generation tasks, multiple communicative goals are plausible and any goal can be put into words, or produced, in multiple ways. |
| Approach: | They characterise the extent to which human production varies lexically, syntactically, and semantically across four NLG tasks, connecting human production variability to aleatoric or data uncertainty. |
| Outcome: | The proposed model can be calibrated to human production variability using multiple samples and, when possible, multiple references. |
Interpretable Neural Predictions with Differentiable Binary Variables (P19-1)
Copied to clipboard
| Challenge: | Neural networks are bringing incredible performance gains on text classification tasks, but they also require interpretability. |
| Approach: | They propose a latent model that selects a rationale and a classifier that learns from the words in the rationale alone. |
| Outcome: | The proposed model can predict expected value of penalties without REINFORCE and can be directly optimised towards a pre-specified text selection rate. |
How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking (2020.emnlp-main)
Copied to clipboard
| Challenge: | Attribution methods assess the contribution of inputs to the model prediction. |
| Approach: | They propose a method which removes subsets of inputs and a model which is based on hidden layers to make the decision to include or disregard an input token. |
| Outcome: | The proposed method is efficient because it predicts rather than searches the inputs. |