Papers by James Henderson
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. |
| Approach: | They propose a framework that can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks. |
| Outcome: | The proposed framework improves performance on the well-known GLUE benchmark while adding only 0.29% parameters per task. |
The Unstoppable Rise of Computational Linguistics in Deep Learning (2020.acl-main)
Copied to clipboard
| Challenge: | a quarter century ago, linguists assumed that language knowledge needed to be innate . but vector-space representations and machine learning algorithms are much more powerful than was thought . |
| Approach: | They trace the history of neural networks applied to natural language understanding tasks . they argue that Transformer is not a sequence model but an induced-structure model . |
| Outcome: | The proposed model is not a sequence model but an induced-structure model, the authors argue . they argue that the nature of language has had a profound impact on progress in machine learning . |
HyperMixer: An MLP-based Low Cost Alternative to Transformers (2023.acl-long)
Copied to clipboard
Florian Mai, Arnaud Pannatier, Fabio Fehr, Haolin Chen, Francois Marelli, Francois Fleuret, James Henderson
| Challenge: | Existing MLP-based architectures that combine multiple features are expensive and require a lot of training data. |
| Approach: | They propose a simple MLP-based model which allows token mixing by dynamically applying hypernetworks to each feature independently. |
| Outcome: | The proposed model performs better than Transformers and lowers costs in terms of processing time, training data, and hyperparameter tuning. |
Fast-and-Frugal Text-Graph Transformers are Effective Link Predictors (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods that encode textual and structural information for inductive link prediction are frugal and fast at training and inference time. |
| Approach: | They propose a Transformer-based framework that unifies textual and structural information for inductive link prediction in text-attributed knowledge graphs by encoding ego-graphs (1-hop neighbourhoods). |
| Outcome: | The proposed framework can achieve superior performance on three popular datasets and reduce the reliance on resource-intensive encoders. |
TESS: Text-to-Text Self-Conditioned Simplex Diffusion (2024.eacl-long)
Copied to clipboard
Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew Peters, Arman Cohan
| Challenge: | Existing models for diffusion generation are expensive and discrete, resulting in a large number of diffusion steps to generate text. |
| Approach: | They propose a text diffusion model that is fully non-autoregressive and employs a new form of self-conditioning and applies the diffusion process on the logit simplex space rather than the learned embedding space. |
| Outcome: | The proposed model outperforms state-of-the-art non-autoregressive models, requires fewer diffusion steps with minimal drop in performance, and is competitive with pretrained autoregressive sequence-to-sequence models. |
Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement (2021.tacl-1)
Copied to clipboard
| Challenge: | RNGTr is a non-recursive Graph-to-Graph Transformer for iterative refinement of graphs . it can improve the accuracy of initial parsers on 13 languages . |
| Approach: | They propose a recursive non-autoregressive Graph-to-Graph Transformer architecture for iterative refinement of arbitrary graphs and apply it to syntactic dependency parsing. |
| Outcome: | The proposed architecture can improve state-of-the-art on 13 languages and the German CoNLL2009 corpus. |
Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing methods to learn mappings in the embedding space of text autoencoders are limited to a single-vector embeddment, which limits how much information can be retained. |
| Approach: | They propose a method to learn mappings in the embedding space of an autoencoder by extending it to Bag-of-Vectors Autoencodeurs (BoV-AEs) this allows to encode and reconstruct much longer texts than standard autoencodings . |
| Outcome: | The proposed method performs better than a standard autoencoder on unsupervised sentiment transfer. |
Weakly-Supervised Concept-based Adversarial Learning for Cross-lingual Word Embeddings (D19-1)
Copied to clipboard
| Challenge: | Current unsupervised adversarial methods for mapping monolingual word embeddings require high quality parallel data, such as a dictionary or sentence-aligned corpus. |
| Approach: | They propose a weakly-supervised adversarial training method which improves performance of previous unsupervised adversaries for most languages. |
| Outcome: | The proposed method improves the performance of previous unsupervised adversarial methods for most languages and especially for typologically distant language pairs. |
What Do Compressed Multilingual Machine Translation Models Forget? (2022.findings-emnlp)
Copied to clipboard
Alireza Mohammadshahi, Vassilina Nikoulina, Alexandre Berard, Caroline Brun, James Henderson, Laurent Besacier
| Challenge: | Recent studies show that pre-trained models achieve state-of-the-art results in NLP tasks but their size makes it more challenging to apply them in resource-constrained environments. |
| Approach: | They assess the impact of compression methods on multilingual Neural Machine Translation models for various language groups, gender, and semantic biases. |
| Outcome: | The proposed compression methods improve models on different benchmarks for language groups, gender, and semantic biases. |
SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languages (2022.emnlp-main)
Copied to clipboard
Alireza Mohammadshahi, Vassilina Nikoulina, Alexandre Berard, Caroline Brun, James Henderson, Laurent Besacier
| Challenge: | Existing models for multilingual machine translation use scaling up the number of parameters to overcome the curse of multilinguality. |
| Approach: | They propose a multilingual machine translation model that shares information between similar languages and scales up the number of parameters to overcome the curse of multilinguality. |
| Outcome: | The proposed model outperforms previous models on low-resource benchmarks while improving inference latency and memory usage. |
Prompt-free and Efficient Few-shot Learning with Language Models (2022.acl-long)
Copied to clipboard
Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, Majid Yazdani
| Challenge: | Existing methods for few-shot fine-tuning of pretrained language models require carefully engineered prompts and verbalizers to convert inputs into a cloze-format that the PLM can score. |
| Approach: | They propose a method for few-shot fine-tuning of pretrained language models that uses task-specific adapters instead of manually engineered prompts and verbalizers. |
| Outcome: | The proposed method outperforms existing state-of-the-art methods on a wide range of few shot NLP tasks. |
Evaluating Unsupervised Argument Aligners via Generation of Conclusions of Structured Scientific Abstracts (2024.eacl-short)
Copied to clipboard
| Challenge: | Scientific abstracts provide a concise summary of research findings. |
| Approach: | They evaluate unsupervised approaches for extracting scientific arguments as aligned premise-conclusion pairs . they find mutual information outperforms other measures on this task . |
| Outcome: | The proposed methods outperform language models on the task of extracting scientific arguments from abstracts. |
Plug and Play Autoencoders for Conditional Text Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Text autoencoders are used for conditional generation tasks such as style transfer. |
| Approach: | They propose a plug-and-play method where any pretrained autoencoder can be used and only requires learning a mapping within the embedding space. |
| Outcome: | The proposed method performs better than or comparable to strong baselines while being up to four times faster. |
RAGferee: Building Contextual Reward Models for Retrieval-Augmented Generation (2025.emnlp-main)
Copied to clipboard
Andrei Catalin Coman, Ionut Teodor Sorodoc, Leonardo F. R. Ribeiro, Bill Byrne, James Henderson, Adrià de Gispert
| Challenge: | Existing Reward Models (RMs) struggle in Retrieval Augmented Generation settings. |
| Approach: | They propose a method that repurposes question-answering datasets into preference pairs that prioritise groundedness over stylistic features. |
| Outcome: | The proposed method surpasses existing RMs trained on larger general corpora with an absolute improvement of +15.5%. |
RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question (2023.findings-acl)
Copied to clipboard
Alireza Mohammadshahi, Thomas Scialom, Majid Yazdani, Pouya Yanki, Angela Fan, James Henderson, Marzieh Saeidi
| Challenge: | Existing metrics for evaluating the quality of automatically generated questions are expensive and penalise valid questions that may not have high lexical or semantic similarity to the reference questions. |
| Approach: | They propose a question-answering and span scorer metric based on the answerability of the candidate question given the context. |
| Outcome: | The proposed metric has higher correlation with human judgment without relying on the reference question. |
Learning to Abstract with Nonparametric Variational Information Bottleneck (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Learning textual embeddings is costly as they are tokenization specific and require different models to be trained for each level of abstraction. |
| Approach: | They propose a character-level representation model which learns different levels of abstraction in different layers of the same model. |
| Outcome: | The proposed model is more robust to adversarial perturbations and can learn to compress representations to different levels of abstraction. |
Multilingual Extraction and Categorization of Lexical Collocations with Graph-aware Transformers (2022.starsem-1)
Copied to clipboard
| Challenge: | lexical collocations exhibit varying degrees of frozenness due to their varying degree of frozenncy. |
| Approach: | They propose a sequence tagging BERT-based model enhanced with a graph-aware transformer architecture and evaluate the task of collocation recognition in context. |
| Outcome: | The proposed model encoding syntactic dependencies is useful, and provides insights on differences in collocation typification in English, Spanish and French. |
Multi-Adversarial Learning for Cross-Lingual Word Embeddings (2021.naacl-main)
Copied to clipboard
| Challenge: | Generative adversarial networks (GANs) have succeeded in inducing cross-lingual word embeddings without supervision, but their performance for distant languages is still not satisfactory. |
| Approach: | They propose a multi-adversarial method that induces the seed cross-lingual dictionary through multiple mappings, each induced to fit the mapping for one subspace. |
| Outcome: | The proposed method improves performance on bilingual lexicon induction and cross-lingual document classification on unsupervised bilingual linguistic induction. |
Document-Level Neural Machine Translation with Hierarchical Attention Networks (D18-1)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) can be improved by including document-level contextual information. |
| Approach: | They propose a hierarchical attention model that captures document-level contextual information and conditioning on the NMT model’s own hidden states. |
| Outcome: | The proposed model improves the BLEU score over a strong NMT baseline with the state-of-the-art in context-aware methods and that both the encoder and decoder benefit from context in complementary ways. |
End-to-End Bias Mitigation by Modelling Biases in Corpora (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that strong natural language understanding models are prone to relying on unwanted dataset biases without learning the underlying task. |
| Approach: | They propose two learning strategies to train neural models that are more robust to dataset biases and transfer better to out-of-domain datasets. |
| Outcome: | The proposed methods improve robustness in all settings and transfer better to out-of-domain datasets. |
Graph-to-Graph Transformer for Transition-based Dependency Parsing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for conditioning on graphs and predicting graphs are weak, but they are effective for transition-based dependency parsing. |
| Approach: | They propose a Transformer architecture for conditioning on and predicting arbitrary graphs. |
| Outcome: | The proposed architecture outperforms the state-of-the-art in transition-based dependency parsing on English Penn Treebank and 13 languages of Universal Dependencies Treebanks. |
Graph Refinement for Coreference Resolution (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing models for coreference resolution are based on independent mention pair-wise decisions. |
| Approach: | They propose a model that learns coreference at the document-level and takes global decisions. |
| Outcome: | The proposed model improves over baselines, reinforcing the hypothesis that document-level information improves conference resolution. |