Papers by Desmond Elliott
Text Rendering Strategies for Pixel Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to rendering text use a large set of almost-equivalent input patches, which may prove sub-optimal for downstream tasks due to redundancy in the input representations. |
| Approach: | They propose four approaches to rendering text in a PIXEL model using character bigrams and patch frequency biases. |
| Outcome: | The proposed models perform better on sentence-level tasks without compromising performance on token-level or multilingual tasks. |
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (2025.acl-short)
Copied to clipboard
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, Alberto Testoni
| Challenge: | Existing evaluations of NLP models with LLMs are based on human judgments . however, there are concerns about their validity and reproducibility in proprietary models . |
| Approach: | They evaluate 11 current LLMs for their ability to replicate annotations. they show substantial variance across models and datasets. |
| Outcome: | The proposed model can replicate human annotations on 20 NLP datasets and show substantial variance across models and datasets. |
How Do Multilingual Language Models Remember Facts? (2025.findings-acl)
Copied to clipboard
| Challenge: | Prior research has focused on English monolingual models, but how these mechanisms generalize to non-English languages remains unexplored. |
| Approach: | They analyze three multilingual LLMs to find out how they can generalize recall mechanisms . they find that subject enrichment is language-independent, object extraction is language dependent . |
| Outcome: | The proposed model performs better in multilingual contexts than in English models . the model is more efficient in multi-lingual context, but it is more complex in multilinguistic models compared to English models. |
The Role of Data Curation in Image Captioning (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing image captioning models treat all samples equally, neglecting mismatched data . Several other techniques have relied on curriculum learning strategies to adapt learning to the difficulty of the task. |
| Approach: | They propose to actively curate difficult samples in datasets using curriculum learning strategies to improve captioning models. |
| Outcome: | The proposed methods outperform existing models on the Flickr30K and COCO datasets. |
Measuring the Diversity of Automatic Image Descriptions (C18-1)
Copied to clipboard
| Challenge: | a lack of diversity in automatic image description systems is a general problem in natural language generation . authors use established metrics to evaluate system performance on the head of the vocabulary . automatic image descriptions are difficult because of the unbounded range of variation in natural languages . |
| Approach: | They propose to frame automatic image description as a word recall task to quantify the production of generic sentences as 'undiversity' they propose to use established metrics to evaluate the diversity of the output . |
| Outcome: | The proposed metrics evaluate the diversity of sentences generated by state-of-the-art systems on a MS COCO dataset. |
MultiFin: A Dataset for Multilingual Financial NLP (2023.findings-eacl)
Copied to clipboard
| Challenge: | Multilingual models are needed to process financial text, which is produced across the world and requires a large dataset. |
| Approach: | They propose to annotate a publicly available financial dataset using a hierarchical label structure and an annotation schema based on a real-world application. |
| Outcome: | The proposed model can be used in high-resource languages, but there is room for improvement in low-resourced languages. |
Adversarial Removal of Demographic Attributes Revisited (D19-1)
Copied to clipboard
| Challenge: | Several approaches have been proposed to learn classifiers that are invariant (unbiased with respect) to protected attributes. |
| Approach: | They propose to use a diagnostic classifier trained on a held-out subsample to find protected attributes for mention detection at above-chance levels. |
| Outcome: | The proposed classifier generalizes poorly to new in-domain and new domains, suggesting it relies on correlations specific to their particular data sample. |
Revisiting Transformer-based Models for Long Document Classification (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent literature in text classification is biased towards short text sequences . multi-page multi-paragraph documents cannot be efficiently encoded by vanilla transformers based on short text. |
| Approach: | They compare different Transformer-based Long Document Classification approaches to mitigate the computational overhead of vanilla transformers to encode much longer text. |
| Outcome: | The proposed models can process longer text and provide practical advice for long document classification tasks. |
Retrieval-augmented Image Captioning (2023.eacl-main)
Copied to clipboard
| Challenge: | Experimental results show that image captioning can be effectively formulated from this new perspective. |
| Approach: | They propose a pretrained visual and language decoders for image captioning that generate sentences from the input image and a set of captions retrieved from a datastore. |
| Outcome: | The proposed model generates sentences given the input image and retrieved captions, while the decoder attends to the multimodal encoder representations. |
Multilingual Pretraining for Pixel Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | PIXEL-M4 model pretrains on four visually and linguistically diverse languages . previous work on pixel-based language models focused on monolingual pretraining on English data . |
| Approach: | They propose a pixel-based language model that is pretrained on four visually diverse languages. |
| Outcome: | The proposed model outperforms an English-only counterpart on non-Latin scripts on semantic and syntactic tasks. |
Visually Grounded Reasoning across Languages and Cultures (2021.emnlp-main)
Copied to clipboard
| Challenge: | a new protocol allows for a multilingual hierarchy of concepts and images based on native speakers . the results suggest that the current models are not robust enough to handle multilingual data . |
| Approach: | They propose a protocol to construct an ImageNet-style hierarchy representative of more languages and cultures. |
| Outcome: | The proposed protocol lets the selection of concepts and images be entirely driven by native speakers, rather than scraping them automatically. |
Adversarial Evaluation of Multimodal Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Existing evidence that visual context helps multimodal machine translation systems is unconvincing due to inconsistencies between text-similarity metrics and human judgements. |
| Approach: | They propose an adversarial evaluation method to examine the utility of image data in multimodal machine translation. |
| Outcome: | The proposed method shows that only one out of three publicly available systems is sensitive to this perturbation of the data. |
The Role of Syntactic Planning in Compositional Image Captioning (2021.eacl-main)
Copied to clipboard
| Challenge: | Image captioning is a core task in multimodal NLP, where the aim is to automatically describe the content of an image in natural language. |
| Approach: | They propose to use syntactic tags and tokens to improve caption generalization . they also propose to model the syntakic structure of a caption to improve generalization. |
| Outcome: | The proposed models improve generalization and performance on standard metrics while requiring syntactic and semantic knowledge of the language. |
LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Prompting (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing captioning models are mostly available on English or are specialised for other high-resource languages. |
| Approach: | They propose an image-blind few-shot multilingual captioning model that prompts a language model with retrieved captions. |
| Outcome: | The proposed model is competitive with fully-supervised multilingual captioning models without requiring any supervised training on captioning data. |
Cross-lingual Visual Verb Sense Disambiguation (N19-1)
Copied to clipboard
| Challenge: | Recent work has shown that visual context improves cross-lingual sense disambiguation for nouns. |
| Approach: | They extend their work to the task of cross-lingual verb sense disambiguation by using a dataset annotated with English, German, and Spanish verbs. |
| Outcome: | The proposed model improves the results of a text-only machine translation system when used for a multimodal translation task. |
Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs (2021.tacl-1)
Copied to clipboard
| Challenge: | Large-scale pretraining and task-specific fine-tuning are now the standard methodology for many tasks in computer vision and natural language processing. |
| Approach: | They propose to combine two types of vision and language BERTs to create a theoretical framework that can be unified under different theoretical frameworks. |
| Outcome: | The proposed models can be classified into single-stream or dual-stream encoders and are unified under a single theoretical framework. |
Textual Supervision for Visually Grounded Spoken Language Understanding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | a new approach to spoken language understanding extracts semantic information directly from speech without relying on transcriptions. |
| Approach: | They propose to use textual supervision to train visually-grounded models of spoken language understanding without relying on transcriptions. |
| Outcome: | The proposed model improves when enough text is available, the study shows . compared with pipeline-based models, the pipeline approach performs better when enough data is available . |
Can Community Notes Replace Professional Fact-Checkers? (2025.acl-short)
Copied to clipboard
| Challenge: | Fact-checkers are crucial in combating misinformation on social media . however, community moderation is often employed in parallel due to the scale of misleading content shared online. |
| Approach: | They use language models to annotate Twitter/X community notes with attributes such as topic, cited sources, and whether they refute misinformation claims. |
| Outcome: | The results show that community notes cite fact-checking sources up to five times more than previously reported. |
PAELLA: Parameter-Efficient Lightweight Language-Agnostic Captioning Model (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing models that only generate English captions are expensive due to the trend of scaling both data and model size. |
| Approach: | They propose a parameter-efficient lightweight language-agnostic captioning model that uses retrieval enhancement to train parameters between a visual model and a multilingual language model. |
| Outcome: | The proposed model outperforms models with more parameters and data and shows strong zero-shot abilities in low-resource languages. |
PHD: Pixel-Based Language Modeling of Historical Documents (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent years have seen a boom in efforts to digitise historical documents in numerous languages and sources, leading to a transformation in the way historians work. |
| Approach: | They propose a method for generating synthetic scans to resemble real historical documents by pre-training a model to reconstruct masked patches instead of predicting token distributions. |
| Outcome: | The proposed model can reconstruct masked patches and understand language well. |
Long Story Short: Disentangling Compositionality and Long-Caption Understanding in Contrastive VLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for vision-language models treat compositionality and long-caption understanding in isolation. |
| Approach: | They analyze when compositional reasoning and long-caption understanding transfer across tasks and when this relationship fails. |
| Outcome: | The proposed model can generalize on poorly grounded captions and with strong visual grounding, while architectural choices can limit compositional learning. |
Fine-Grained Grounding for Multimodal Speech Recognition (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models rely on global visual features that represent the entire image, but localizing the relevant regions of the image will make it possible to recover a larger set of words, such as adjectives and verbs. |
| Approach: | They propose a multimodal automatic speech recognition system that uses visual information from different parts of the image to ground the speech in the visual context. |
| Outcome: | The proposed model improves over approaches that use global visual features and localizes the correct proposals. |
mDAPT: Multilingual Domain Adaptive Pretraining in a Single Model (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing domain-specific multilingual pretraining data is difficult to obtain due to regulations, legislation, or simply a lack of language- and domain- specific text. |
| Approach: | They propose to continue pretraining a language model on domain-specific unlabelled text . this allows for better modelling of text for downstream tasks within the domain . |
| Outcome: | The proposed approach outperforms the general multilingual model and performs close to its monolingual counterpart. |
Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Era (2025.findings-acl)
Copied to clipboard
| Challenge: | danoneata, et al., 2021): human learning and conceptual representation is grounded in sensorimotor experience. |
| Approach: | They evaluate image encoders and language-only models to learn which attributes are salient to the models. |
| Outcome: | The proposed models outperform language-only models on attributes predicting extended denser McRae norms and newer Binder datasets. |
The Sensitivity of Language Models and Humans to Winograd Schema Perturbations (2020.acl-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models are driving recent improvements in perfromance on the Winograd Schema Challenge . a diagnostic dataset shows that these models are sensitive to linguistic perturbations that minimally affect human understanding . |
| Approach: | They propose to use a dataset to test pre-trained language models for the Winograd Schema Challenge . they show that these models are sensitive to linguistic perturbations that minimally affect human understanding . |
| Outcome: | The proposed models are sensitive to linguistic perturbations that minimally affect human understanding. |
Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained vision-and-language BERTs aim to learn representations that combine information from both modalities. |
| Approach: | They propose a diagnostic method based on cross-modal input ablation to assess the extent to which pretrained models integrate cross-module information. |
| Outcome: | The proposed method evaluates the model's performance on the other modality based on inputs from one or both modality. |
FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culture (2024.emnlp-main)
Copied to clipboard
Wenyan Li, Crystina Zhang, Jiaang Li, Qiwei Peng, Raphael Tang, Li Zhou, Weijia Zhang, Guimin Hu, Yifei Yuan, Anders Søgaard, Daniel Hershcovich, Desmond Elliott
| Challenge: | FoodieQA is a manually curated, fine-grained image-text dataset capturing the intricate features of food cultures across various regions in China. |
| Approach: | They evaluate vision–language Models and large language models on unseen food images and corresponding questions. |
| Outcome: | The proposed dataset evaluates vision–language Models and large language models on unseen food images and corresponding questions. |
Understanding the Effect of Textual Adversaries in Multimodal Machine Translation (D19-64)
Copied to clipboard
| Challenge: | Existing studies show that multimodal machine translation systems are better than text-only systems at translating phrases that have a direct correspondence in the image. |
| Approach: | They conduct experiments with both visual and textual adversaries to understand the role of textual inputs in multimodal machine translation. |
| Outcome: | The proposed model can recover masked tokens in the source sentences . the proposed model is based on a model with a visual modality . |
Understanding Retrieval Robustness for Retrieval-augmented Image Captioning (2024.acl-long)
Copied to clipboard
| Challenge: | Recent retrieval-augmented models for image captioning are not perfect in practice. |
| Approach: | They propose to train a retrieval-augmented captioning model SmallCap by sampling retrieved captions from more diverse sets. |
| Outcome: | The proposed model is sensitive to tokens that appear in the majority of retrieved captions . the proposed model improves both in-domain and cross-domain performance . |
CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning (2020.acl-main)
Copied to clipboard
Alessandro Suglia, Ioannis Konstas, Andrea Vanzo, Emanuele Bastianelli, Desmond Elliott, Stella Frank, Oliver Lemon
| Challenge: | Approaches to Grounded Language Learning focus on a single task-based final performance measure which may not depend on desirable properties of the learned hidden representations. |
| Approach: | They propose an evaluation framework for Grounded Language Learning with Attributes based on three sub-tasks: 1) Goal-oriented evaluation; 2) Object attribute prediction evaluation; and 3) Zero-shot evaluation. |
| Outcome: | The proposed framework evaluates the quality of learned representations with respect to attribute grounding. |
Sequential Compositional Generalization in Multimodal Models (2024.naacl-long)
Copied to clipboard
| Challenge: | a growing number of multimodal models have a limited capacity for generalization . however, prior studies into compositionality have focused on visual grounding and downstream tasks like image captioning. |
| Approach: | They examine compositional generalization using egocentric kitchen activity videos . they find bi-modal and tri-modal models exhibit a clear edge over their text-only counterparts . |
| Outcome: | The proposed model outperforms text-only models in a multimodal setting. |
Multilingual Multimodal Learning with Machine Translated Text (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, most vision-and-language pretraining research focuses on English tasks due to the availability of datasets. |
| Approach: | They propose a framework for machine translating English multimodal data to improve training data . they propose two metrics to prevent models from learning from low-quality translated text . |
| Outcome: | The proposed framework can be applied to any multimodal dataset and model. |
On Forgetting to Cite Older Papers: An Analysis of the ACL Anthology (2020.acl-main)
Copied to clipboard
| Challenge: | a growing number of published papers are citing older work, but the rate of citations is stable . a recent paper cited work from recent years, whereas papers published 15 or more years ago are cited at a stable rate. |
| Approach: | They analyze citations in papers published at selected ACL venues between 2010 and 2019 . they find that recent papers are cited significantly more often in recent years . |
| Outcome: | The authors analyze citations in journals and conferences between 2010 and 2019 . they find that recent papers cite more recent work, but papers published 15 or more years ago are cited at a stable rate. |
Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained machine learning models perpetuate and even amplify existing biases in data . this can result in unfair outcomes that ultimately impact user experience . |
| Approach: | They quantify bias amplification in pretraining and after fine-tuning on vision-and-language models. |
| Outcome: | The results show that pretrained models can perpetuate and even amplify biases in data without compromising performance. |