Papers by Emanuele Bugliarello
Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work in vision-and-language pretraining has investigated supervised signals from object detection data to learn better, fine-grained multimodal representations. |
| Approach: | They propose two approaches to contextualise visual entities in a multimodal setup by using verbalised scene graphs and masked relation prediction. |
| Outcome: | The proposed models can learn better representations from weakly-supervised relations data. |
Visually Grounded Reasoning across Languages and Cultures (2021.emnlp-main)
Copied to clipboard
| Challenge: | a new protocol allows for a multilingual hierarchy of concepts and images based on native speakers . the results suggest that the current models are not robust enough to handle multilingual data . |
| Approach: | They propose a protocol to construct an ImageNet-style hierarchy representative of more languages and cultures. |
| Outcome: | The proposed protocol lets the selection of concepts and images be entirely driven by native speakers, rather than scraping them automatically. |
The Role of Syntactic Planning in Compositional Image Captioning (2021.eacl-main)
Copied to clipboard
| Challenge: | Image captioning is a core task in multimodal NLP, where the aim is to automatically describe the content of an image in natural language. |
| Approach: | They propose to use syntactic tags and tokens to improve caption generalization . they also propose to model the syntakic structure of a caption to improve generalization. |
| Outcome: | The proposed models improve generalization and performance on standard metrics while requiring syntactic and semantic knowledge of the language. |
Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs (2021.tacl-1)
Copied to clipboard
| Challenge: | Large-scale pretraining and task-specific fine-tuning are now the standard methodology for many tasks in computer vision and natural language processing. |
| Approach: | They propose to combine two types of vision and language BERTs to create a theoretical framework that can be unified under different theoretical frameworks. |
| Outcome: | The proposed models can be classified into single-stream or dual-stream encoders and are unified under a single theoretical framework. |
PAELLA: Parameter-Efficient Lightweight Language-Agnostic Captioning Model (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing models that only generate English captions are expensive due to the trend of scaling both data and model size. |
| Approach: | They propose a parameter-efficient lightweight language-agnostic captioning model that uses retrieval enhancement to train parameters between a visual model and a multilingual language model. |
| Outcome: | The proposed model outperforms models with more parameters and data and shows strong zero-shot abilities in low-resource languages. |
MuLan: A Study of Fact Mutability in Language Models (2024.naacl-short)
Copied to clipboard
| Challenge: | Pretrained and large language models encode factual knowledge, but factual information changes over time and mutates with the passage of time. |
| Approach: | They propose to use a model to evaluate the ability of English language models to anticipate time-contingency by comparing their models to a benchmark model. |
| Outcome: | The proposed model can predict the president of a country or the winner of sa championship in time, but it is difficult to update them due to their mutability. |
Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization (2023.findings-eacl)
Copied to clipboard
Aishwarya Agrawal, Ivana Kajic, Emanuele Bugliarello, Elnaz Davoodi, Anita Gergely, Phil Blunsom, Aida Nematzadeh
| Challenge: | Visual question answering (VQA) is a task of answering open-ended questions about images. |
| Approach: | They evaluate two vision-and-language (V&L) models under different settings . they find they tend to learn to solve the benchmark rather than the skills required by VQA . |
| Outcome: | The proposed models exhibit poor generalization under out-of-distribution settings. |
Enhancing Machine Translation with Dependency-Aware Self-Attention (2020.acl-main)
Copied to clipboard
| Challenge: | Currently, most neural machine translation models rely on pairs of parallel sentences, assuming syntactic information is automatically learned by an attention mechanism. |
| Approach: | They propose a parameter-free, dependency-aware self-attention mechanism that integrates syntactic knowledge into a Transformer model and propose 'a parameter free approach' they also propose - a novel mechanism that improves translation quality for long sentences and in low-resource scenarios. |
| Outcome: | The proposed approach improves translation quality on English-German and English-Turkish translation tasks and in low-resource scenarios. |
Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained vision-and-language BERTs aim to learn representations that combine information from both modalities. |
| Approach: | They propose a diagnostic method based on cross-modal input ablation to assess the extent to which pretrained models integrate cross-module information. |
| Outcome: | The proposed method evaluates the model's performance on the other modality based on inputs from one or both modality. |
Challenges and Strategies in Cross-Cultural NLP (2022.acl-long)
Copied to clipboard
Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, Anders Søgaard
| Challenge: | Various efforts have been made to accommodate linguistic diversity and serve speakers of many different languages. |
| Approach: | They propose a framework to examine cultural differences in NLP to better serve users . they argue that cultural knowledge, preferences and values can affect NLP practices . |
| Outcome: | The proposed framework examines how cultural knowledge, preferences and values can affect NLP practices. |
Measuring Progress in Fine-grained Vision-and-Language Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | X-VLM models lack "fine-grained" understanding of relationships, verbs and numbers in images . pretraining on large-scale image–text data from the Web has facilitated rapid progress on many vision-and-language tasks . |
| Approach: | They investigate models that outperform other baselines on fine-grained data . they highlight importance of novel losses and rich data sources for learning fine-grain skills . |
| Outcome: | The proposed model outperforms baseline models on four fine-grained benchmarks . the model outpersforms other baseline models and even degrades performance . |
Multilingual Multimodal Learning with Machine Translated Text (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, most vision-and-language pretraining research focuses on English tasks due to the availability of datasets. |
| Approach: | They propose a framework for machine translating English multimodal data to improve training data . they propose two metrics to prevent models from learning from low-quality translated text . |
| Outcome: | The proposed framework can be applied to any multimodal dataset and model. |
Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained machine learning models perpetuate and even amplify existing biases in data . this can result in unfair outcomes that ultimately impact user experience . |
| Approach: | They quantify bias amplification in pretraining and after fine-tuning on vision-and-language models. |
| Outcome: | The results show that pretrained models can perpetuate and even amplify biases in data without compromising performance. |
It’s Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information (2020.acl-main)
Copied to clipboard
| Challenge: | Current state-of-the-art MT systems are based on neural networks, but it is unclear whether all translation directions are equally easy (or hard) to model for NMT. |
| Approach: | They propose an asymmetric information-theoretic metric of machine translation difficulty that exploits the probabilistic nature of most neural machine translation models. |
| Outcome: | The proposed metric allows us to better evaluate the difficulty of translating text into the target language while controlling for the difficulty independent of the translation task. |