The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language-and-vision models have impressive performance in downstream tasks, but it remains unclear whether this improves understanding of image-text interaction. |
| Approach: | They propose to use BLA to evaluate multimodal models on basic linguistic constructions that even preschool children can typically master. |
| Outcome: | The proposed model improves basic language skills in a zero-shot learning setting. |
Similar Papers
Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies? (2022.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examines whether vision-and-language models learn syntactic dependencies . a controlled evaluation of the models is crucial for a precise and rigorous test of their knowledge . |
| Approach: | They propose a task to evaluate understanding of predicate-noun dependencies in a controlled setup. |
| Outcome: | This study compares state-of-the-art models with a case study on predicate-noun dependencies. |
Be Different to Be Better! A Benchmark to Leverage the Complementarity of Language and Vision (2020.findings-emnlp)
Copied to clipboard
| Challenge: | BD2BB is a language and vision benchmark that requires multimodal models combine complementary information from the two modalities. |
| Approach: | They propose a novel language and vision benchmark that requires multimodal models combine complementary information from both modalities. |
| Outcome: | The proposed model is easy for humans, but poor for humans . it compares state-of-the-art models against human speakers to show that it performs well. |
Vision-Language Pretraining: Current Trends and the Future (2022.acl-tutorials)
Copied to clipboard
| Challenge: | Recent vision-language models are being used for downstream tasks that require large datasets and supervised datasets. |
| Approach: | They focus on recent vision-language pretraining paradigms and their strengths and shortcomings . they compare the different family of models used for vision- language pretraining . |
| Outcome: | This paper provides the background on image–language datasets, benchmarks, and modeling innovations before the multimodal pretraining area. |
How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input? (2022.coling-1)
Copied to clipboard
| Challenge: | Current language models have been criticised for learning language from text alone without connection between words and their meaning. |
| Approach: | They propose to train models on more sources than text to provide the lacking connection between words and their meanings. |
| Outcome: | The proposed model adaptation methods perform differently for different models and unimodal model counterparts perform on par with the VL models regardless of adaptation. |
Word Representation Learning in Multimodal Pre-Trained Transformers: An Intrinsic Evaluation (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing models for linguistic representations of words are based on information extracted from large text corpora, and the sensory-motor experiences humans have with the world play an important role in determining word meaning. |
| Approach: | They propose to use contextualized word representations to learn semantic representations of words that align with human semantic intuitions. |
| Outcome: | The proposed models are shown to be more efficient on concrete word pairs than on abstract ones. |
Cross-lingual Visual Pre-training for Multimodal Machine Translation (2021.eacl-main)
Copied to clipboard
Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, Lucia Specia
| Challenge: | Pre-trained language models have been shown to improve performance in many natural language tasks. |
| Approach: | They propose to combine cross-lingual and visual pre-training to learn visually-grounded cross-linguistic representations using masked region classification and three-way parallel vision & language corpora. |
| Outcome: | The proposed models obtain state-of-the-art performance when fine-tuned for multimodal machine translation. |
Probing Image-Language Transformers for Verb Understanding (2021.findings-acl)
Copied to clipboard
| Challenge: | Multimodal image-language transformers have achieved impressive results on a variety of tasks that rely on fine-tuning. |
| Approach: | They collect a dataset of image-sentence pairs consisting of 421 verbs . they evaluate pretrained image-language transformers and find they fail more in situations that require verb understanding compared to other parts of speech. |
| Outcome: | The proposed model trains on a manually-annotated and smaller dataset does better on the task. |
Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs (2021.tacl-1)
Copied to clipboard
| Challenge: | Large-scale pretraining and task-specific fine-tuning are now the standard methodology for many tasks in computer vision and natural language processing. |
| Approach: | They propose to combine two types of vision and language BERTs to create a theoretical framework that can be unified under different theoretical frameworks. |
| Outcome: | The proposed models can be classified into single-stream or dual-stream encoders and are unified under a single theoretical framework. |
Are Multimodal Large Language Models Pragmatically Competent Listeners in Simple Reference Resolution Tasks? (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models are unable to resolve references to abstract visual stimuli, such as color patches and color grids, but their pragmatic capabilities are still a challenge for state-of-the-art MLLMs. |
| Approach: | They investigate whether multimodal large language models are able to resolve references to abstract visual stimuli, such as color patches and color grids, in a well-known reference resolution paradigm. |
| Outcome: | The proposed model can resolve references to abstract visual stimuli in dyadic reference games. |
Life after BERT: What do Other Muppets Understand about Language? (2022.acl-long)
Copied to clipboard
| Challenge: | Existing pre-trained transformer analysis studies focus on one or two model families at a time, overlooking the variability of the architecture and pre-training objectives. |
| Approach: | They utilize oLMpics bench- mark and psycholinguistic probing datasets for a diverse set of 29 models including T5, BART, and ALBERT. |
| Outcome: | The proposed model fails to resolve compositional questions in a zero-shot fashion, suggesting that pre-training objectives are not predictive of a model’s linguistic capabilities. |