Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing foundation models can only perform the best in one type of understanding tasks. |
| Approach: | They propose a method for training a general foundation model, X-FM, using text, image, and image-text data. |
| Outcome: | The proposed method outperforms existing foundation models on language, vision, and vision-language understanding tasks. |
Similar Papers
Vision-Language Pretraining: Current Trends and the Future (2022.acl-tutorials)
Copied to clipboard
| Challenge: | Recent vision-language models are being used for downstream tasks that require large datasets and supervised datasets. |
| Approach: | They focus on recent vision-language pretraining paradigms and their strengths and shortcomings . they compare the different family of models used for vision- language pretraining . |
| Outcome: | This paper provides the background on image–language datasets, benchmarks, and modeling innovations before the multimodal pretraining area. |
Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs (2021.tacl-1)
Copied to clipboard
| Challenge: | Large-scale pretraining and task-specific fine-tuning are now the standard methodology for many tasks in computer vision and natural language processing. |
| Approach: | They propose to combine two types of vision and language BERTs to create a theoretical framework that can be unified under different theoretical frameworks. |
| Outcome: | The proposed models can be classified into single-stream or dual-stream encoders and are unified under a single theoretical framework. |
Cross-lingual Visual Pre-training for Multimodal Machine Translation (2021.eacl-main)
Copied to clipboard
Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, Lucia Specia
| Challenge: | Pre-trained language models have been shown to improve performance in many natural language tasks. |
| Approach: | They propose to combine cross-lingual and visual pre-training to learn visually-grounded cross-linguistic representations using masked region classification and three-way parallel vision & language corpora. |
| Outcome: | The proposed models obtain state-of-the-art performance when fine-tuned for multimodal machine translation. |
LXMERT: Learning Cross-Modality Encoder Representations from Transformers (D19-1)
Copied to clipboard
| Challenge: | Existing models with better representations of visual content and language have been developed for visual-content understanding. |
| Approach: | They propose a framework to learn vision-and-language connections from Transformers models . they pre-train a large-scale Transformer model with large amounts of image-and sentence pairs . |
| Outcome: | The proposed model improves state-of-the-art on two visual-reasoning tasks by 22% . the proposed model is based on a large-scale Transformer model with three encoders . |
Medical Vision-Language Pre-Training for Brain Abnormalities (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing vision-language models lack expertise for medical applications due to the scarcity and complexity of data. |
| Approach: | They propose a pipeline to collect medical image-text aligned data for pretraining from public resources such as PubMed and build a high-performance vision-language model tailored to specific medical tasks. |
| Outcome: | The proposed model is based on a large brain image-text dataset and will be released to the public. |
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models (2024.naacl-demo)
Copied to clipboard
| Challenge: | Foundation models have demonstrated a great ability to achieve general human-level intelligence far beyond traditional approaches. |
| Approach: | They propose a toolkit to simplify the finetuning of general foundation models. |
| Outcome: | The proposed toolkit simplifies the domain- and task-aware finetuning of general foundation models with limited computing resources. |
Efficient Vision-Language pre-training via domain-specific learning for human activities (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current vision-language models owe their success to large-scale pretraining on web-collected data. |
| Approach: | They propose a domain-aligned pretraining strategy that aligns the downstream tasks to the downstream domain without additional data collection. |
| Outcome: | The proposed method outperforms existing models on large-scale vision-language training datasets while preserving generalist knowledge. |
Visually-Enhanced Phrase Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Large-scale vision-language pre-training models generate high-quality textual representations, which often outperform models that are purely text-based, such as BERT. |
| Approach: | They propose to utilize both textual and visual encoders of multi-modal pre-trained models to enhance language understanding tasks by generating an image associated with a textual prompt. |
| Outcome: | The proposed method outperforms models that are purely text-based on visual and textual understanding tasks and significantly improves the entity clustering task. |
Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies? (2022.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examines whether vision-and-language models learn syntactic dependencies . a controlled evaluation of the models is crucial for a precise and rigorous test of their knowledge . |
| Approach: | They propose a task to evaluate understanding of predicate-noun dependencies in a controlled setup. |
| Outcome: | This study compares state-of-the-art models with a case study on predicate-noun dependencies. |
NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data (2024.emnlp-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a core component of natural language processing, present in a variety of applications such as medical coding, financial news analysis, or legal documents parsing. |
| Approach: | They propose to use Large Language Models (LLMs) to create NuNER, a compact language representation model specialized in the Named Entity Recognition task. |
| Outcome: | The proposed model outperforms similar-sized foundation models in the few-shot regime and is based on a human-annotated dataset. |