Multi-stage Pre-training over Simplified Multimodal Pre-training Models (2021.acl-long)
Copied to clipboard
| Challenge: | Existing multimodal pre-training models require large amounts of training data and have huge model sizes, making them impossible to apply in low-resource situations. |
| Approach: | They propose a multi-stage pre-training method which uses information at different granularities from word, phrase to sentence in both texts and images to pre-train a model in stages. |
| Outcome: | The proposed method outperforms the original model in Image-Text Retrieval task and outperformed the original LXMERT model in downstream tasks. |
Similar Papers
MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators (2022.acl-long)
Copied to clipboard
| Challenge: | Prompting has been shown to be a promising approach for applying pre-trained language models to perform downstream tasks. |
| Approach: | They propose a method that divides the translation process into three stages using pre-trained language models. |
| Outcome: | The proposed method significantly improves translation performance of pre-trained language models on three translation tasks. |
Complicate Then Simplify: A Novel Way to Explore Pre-trained Models for Text Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing frameworks for text classification employing pre-trained models are constrained by the difficulty of the task. |
| Approach: | They propose a framework which implements a two-stage training strategy to fully exploit the knowledge in pre-trained models. |
| Outcome: | The proposed framework outperforms state-of-the-art classification models on six text classification corpora. |
Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition (2024.lrec-main)
Copied to clipboard
Yash Jain, David M. Chan, Pranav Dheram, Aparna Khare, Olabanji Shonibare, Venkatesh Ravichandran, Shalini Ghosh
| Challenge: | Existing methods for pre-training for automatic speech recognition (ASR) focus on single-stage pre-train followed by fine-tuning on downstream task. |
| Approach: | They propose a multi-modal pre-training method that combines unsupervised pre-training with translation-based supervised mid-training. |
| Outcome: | The proposed method improves WERs by 38.45% over baselines on both Librispeech and SUPERB. |
MGDoc: Pre-training with Multi-granular Hierarchy for Document Image Understanding (2022.emnlp-main)
Copied to clipboard
Zilong Wang, Jiuxiang Gu, Chris Tensmeyer, Nikolaos Barmpalios, Ani Nenkova, Tong Sun, Jingbo Shang, Vlad Morariu
| Challenge: | Existing methods learn features from word-level or region-level but fail to consider both simultaneously. |
| Approach: | They propose a multi-modal multi-granular pre-training framework that encodes page-level, region-level and word-level information at the same time. |
| Outcome: | The proposed model learns features from word-level and region-level but fails to consider both simultaneously. |
Multi-Stage Pre-training for Low-Resource Domain Adaptation (2020.emnlp-main)
Copied to clipboard
Rong Zhang, Revanth Gangi Reddy, Md Arafat Sultan, Vittorio Castelli, Anthony Ferritto, Radu Florian, Efsun Sarioglu Kayi, Salim Roukos, Avi Sil, Todd Ward
| Challenge: | Existing approaches to transfer learning target data to in-domain text . prior work has adapted pre-trained LMs to specific domains . |
| Approach: | They extend the vocabulary of a pretrained language model with domain-specific terms to create synthetic tasks that help it transfer to downstream tasks. |
| Outcome: | The proposed approaches show significant performance gains on extractive reading comprehension, document ranking and duplicate question detection tasks. |
Different Strokes for Different Folks: Investigating Appropriate Further Pre-training Approaches for Diverse Dialogue Tasks (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models can be fine-tuned on domain-specific unlabeled data . however, most further pre-training works just keep running the conventional pre- training task . |
| Approach: | They propose to add a further pre-training phase to the model to improve downstream tasks . they propose to use a domain-adaptive pre-tuning phase to fine-tune the models on unlabeled data . |
| Outcome: | The proposed method improves multiple task-oriented dialogue downstream tasks. |
Advances in Pre-Training Distributed Word Representations (L18-1)
Copied to clipboard
| Challenge: | Pre-trained word representations are a building block of many Natural Language Processing and Machine Learning applications. |
| Approach: | They propose to combine known tricks and a set of publicly available pre-trained word vector representations to train high-quality representations. |
| Outcome: | The proposed models outperform the current state of the art on a number of tasks while maintaining a high training speed to scale to massive amount of data. |
An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-training large language models is a standard practice in the natural language processing community. |
| Approach: | They propose to use elastic weight consolidation to mitigate catastrophic forgetting when pre-trained large language models are evaluated on generic benchmarks. |
| Outcome: | The proposed model achieves state-of-the-art on out-of domain tasks with minimal pre-training . elastic weight consolidation provides best overall scores yielding only a 0.33% drop in performance across seven generic tasks while remaining competitive in bio-medical tasks. |
Cross-lingual Visual Pre-training for Multimodal Machine Translation (2021.eacl-main)
Copied to clipboard
Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, Lucia Specia
| Challenge: | Pre-trained language models have been shown to improve performance in many natural language tasks. |
| Approach: | They propose to combine cross-lingual and visual pre-training to learn visually-grounded cross-linguistic representations using masked region classification and three-way parallel vision & language corpora. |
| Outcome: | The proposed models obtain state-of-the-art performance when fine-tuned for multimodal machine translation. |
Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing strategies to teach pre-trained models to generate simple texts are inadequate. |
| Approach: | They propose a continued pre-training strategy to teach pre-trained models to generate simple texts by randomly masking text spans in ordinary texts. |
| Outcome: | The proposed strategy improves on lexical simplification, sentence simplification and document-level simplification tasks over existing models. |