Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for training language-vision models only consider monolingual learning, especially English. |
| Approach: | They propose to extend an English language-vision model into a multilingual and code-mixed model by using knowledge distillation techniques. |
| Outcome: | The proposed model outperforms existing models on multilingual and code-mixed VQA datasets on eleven languages. |
Similar Papers
A Unified Framework for Multilingual and Code-Mixed Visual Question Answering (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing techniques for visual question answering focus on English questions, but many applications require a multilingual module. |
| Approach: | They propose a deep learning framework for multilingual and code- mixed visual question answering . they create Hindi and Code-mixed VQA datasets by exploiting linguistic properties of these languages . |
| Outcome: | The proposed model is capable of predicting answers from the questions in Hindi, English or Code- mixed (Hindi-English) languages. |
Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for improving multilingual models did not focus on learning the semantic structure of representation. |
| Approach: | They propose a method to improve multilingual language models by aligning parallel sentences . they propose token-, word-, sentence- and structure-level alignment objectives . |
| Outcome: | The proposed method outperforms baseline models on XNLI, PAWS-X, and XQuAD . it obtains comparable performance on low-resource languages, the authors show . |
Distilling Multilingual Transformers into CNNs for Scalable Intent Classification (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Existing multilingual models for voice assistants are limited by their prohibitive inference time and limited performance. |
| Approach: | They propose to distill and deploy multilingual Transformer models for voice assistants using a teacher-student framework that uses teacher-trained models to supervise student model training. |
| Outcome: | The proposed model outperforms a teacher model trained on unlabelled data and achieves equivalent performance. |
Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages (2023.findings-acl)
Copied to clipboard
| Challenge: | Cross-Lingual Knowledge Distillation (CLKD) is a method to train AS2 models for low-resource languages without labeled data. |
| Approach: | They propose a method to train AS2 models for low-resource languages without labeled data . they use a translation-based WikiQA dataset and a multilingual AS2 dataset . |
| Outcome: | The proposed method outperforms or rivals fine-tuning with labeled data and machine translation and the teacher model. |
One-Teacher and Multiple-Student Knowledge Distillation on Sentiment Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing knowledge distillation models require large computing resources and long inference time to perform. |
| Approach: | They propose a one-teacher and multiple-student knowledge distillation approach to distill a deep pre-trained teacher model into multiple shallow student models with ensemble learning. |
| Outcome: | The proposed method achieves better results with fewer parameters and extremely high speedup ratios on three sentiment classification tasks. |
Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on knowledge distillation have shown that not all knowledge is necessary for learning a good student model. |
| Approach: | They propose an actor-critic approach to selecting appropriate knowledge to transfer during the process of knowledge distillation. |
| Outcome: | The proposed method outperforms several strong knowledge distillation baselines significantly on the GLUE datasets. |
MergeDistill: Merging Language Models using Pre-trained Distillation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-trained multilingual language models often lack capacity and skewed data . this leads to inequitable representation of languages due to limited capacity and sub-optimal vocabularies. |
| Approach: | They propose a framework to merge pre-trained multilingual language models to maximize their assets with minimal dependencies. |
| Outcome: | The proposed framework outperforms teacher-trained models on multiple datasets and with a fixed model capacity. |
Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity Matching (2022.findings-naacl)
Copied to clipboard
Kunbo Ding, Weijie Liu, Yuejian Fang, Zhe Zhao, Qi Ju, Xuefeng Yang, Rong Tian, Zhu Tao, Haoyan Liu, Han Guo, Xingyu Bai, Weiquan Mao, Yudong Li, Weigang Guo, Taiqiang Wu, Ningyuan Sun
| Challenge: | Existing studies have shown that cross-lingual knowledge distillation can improve the performance of pre-trained models for cross-linguistic similarity matching tasks. |
| Approach: | They propose a multi-stage distillation framework for constructing a small-size but high-performance cross-lingual model using contrastive learning, bottleneck, and parameter recurrent strategies. |
| Outcome: | The proposed model can compress the size of XLM-R and MiniLM by more than 50% while the performance is only reduced by about 1%. |
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |
Beyond the Mode: Sequence-Level Distillation of Multilingual Translation Models for Low-Resource Language Pairs (2025.findings-naacl)
Copied to clipboard
Aarón Galiano-Jiménez, Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez, Víctor M. Sánchez-Cartagena
| Challenge: | Existing multilingual pre-trained models for low-resource languages have outperformed those trained from scratch for low resources due to high hardware requirements. |
| Approach: | They propose to use beam search to decode the whole output distribution of the teacher to improve student learning. |
| Outcome: | The proposed methods improve student model performance and reduce gender bias amplification common to beam search based methods. |