Multi-teacher Distillation for Multilingual Spelling Correction (2023.emnlp-industry)
Copied to clipboard
| Challenge: | a multilingual spelling correction model is needed to meet the tight latency requirements of multilingual NLP . a monolingual teacher model is trained for each language/locale, and individual models are distilled into a single student model . |
| Approach: | They propose a multilingual approach to spelling correction using multi-teacher distillation . they train a monolingual teacher model for each language and distill them into a single model . |
| Outcome: | The proposed model can meet the tight latency requirements of deployed services. |
Similar Papers
Distilling Multilingual Transformers into CNNs for Scalable Intent Classification (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Existing multilingual models for voice assistants are limited by their prohibitive inference time and limited performance. |
| Approach: | They propose to distill and deploy multilingual Transformer models for voice assistants using a teacher-student framework that uses teacher-trained models to supervise student model training. |
| Outcome: | The proposed model outperforms a teacher model trained on unlabelled data and achieves equivalent performance. |
LeSpell - A Multi-Lingual Benchmark Corpus of Spelling Errors to Develop Spellchecking Methods for Learner Language (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing spellcheckers do not work well with learner data. |
| Approach: | They propose a multi-lingual evaluation data set of spelling mistakes in context that is highly customizable for the DKPro architecture. |
| Outcome: | The proposed spellchecker improves performance in many settings and can be customized to meet learners' needs. |
Training a Better Chinese Spelling Correction Model via Prior-knowledge Guided Teacher (2024.findings-acl)
Copied to clipboard
| Challenge: | Chinese Spelling Correction models are prone to over-correct and poor generalization for error patterns outside the standard distribution. |
| Approach: | They propose a teacher network guided by prior knowledge for distillation learning of CSC models. |
| Outcome: | The proposed method significantly enhances the CSC model’s language modeling capabilities, crucial for minimizing over-correction. |
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on pre-trained language models focuses on reducing the size of these models into shallow ones. |
| Approach: | They propose a knowledge distillation technique that leverages teacher internal representations to reduce the size of pre-trained language models. |
| Outcome: | The proposed method outperforms previous methods in multilingual Named Entity Recognition (NER) it reduces the size of teacher models by 35x while retaining 95% of its F1 score. |
Multilingual Arbitration: Optimizing Data Pools to Accelerate Multilingual Progress (2025.acl-long)
Copied to clipboard
| Challenge: | Synthetic data generation relies on a single oracle teacher model, which can lead to model collapse and bias propagation. |
| Approach: | They propose a multilingual arbitration approach that exploits performance variations among multiple models for each language. |
| Outcome: | The proposed approach surpasses single-teacher distillation with 80% win rates over proprietary and open-weight models with the largest improvements in low-resource languages. |
Multilingual AMR Parsing with Noisy Knowledge Distillation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) parsing is a broad-coverage semantic formalism that encodes the meaning of a sentence as a rooted, directed, and labeled graph. |
| Approach: | They propose to use existing English parser to learn and improve multilingual AMR parsers . their results show that noisy input and precise output are key to successful distillation . |
| Outcome: | The proposed model outperforms the current state-of-the-art English-only parser on four different languages. |
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (P19-1)
Copied to clipboard
| Challenge: | Existing methods to train multi-task neural networks outperform or even match their single-task counterparts are difficult to implement. |
| Approach: | They propose a method that uses knowledge distillation to train multi-task neural networks that outperform or even match their single-task counterparts. |
| Outcome: | The proposed method outperforms or matches single-task neural networks on the GLUE benchmark. |
One-Teacher and Multiple-Student Knowledge Distillation on Sentiment Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing knowledge distillation models require large computing resources and long inference time to perform. |
| Approach: | They propose a one-teacher and multiple-student knowledge distillation approach to distill a deep pre-trained teacher model into multiple shallow student models with ensemble learning. |
| Outcome: | The proposed method achieves better results with fewer parameters and extremely high speedup ratios on three sentiment classification tasks. |
Beyond the Mode: Sequence-Level Distillation of Multilingual Translation Models for Low-Resource Language Pairs (2025.findings-naacl)
Copied to clipboard
Aarón Galiano-Jiménez, Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez, Víctor M. Sánchez-Cartagena
| Challenge: | Existing multilingual pre-trained models for low-resource languages have outperformed those trained from scratch for low resources due to high hardware requirements. |
| Approach: | They propose to use beam search to decode the whole output distribution of the teacher to improve student learning. |
| Outcome: | The proposed methods improve student model performance and reduce gender bias amplification common to beam search based methods. |
Improving Zero-Shot Multilingual Text Generation via Iterative Distillation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing approaches to generalize multilingual dialogue systems to multilingual settings often make assumptions about data availability. |
| Approach: | They propose to transfer inductive biases for target languages learned by pretrained teacher models to student models via sequence-level knowledge distillation. |
| Outcome: | The proposed method performs well on the multiATIS++ benchmark, and is comparable to human annotations in both slot F1 and intent accuracy. |