Comprehensive and Efficient Distillation for Lightweight Sentiment Analysis Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent efforts to develop lightweight and practical sentiment analysis models are limited by manual instruction and large-scale user texts. |
| Approach: | They propose a framework for sentiment analysis that uses attribute-based instruction construction and difficulty-based data filtering to distill knowledge. |
| Outcome: | The proposed framework outperforms baseline methods in data efficiency and performance. |
Similar Papers
Targeted Distillation for Sentiment Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies demonstrate that large language models exhibit remarkable capabilities and achieve state-of-the-art performance in diverse sentiment analysis tasks. |
| Approach: | They propose a distillation framework that decouples knowledge from alignment and introduces a sentiment analysis benchmark that covers a diverse set of tasks. |
| Outcome: | The proposed framework improves models' generalization to unseen tasks and their generalization is strong against existing small-scale models. |
One-Teacher and Multiple-Student Knowledge Distillation on Sentiment Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing knowledge distillation models require large computing resources and long inference time to perform. |
| Approach: | They propose a one-teacher and multiple-student knowledge distillation approach to distill a deep pre-trained teacher model into multiple shallow student models with ensemble learning. |
| Outcome: | The proposed method achieves better results with fewer parameters and extremely high speedup ratios on three sentiment classification tasks. |
uDistil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimes (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work on distilling Whisper’s knowledge into small models using pseudo-labels shows promising performance while reducing the size by up to 50%. |
| Approach: | They propose a framework that distills Whisper’s knowledge into small models using pseudo-labels and reduces the size by up to 50%. |
| Outcome: | The proposed model outperforms the teacher model by 5-7 WER points and is 25-50% more efficient when scaling the data. |
DistilQwen2.5: Industrial Practices of Training Distilled Open Lightweight Language Models (2025.acl-industry)
Copied to clipboard
| Challenge: | Existing studies on distilled lightweight LLMs have focused on transferring knowledge from a larger model (the teacher) to a smaller model (sector). |
| Approach: | They propose a family of distilled, lightweight LLMs derived from Qwen2.5 models. |
| Outcome: | Experimental results show that the distilled models have significantly stronger instruction-following capabilities than the original models. |
Towards General-Domain Word Sense Disambiguation: Distilling Large Language Model into Compact Disambiguator (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for Word Sense Disambiguation rely heavily on manually annotated data, which limits coverage and generalization. |
| Approach: | They propose a framework that leverages large language models as knowledge distillers to build silver-standard WSD corpora by combining generation-based distillation and annotation-based disambiguation. |
| Outcome: | The proposed framework outperforms existing methods on general-domain benchmarks by 50% on the most challenging test set and by 1000 times fewer parameters. |
DistillCSE: Distilled Contrastive Learning for Sentence Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to sentence embeddings are based on contrastive learning (CL) . |
| Approach: | They propose a framework which performs contrastive learning under the self-training paradigm with knowledge distillation and propose 'Group-P shuffling strategy' and averaging logits from multiple teacher components. |
| Outcome: | The proposed framework outperforms many strong baseline methods and yields a new state-of-the-art performance. |
To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing models for multilingual automatic speech recognition (ASR) are computationallyintensive and lack proper comprehensive evaluations. |
| Approach: | They propose to distill knowledge from large teacher models into smaller student variants that are more efficient. |
| Outcome: | The proposed model outperforms existing models on standard benchmarks and dialectal data. |
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on pre-trained language models focuses on reducing the size of these models into shallow ones. |
| Approach: | They propose a knowledge distillation technique that leverages teacher internal representations to reduce the size of pre-trained language models. |
| Outcome: | The proposed method outperforms previous methods in multilingual Named Entity Recognition (NER) it reduces the size of teacher models by 35x while retaining 95% of its F1 score. |
LEAF: Knowledge Distillation of Text Embedding Models with Teacher-Aligned Representations (2026.acl-long)
Copied to clipboard
| Challenge: | a new framework for text embedding models is available for free . asymmetrical architectures allow for flexible asymmetry, while asynchronous architectures require small batches . |
| Approach: | They propose a knowledge distillation framework for text embedding models that is compatible with their teacher . they publish leaf-ir, a 23M parameters information retrieval oriented model that ranks no.1 on BEIR . |
| Outcome: | The proposed model is compatible with teacher, enabling flexible asymmetric architectures . it sets a new state-of-the-art (SOTA) on BEIR, and achieves no.1 on the leaderboard . |
Skill-Aware Data Selection and Fine-Tuning for Data-Efficient Reasoning Distillation (2026.acl-short)
Copied to clipboard
| Challenge: | Large reasoning models such as DeepSeek-R1 and their distilled variants achieve impressive performance on complex reasoning tasks, yet their costs remain substantial. |
| Approach: | They propose a skill-centric distillation framework that efficiently transfers reasoning ability to weaker models with two components: (1) Skill-based data selection, which prioritizes examples targeting the student model’s weaker skills, and (2) Skillaware fine-tuning, which encourages explicit skill decomposition during problem solving. |
| Outcome: | The proposed framework surpasses baselines on Qwen3-4B and Qwend3-8B and focuses on skills emphasized during training. |