Learning from Imperfect Data: Towards Efficient Knowledge Distillation of Autoregressive Language Models for Text-to-SQL (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text-to-SQL LLMs are computationally expensive and difficult to deploy in real-world applications. |
| Approach: | They propose to distill a larger teacher model into a smaller student model by using imperfect data to improve the KD. |
| Outcome: | The proposed method achieves the best tradeoff between performance and efficiency on 5 text-to-SQL benchmarks. |
Similar Papers
Revisiting Knowledge Distillation for Autoregressive Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Autoregressive language models (LMs) are expensive and memory intensive, preventing the development of industrial applications. |
| Approach: | They propose an adaptive teaching approach to improve the KD of autoregressive language models by distilling knowledge into a small student model. |
| Outcome: | The proposed method can achieve consistent and significant performance gains across all model types and sizes. |
Learning with Less: Knowledge Distillation from Large Language Models via Unlabeled Data (2025.findings-naacl)
Copied to clipboard
Juanhui Li, Sreyashi Nag, Hui Liu, Xianfeng Tang, Sheikh Muhammad Sarwar, Limeng Cui, Hansu Gu, Suhang Wang, Qi He, Jiliang Tang
| Challenge: | Large Language Models (LLMs) have demonstrated superior language understanding abilities in many real-world NLP applications. |
| Approach: | They propose a learning-based sample selection method that incorporates signals from both teacher and student to enhance model performance. |
| Outcome: | The proposed method improves model performance across datasets with higher data efficiency. |
Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) face high computational demands at inference time due to high computational costs. |
| Approach: | They propose a cost-effective and high-throughput solution for large language models . PGKD distills the knowledge of LLMs into smaller, task-specific models based on teacher-student knowledge distillation . |
| Outcome: | PGKD outperforms BERT-based models and other knowledge distillation methods on multi-class classification datasets. |
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |
On the Generalization vs Fidelity Paradox in Knowledge Distillation (2025.findings-acl)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a key technique for compressing large language models into smaller ones while preserving performance. |
| Approach: | They propose to use knowledge distillation to compress large language models into smaller ones while preserving performance. |
| Outcome: | The proposed technique improves the performance of smaller models by 10% while providing only marginal benefits for larger models. |
Evolving Knowledge Distillation with Large Language Models and Active Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the direct use of large language models for text generation and labeling, without fully exploring their potential to comprehend the target task and acquire valuable knowledge. |
| Approach: | They propose to distill the knowledge of large language models into smaller models by generating annotated data. |
| Outcome: | The proposed method improves the performance of small domain models while enhancing the ability of large language models. |
GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to generate data using LLMs are limited by sampling from the center of original content distribution. |
| Approach: | They propose a task-agnostic data generation and knowledge distillation framework for LLMs that employs an iterative out-of-distribution-guided feedback mechanism to generate data. |
| Outcome: | The proposed framework outperforms prior arts and the LLM on 10 different classification tasks and noisey generated data. |
Autoregressive Knowledge Distillation through Imitation Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive models are ubiquitous in natural language processing due to the sequential nature of text generation. |
| Approach: | They propose a compression technique for autoregressive models driven by an imitation learning perspective on knowledge distillation. |
| Outcome: | The proposed method outperforms other distillation algorithms on translation and summarization tasks while increasing inference speed 14 times. |
A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training (2023.acl-long)
Copied to clipboard
| Challenge: | Modern Natural Language Generation models come with massive computational and storage requirements. |
| Approach: | They propose a method that applies word-level knowledge distillation to multiple PTs generated by both teacher and student. |
| Outcome: | The proposed techniques can be used to compress natural language models while preserving their performance. |
Knowledge Distillation for Language Models (2025.naacl-tutorial)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) aims to transfer knowledge from a teacher to a student . this tutorial will cover topics ranging from LLM sequence compression to LLM self-distillation . |
| Approach: | They propose to introduce intermediate-layer matching and prediction matching . they will then present advanced techniques such as reinforcement learning-based KD and multi-teacher distillation . |
| Outcome: | This tutorial aims to provide participants with a comprehensive understanding of the techniques and applications of knowledge distillation for language models. |