Metric-guided Distillation: Distilling Knowledge from the Metric to Ranker and Retriever for Generative Commonsense Reasoning

Challenge:	Existing work on commonsense generation requires models to have relational reasoning and compositional generalization capabilities.
Approach:	They propose a metric distillation rule to distill knowledge from a standard metric to a ranker and transfer it to re-ranking a retriever.
Outcome:	The proposed method surpasses the previous SOTA.

Similar Papers

Overview

Challenge

Approach

Outcome

Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)

Copied to clipboard

Luke Melas-Kyriazi, George Han, Celine Liang

Responsible & Trustworthy NLP

Challenge:	Recent research points to knowledge distillation as a potential solution for NLU tasks.
Approach:	They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples.
Outcome:	The proposed approach outperforms BERT training approaches while using 300 times fewer parameters.

I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation (2023.acl-long)

Copied to clipboard

Chandra Bhagavatula, Jena D. Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, Lianhui Qin, Keisuke Sakaguchi, Swabha Swayamdipta, Peter West, Yejin Choi

Semantic Text Processing

Challenge:	Empirical results suggest that scale is not the only way to build commonsense capabilities.
Approach:	They propose a commonsense distillation framework that can achieve a competitive level of commonsensing without relying on the benefits of scale.
Outcome:	The proposed framework breaks the dependence on the extreme-scale teacher model with two innovations: (1) the novel adaptation of NeuroLogic Decoding to enhance the generation quality of the weak, off-the-shelf language models, and (2) self-imitation learning to iteratively learn from the model’s own enhanced commonsense acquisition capabilities.

Query Distillation: BERT-based Distillation for Ensemble Ranking (2020.coling-industry)

Copied to clipboard

Wangshu Zhang, Junhong Liu, Zujie Wen, Yafang Wang, Gerard de Melo

Language Models

Challenge:	Recent years have witnessed substantial progress in the development of neural ranking networks, but an increasingly heavy computational burden due to growing numbers of parameters and the adoption of model ensembles.
Approach:	They propose a two-stage distillation method that allows a smaller student model to be trained while benefiting from the better performance of the teacher model.
Outcome:	The proposed method shows higher-quality rankings compared to the teacher model.

Accurate Knowledge Distillation via n-best Reranking (2024.naacl-long)

Copied to clipboard

Hendra Setiawan

Language Models

Challenge:	Existing studies using sequencelevel knowledge distillation (KD) have adopted this approach.
Approach:	They propose to utilize n-best reranking to enhance Sequence-Level Knowledge Distillation by utilizing a diverse set of models with different inductive biases, objective functions or architectures to pick the highest-quality hypotheses as labels.
Outcome:	The proposed approach is validated on the WMT’21 German English and Chinese english translation tasks.

Enhancing Task-Specific Distillation in Small Data Regimes through Language Generation (2022.coling-1)

Copied to clipboard

Husam Quteineh, Spyridon Samothrakis, Richard Sutcliffe

Language Models

Challenge:	Large-scale pretrained language models have led to significant improvements in Natural Language Processing, but they come at the cost of high computational and storage requirements.
Approach:	They propose to distill knowledge from larger models to smaller ones through pseudo-labels on task-specific datasets.
Outcome:	The proposed approach improves on the SST-2, MRPC, YELP-2, and TREC-6 datasets.

Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model (2024.naacl-long)

Copied to clipboard

Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, Yejin Choi

Text Generation

Challenge:	Impossible Distillation is a framework for paraphrasing and sentence summarization that can be trained from a low-quality teacher model.
Approach:	They propose a framework that distills a high-quality dataset from a low-quality teacher . they hypothesize and verify the paraphrastic proximity intrinsic to pre-trained LMs .
Outcome:	The proposed framework outperforms baseline models on unconstrained paraphrase generation and sentence summarization benchmarks.

Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain Conversation (2021.findings-emnlp)

Copied to clipboard

Beomsu Kim, Seokjun Seo, Seungju Han, Enkhbayar Erdenee, Buru Chang

Green & Sustainable NLP

Challenge:	generative models are less practical for building real-time conversation systems due to high latency and large memory footprint.
Approach:	They propose a method that preserves the efficiency of a retrieval model while leveraging the conversational ability of generative models.
Outcome:	The proposed method preserves the efficiency of a retrieval model while leveraging the conversational ability of generative models.

Bridging Relevance and Reasoning: Rationale Distillation in Retrieval-Augmented Generation (2025.findings-acl)

Copied to clipboard

Pengyue Jia, Derong Xu, Xiaopeng Li, Zhaocheng Du, Xiangyang Li, Yichao Wang, Yuhao Wang, Qidong Liu, Maolin Wang, Huifeng Guo, Ruiming Tang, Xiangyu Zhao

Responsible & Trustworthy NLP

Challenge:	Existing approaches to rerank and align documents based on reasoning capabilities of large language models (LLMs) . prior work shows that LLMs have exceptional reasoning and text generation capabilities .
Approach:	They propose a rationale extraction method that leverages reasoning capabilities of large language models to extract the rationales necessary for answering a query.
Outcome:	The proposed method is compared with baseline methods on two tasks across three datasets.

Retrieval Enhanced Model for Commonsense Generation (2021.findings-acl)

Copied to clipboard

Han Wang, Yang Liu, Chenguang Zhu, Linjun Shou, Ming Gong, Yichong Xu, Michael Zeng

Reasoning

Challenge:	Existing frameworks for commonsense generation are lacking for pre-trained models.
Approach:	They propose a framework that uses concept matching to retrieve prototype sentences and trainable sentence retriever to enhance pre-training and fine-tuning.
Outcome:	The proposed framework achieves state-of-the-art on the large-scale Common-Gen benchmark.

Distilling LLM Reasoning into Dense Encoders: Bridging the Accuracy-Efficiency Gap in Recommendation (2026.findings-acl)

Copied to clipboard

Donghee Han, Daeyoung Roh, A Young Kim, Hwanjun Song, Mun Yong Yi

Language Models

Challenge:	Existing distillation approaches target Small Language Models (SLMs) or Conventional Recommendation Models, but face a critical trade-off between computational cost and semantic reasoning capacity.
Approach:	They propose a framework that establishes a text encoder as the optimal student architecture for scalable recommendation.
Outcome:	Experiments on four datasets show that the proposed framework outperforms state-of-the-art models and achieves significantly reduced latency.