DISTILLATION

Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks (2022.emnlp-industry)

Challenge:	Large language models are often inefficient for real-world deployment due to expensive inference costs.
Approach:	They propose to use knowledge distillation to transfer the knowledge of the original model to a smaller, more efficient student model.
Outcome:	The proposed method is the best for multi-lingual and multilingual student architectures.

Copied to clipboard

Charith Peris, Lizhen Tan, Thomas Gueudre, Turan Gojayev, Pan Wei, Gokmen Oz

LMdiff: A Visual Diff Tool to Compare Language Models (2021.emnlp-demo)

Challenge:	Domain Classification (DC) and Intent Classification/Named Entity Recognition (ICNER) are the most common methods for reducing teacher-student knowledge into manageable sizes for low-latency downstream applications.
Approach:	They investigate whether distillation from a generic LM benefits downstream tasks . a domain classification and a task-specific data set are used to fine tune the model .
Outcome:	The proposed model improves across tasks and test sets when only task-specific data is used.

Copied to clipboard

Hendrik Strobelt, Benjamin Hoover, Arvind Satyanaryan, Sebastian Gehrmann

Multimodality

Challenge:	LMdiff visually compares probability distributions of two different language models . notably absent from the range of available tools are those that aim to compare distributions produced by different models.
Approach:	They propose a tool that visually compares probability distributions of two different language models that differ through finetuning, distillation, or simply training with different parameter sizes.
Outcome:	The proposed tool allows the generation of hypotheses about model behavior by investigating text instances token by token and further assists in choosing interesting text instances from large corpora.

Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)

Copied to clipboard

Luke Melas-Kyriazi, George Han, Celine Liang

SPPD: Self-training with Process Preference Learning Using Dynamic Value Margin (2025.findings-emnlp)

Challenge:	Recent research points to knowledge distillation as a potential solution for NLU tasks.
Approach:	They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples.
Outcome:	The proposed approach outperforms BERT training approaches while using 300 times fewer parameters.

Copied to clipboard

Hao Yi, Qingyang Li, Yulan Hu, Fuzheng Zhang, Di Zhang, Yong Liu

Numerical Reasoning

Challenge:	Existing approaches to improve numerical and logical reasoning of Large Language Models are limited . existing approaches rely on prompt engineering and pretrained knowledge to ensure correctness .
Approach:	They propose to train LLMs with process-based reasoning using a dynamic value margin . they use the Bellman optimality equation to derive a value margin for step-level preference optimization .
Outcome:	The proposed method is equivalent to on-policy policy gradient methods under constrained reward functions.

BMCook: A Task-agnostic Compression Toolkit for Big Models (2022.emnlp-demos)

Copied to clipboard

Zhengyan Zhang, Baitao Gong, Yingfa Chen, Xu Han, Guoyang Zeng, Weilin Zhao, Yanxu Chen, Zhiyuan Liu, Maosong Sun

LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference (2026.acl-industry)

Challenge:	Existing efforts to compress medium-sized models for specific tasks have limited results.
Approach:	They propose a task-agnostic compression toolkit for big models that implements quantization, pruning, distillation and MoEfication methods.
Outcome:	The proposed tool improves performance on a model with 3 billion parameters by 12x . it also outperforms the original model on three typical NLP benchmarks.

Copied to clipboard

Shashank Kapadia, Deep Narayan Mishra, Sujal Reddy Alugubelli, Haoan Wang, Saipraveen Vabbilisetty, Rishi Bhatia, Anupriya Sharma

CORD: Balancing COnsistency and Rank Distillation for Robust Retrieval-Augmented Generation (2025.naacl-short)

Challenge:	Layer-aligned distillation and convergence-based early exit are dominant computational efficiency paradigms for transformer inference.
Approach:	They propose a training objective that aligns intermediate student layers to teacher representations and reconciles this incompatibility with standard distillation.
Outcome:	The proposed model achieves 1.61 measured wall-clock speedup with 91.9% of samples exiting by layer 7 and 1.80 theoretical layer reduction, where standard distilled models achieve zero effective speedup.

Copied to clipboard

Youngwon Lee, Seung-won Hwang, Daniel F Campos, Filip Graliński, Zhewei Yao, Yuxiong He

BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillation (2025.coling-main)

Challenge:	Existing methods to ground large language models fail to adequately attend to all contexts . position bias is hindered by retrieval-augmented generation, which requires constant attention .
Approach:	They propose to augment and distill training instances with their perturbed positions to encourage consistent predictions . they also propose to balance COnsistency and Rank Distillation by combining noise-controlled perturbations with augmentation and distillation.
Outcome:	The proposed method outperforms existing methods in diverse RAG benchmarks.

Copied to clipboard

Minchong Li, Feng Zhou, Xiaohui Song

Structured Pruning Learns Compact and Accurate Models (2022.acl-long)

Challenge:	Knowledge distillation (KD) is a method for reducing model size while preserving performance.
Approach:	They propose a method to distill large language models at the logit level by transferring knowledge from a large teacher model to a smaller student model.
Outcome:	The proposed method outperforms supervised fine-tuning, vanilla KL loss and five other distillation methods on 13 datasets.

Copied to clipboard

Mengzhou Xia, Zexuan Zhong, Danqi Chen

Combining Curriculum Learning and Knowledge Distillation for Dialogue Generation (2021.findings-emnlp)

Challenge:	Pre-trained language models have high costs in terms of storage, memory, and computation time.
Approach:	They propose a task-specific structured pruning method CoFi which provides highly parallelizable subnetworks and matches distillation methods in both accuracy and latency.
Outcome:	The proposed method matches the distillation methods in accuracy and latency without resorting to unlabeled data.

Copied to clipboard

Qingqing Zhu, Xiuying Chen, Pengfei Wu, JunFei Liu, Dongyan Zhao

Dialogue Systems & Conversational Agents

Challenge:	Existing studies have shown that curriculum learning facilitates dialogue generation tasks while knowledge distillation can yield significant performance boosts for student models.
Approach:	They propose a combination of curriculum learning and knowledge distillation for dialogue generation models . they cluster training cases according to their complexity and employ an adversarial training strategy .
Outcome:	The proposed model improves compared with baselines.

Tailoring Instructions to Student’s Learning Levels Boosts Knowledge Distillation (2023.acl-long)

Copied to clipboard

Yuxin Ren, Zihan Zhong, Xingjian Shi, Yi Zhu, Chun Yuan, Mu Li

Distilling Knowledge for Search-based Structured Prediction (P18-1)

Challenge:	Recent success of natural language processing (NLP) is driven by the adoption of large-scale pretrained language models.
Approach:	They propose a method to determine the impact of distillation influence on student generalization ability by prioritizing samples likely to enhance the student's generalization abilities.
Outcome:	The proposed method outperforms 10 common knowledge distillation baselines on 6 text classification tasks in the GLUE benchmark.

Copied to clipboard

Yijia Liu, Wanxiang Che, Huaipeng Zhao, Bing Qin, Ting Liu

Efficient Nearest Neighbor Search for Cross-Encoder Models using Matrix Factorization (2022.emnlp-main)

Challenge:	Existing studies have focused on the performance of structured prediction models, but they are often limited by the ambiguities of the reference policy.
Approach:	They propose to distill an ensemble of multiple models trained with different initializations into a single model and use it to explore the search space.
Outcome:	The proposed model outperforms the greedy models on two typical search-based structured prediction tasks and achieves 1.32 in LAS and 2.65 in BLEU over strong baselines.

Copied to clipboard

Nishant Yadav, Nicholas Monath, Rico Angell, Manzil Zaheer, Andrew McCallum

PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning (2024.naacl-long)

Challenge:	Efficient k-nearest neighbor search is a fundamental task, foundational for many problems in NLP.
Approach:	They propose an approach that avoids the use of a dual-encoder for retrieval, relying solely on the cross-encoding model.
Outcome:	Empirically, for k > 10, our approach provides test-time recall-vs-computational cost trade-offs superior to the current widely-used methods that re-rank items retrieved using a dual-encoder or TF-IDF.

Copied to clipboard

Xuekai Zhu, Biqing Qi, Kaiyan Zhang, Xinwei Long, Zhouhan Lin, Bowen Zhou

Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification (2026.findings-eacl)

Challenge:	Large language models excel in various tasks, but their huge size and inaccessibility of parameters present challenges for practical deployment.
Approach:	They propose to use CoT data to distill task-specific ability from large language models to smaller models . they use reasoning programs to suppress errors in distilled data and improve distillation quality .
Outcome:	The proposed model outperforms LLMs on arithmetic reasoning, symbolic reasoning, and general ability.

Copied to clipboard

Branislav Pecher, Jan Cegin, Robert Belanec, Ivan Srba, Jakub Simko, Maria Bielikova

Information Retrieval

Challenge:	Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, making them promising tools in both high- and low-resource languages.
Approach:	They use a multilingual LLM to generate synthetic datasets covering 11 languages and 4 classification tasks and use them to train smaller models.
Outcome:	The proposed model outperforms the large generator in low-resource languages and tasks.

Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation (2021.findings-emnlp)

Copied to clipboard

Humair Raj Khan, Deepak Gupta, Asif Ekbal

TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors (2026.acl-long)

Challenge:	Existing methods for training language-vision models only consider monolingual learning, especially English.
Approach:	They propose to extend an English language-vision model into a multilingual and code-mixed model by using knowledge distillation techniques.
Outcome:	The proposed model outperforms existing models on multilingual and code-mixed VQA datasets on eleven languages.

Copied to clipboard

Ido Andrew Atad, Itamar Zimerman, Shahar Katz, Lior Wolf

Operator Selection and Ordering in a Pipeline Approach to Efficiency Optimizations for Transformers (2023.findings-acl)

Challenge:	Existing attention-aggregation methods focus on individual attention heads or layers, failing to account for the model’s global behavior.
Approach:	They propose a unified attention representation that captures the entire transformer as a single, input-dependent linear operator expressed through a high-order attention-interaction tensor.
Outcome:	The proposed model encapsulates the entire transformer as a single, input-dependent linear operator expressed through a high-order attention-interaction tensor.

Copied to clipboard

Ji Xin, Raphael Tang, Zhiying Jiang, Yaoliang Yu, Jimmy Lin

OBLIVIATE: Robust and Practical Machine Unlearning for Large Language Models (2025.emnlp-main)

Challenge:	Natural language processing tasks rely on complex neural models . transformer-based models are typically slow to execute, making it a non-trivial challenge to apply them in real-world applications.
Approach:	They propose to consider an efficiency method as an operator applied on a model . they find that the commutativity and cumulativeness of efficiency operators are plausible .
Outcome:	The proposed method is commutative and cumulative, and the results are estimated by combining methods.

Copied to clipboard

Xiaoyu Xu, Minxin Du, Qingqing Ye, Haibo Hu

Towards Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learning (2020.coling-main)

Challenge:	Large language models (LLMs) trained over corpora risk memorizing sensitive, copyrighted, or toxic content.
Approach:	They propose a framework that removes targeted data while preserving model utility.
Outcome:	The proposed framework resists membership inference attacks, minimizes impact on retained data, and maintains robustness across diverse scenarios.

Copied to clipboard

Weipeng Huang, Xingyi Cheng, Kunlong Chen, Taifeng Wang, Wei Chu

Distillation of encoder-decoder transformers for sequence labelling (2023.findings-eacl)

Challenge:	Chinese word segmentation datasets have ambiguous annotation criteria resulting in multi-grained compounds.
Approach:	They propose a domain adaptive segmenter to exploit diverse annotation criteria of datasets . they use bidirectional encoder representations from transformers to introduce open-domain knowledge .
Outcome:	The proposed model outperforms the state-of-the-art models on 10 Chinese word datasets with superior efficiency.

Copied to clipboard

Marco Farina, Duccio Pappadopulo, Anant Gupta, Leslie Huang, Ozan Irsoy, Thamar Solorio

DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation (2024.findings-naacl)

Challenge:	despite the strong trend in NLP to explore the use of large language models, there is still limited work evaluating prompting and decoding mechanisms for SL tasks.
Approach:	They propose a hallucination-free framework for sequence tagging that is especially suited for distillation.
Outcome:	The proposed framework performs well across multiple sequence labelling datasets and in a few-shot learning scenario.

Copied to clipboard

Aru Maekawa, Satoshi Kosugi, Kotaro Funakoshi, Manabu Okumura

Learning to Reason via Self-Iterative Process Feedback for Small Language Models (2025.coling-main)

Challenge:	Existing methods to extract word embeddings from training datasets are not efficient for training other models.
Approach:	They propose a method to distill a training dataset into a textual model by combining a small number of informative synthetic samples.
Outcome:	The proposed method outperforms existing methods on training datasets and language models.

Copied to clipboard

Kaiyuan Chen, Jin Wang, Xuejie Zhang

DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows (2024.acl-long)

Challenge:	Existing methods for enhancing SLMs’ reasoning depend on costly external signals, resulting in SLM overly confident with limited supervision signals.
Approach:	They propose to fine-tune and align SLMs using positive and negative feedback signals and introduce process supervision for rewards in preference alignment by sampling-based inference simulation and process reward models.
Outcome:	The proposed method improves Gemma-2B's performance on GSM8K and MBPP, and out-of-domain generalization capabilities on MMLU_Math and HumanEval.

Copied to clipboard

Ajay Patel, Colin Raffel, Chris Callison-Burch

Aspect-Based Sentiment Analysis with Syntax-Opinion-Sentiment Reasoning Chain (2025.coling-main)

Challenge:	Large language models (LLMs) have become a dominant tool for NLP researchers in a wide range of tasks.
Approach:	They propose an open source Python library that allows researchers to write simple code to implement powerful LLM workflows.
Outcome:	The proposed library is open source and can be used to implement powerful LLM workflows.

Copied to clipboard

Rui Fan, Shu Li, Tingting He, Yu Liu

Sentiment Analysis

Challenge:	Syntactic structures are crucial for capturing aspect-opinion relationships . syntactically based models struggle with linguistic complexities .
Approach:	They propose a syntactic-opinion-sentiment reasoning framework that leverages syntaktic information to improve ABSA performance.
Outcome:	The proposed framework improves ABSA performance, though smaller LLMs exhibit weaker performance.

Continual Contrastive Spoken Language Understanding (2024.findings-acl)

Copied to clipboard

Umberto Cappellazzo, Enrico Fini, Muqiao Yang, Daniele Falavigna, Alessio Brutti, Bhiksha Raj

Challenge:	Recent advances in speech processing require extensive offline training . however, these models struggle to retain their previously acquired knowledge when learning new tasks continuously.
Approach:	They propose a method that relies on experience replay and contrastive learning to preserve the learned representations by pulling closer samples from the same class and pushing away the others.
Outcome:	The proposed method preserves the learned representations by pulling closer samples from the same class and pushing away the others.

Getting the Most out of Simile Recognition (2022.findings-emnlp)

Copied to clipboard

Xiaoyue Wang, Linfeng Song, Xin Liu, Chulun Zhou, Hualin Zeng, Jinsong Su

Challenge:	Recent work ignores features other than surface strings and suffers from data hunger issue.
Approach:	They propose to use simile sentence classification and simile component extraction to find simile components.
Outcome:	The proposed model outperforms current state-of-the-art systems and baselines.

Multilingual AMR Parsing with Noisy Knowledge Distillation (2021.findings-emnlp)

Copied to clipboard

Deng Cai, Xin Li, Jackie Chun-Sing Ho, Lidong Bing, Wai Lam

Quantification of Large Language Model Distillation (2025.acl-long)

Challenge:	Abstract Meaning Representation (AMR) parsing is a broad-coverage semantic formalism that encodes the meaning of a sentence as a rooted, directed, and labeled graph.
Approach:	They propose to use existing English parser to learn and improve multilingual AMR parsers . their results show that noisy input and precise output are key to successful distillation .
Outcome:	The proposed model outperforms the current state-of-the-art English-only parser on four different languages.

Copied to clipboard

Sunbowen Lee, Junting Zhou, Chang Ao, Kaige Li, Xeron Du, Sirui He, Haihong Wu, Tianci Liu, Jiaheng Liu, Hamid Alinejad-Rokny, Min Yang, Yitao Liang, Zhoufutu Wen, Shiwen Ni

DTDES-KGE: Dual-Teacher Knowledge Distillation with Distinct Embedding Spaces for Knowledge Graph Embeddings (2025.findings-emnlp)

Challenge:	Existing studies have revealed the robustness degra-dation caused by data distillation.
Approach:	They propose a framework to evaluate and quantify model distillation . they aim to identify identity cognition contradictions and analyse multi-granularity response similarities across models to measure the extent of homogenization.
Outcome:	The proposed framework addresses two key aspects: (1) Identifying identity cognition contradictions to assess discrepancies in how models perceive and represent identity-related information; (2) Analyzing multi-granularity response similarities across models to measure the extent of homogenization.

Copied to clipboard

Bofan Wei, Hongyuan Xu, Yuhang Niu, Jiarui Ren, Yanlong Wen, Xiaojie Yuan

Aligned Weight Regularizers for Pruning Pretrained Neural Networks (2022.findings-acl)

Challenge:	Existing knowledge distillation methods rely on a single teacher embedding space . existing methods overlook valuable complementary knowledge from teachers in distinct embeddable spaces.
Approach:	They propose a knowledge distillation framework that leverages dual teachers in embedding spaces to enhance performance.
Outcome:	The proposed framework significantly improves knowledge distillation performance by leveraging dual teachers in distinct embedding spaces.

Copied to clipboard

James O’ Neill, Sourav Dutta, Haytham Assem

Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021.findings-emnlp)

Challenge:	Pruning aims to reduce the number of parameters while maintaining performance close to the original network.
Approach:	They propose a self-distilled pruning strategy that maximizes representational similarity between pruned and unpruned networks.
Outcome:	The proposed pruning strategy outperforms smaller models and outperformed smaller ones with an equal number of parameters and is competitive against (6 times) larger distilled networks.

Copied to clipboard

Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, Orhan Firat

Causal Distillation for Language Models (2022.naacl-main)

Challenge:	Sparse Mixture-of-Experts (MoE) is a successful approach for scaling multilingual translation models to billions of parameters without a proportional increase in training computation.
Approach:	They propose to use a task-level routing approach to extract smaller, ready-to-deploy sub-networks from large sparse models by ignoring distillation.
Outcome:	Experiments on WMT and a web-scale dataset show that task-level routing outperforms token-level MoE models by +1.0 BLEU on average across 30 language pairs.

Copied to clipboard

Zhengxuan Wu, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, Noah Goodman

Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models (2022.findings-emnlp)

Challenge:	Distillation efforts have led to language models that are more compact and efficient without serious drops in performance.
Approach:	They propose to augment distillation with a third objective that encourages the student model to imitate the causal dynamics of the teacher through a distillation interchange intervention training objective (DIITO).
Outcome:	The proposed method lowers perplexity on the WikiText-103M corpus and improves on the GLUE benchmark, SQuAD, and CoNLL-2003.

Copied to clipboard

Clara Na, Sanket Vaibhav Mehta, Emma Strubell

Distillation-Resistant Watermarking for Model Protection in NLP (2022.findings-emnlp)

Challenge:	Recent advances in hardware, modeling, and optimization for deep neural networks have led to improvements in memory and inference efficiency.
Approach:	They propose to combine sharpness-aware minimization with various model compression methods to improve model compressibility.
Outcome:	Empirically, optimizing for flatter minima leads to greater compressibility of parameters compared to vanilla Adam when fine-tuning BERT models, with little to no loss in accuracy on the GLUE text classification and SQuAD question answering benchmarks.

Copied to clipboard

Xuandong Zhao, Lei Li, Yu-Xiang Wang

Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer Rationales (2025.findings-emnlp)

Challenge:	Existing protection methods such as watermarking only work for images but are not applicable to text.
Approach:	They propose a technique that injects watermarks into the victim’s prediction probability corresponding to a secret key and is able to detect such a key by probing a suspect model.
Outcome:	The proposed technique detects stealing suspects at 100% accuracy on four NLP tasks while the prior method fails on two.

Copied to clipboard

JianZhi Yan, Le Liu, Youcheng Pan, Shiwei Chen, Yang Xiang, Buzhou Tang

Personalized Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation (2023.emnlp-main)

Challenge:	Existing work on rationale quality underestimates the importance of CoT distillation, focusing primarily on data quantity, which may result in transferring noisy or incorrect information to the student model.
Approach:	They propose a method which can discern and select high quality rationales for distillation and a Rationale Difficulty metric to measure the ability of the student model to generate the correct answer under a given rationale.
Outcome:	The proposed method achieves 4.6% accuracy improvement over baseline data on seven datasets over three tasks, controlling accuracy, diversity, and difficulty.

Copied to clipboard

Hailin Chen, Amrita Saha, Steven Hoi, Shafiq Joty

Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher (2022.coling-1)

Challenge:	Recent studies have shown that close-sourced LLMs lack the ability to integrate into real-world applications due to their high associated costs and ethical concerns.
Approach:	They propose to use student model to refine its own solution by querying ChatGPT to generate task instruction and solution pairs and querying data to refine model.
Outcome:	The proposed model outperforms standard distillation with only one third of the data.

Copied to clipboard

Mehdi Rezagholizadeh, Aref Jafari, Puneeth S.M. Saladi, Pranav Sharma, Ali Saheb Pasand, Ali Ghodsi

On the Analysis and Distillation of Emergent Outlier Properties in Pre-trained Language Models (2025.naacl-long)

Challenge:	Knowledge distillation (KD) is a powerful tool for deep learning applications.
Approach:	They propose a method which defines a smoother training path for the student by following the training footprints of the teacher rather than solely relying on distilling from a single mature fully-trained teacher.
Outcome:	The proposed technique is quite effective in mitigating the capacity-gap problem and the checkpoint search problem.

Copied to clipboard

Tianyang Zhao, Kunwar Yashraj Singh, Srikar Appalaraju, Peng Tang, Ying Nian Wu, Li Erran Li

Patient Knowledge Distillation for BERT Model Compression (D19-1)

Challenge:	Existing studies show that a small subset of dimensions within language Transformers’ representation spaces emerge as "outliers" during pretraining.
Approach:	They propose a method that prioritizes critical outlier dimensions in distillation using a weighted MSE loss.
Outcome:	The proposed method outperforms state-of-the-art distillation methods and generalizes well across Encoder-only BERT, Decoder-only GPT-2, and Encodeer-Decoder T5 architectures.

Copied to clipboard

Siqi Sun, Yu Cheng, Zhe Gan, Jingjing Liu

An Empirical Study of Multilingual Reasoning Distillation for Question Answering (2024.emnlp-main)

Challenge:	Pre-trained language models such as BERT have proven to be highly effective for natural language processing tasks, but the high demand for computing resources hinders their application in practice.
Approach:	They propose to compress an original large model (teacher) into an equally-effective lightweight shallow network (student) Empirically, this translates into improved results on multiple NLP tasks with a significant gain in training efficiency, without sacrificing model accuracy.
Outcome:	The proposed model reduces the computational cost of training models using the teacher model into a lightweight shallow network.

Copied to clipboard

Patomporn Payoungkhamdee, Peerat Limkonchotiwat, Jinheon Baek, Potsawee Manakul, Can Udomcharoenchaikit, Ekapol Chuangsuwanich, Sarana Nutanong

IIET: Efficient Numerical Transformer via Implicit Iterative Euler Method (2025.emnlp-main)

Challenge:	Existing efforts to distill reasoning capabilities have focused mainly on English, leaving multilingual distillation underexplored.
Approach:	They propose a method that incorporates incorrect rationales as additional guidance to improve multilingual reasoning in large language models.
Outcome:	Empirical results show that d-CoT-nR significantly surpasses the baseline, improving accuracy in unseen languages and correctness in step-by-step reasoning.

Copied to clipboard

Xinyu Liu, Bei Li, Jiahao Liu, Junhao Ruan, Kechen Jiao, Hongyin Tang, Jingang Wang, Tong Xiao, JingBo Zhu

Effective Distillation of Table-based Reasoning Ability from LLMs (2024.lrec-main)

Challenge:	High-order numerical methods enhance performance in tasks like NLP but introduce a performance-efficiency trade-off due to increased computational overhead.
Approach:	They propose an iterative implicit Euler Transformer which simplifies high-order numerical methods by iterating implicit Eule.
Outcome:	The proposed method improves accuracy and reduces inference overhead by 55% while retaining 99.4% of the original task accuracy.

Copied to clipboard

Bohao Yang, Chen Tang, Kun Zhao, Chenghao Xiao, Chenghua Lin

Selective Knowledge Distillation for Neural Machine Translation (2021.acl-long)

Challenge:	Existing work on table-based reasoning distillation has focused on smaller models with limited performance.
Approach:	They propose a table-based reasoning distillation approach to distill LLMs into smaller models . their results show that a 220 million parameter model fine-tuned using distilled data improves performance .
Outcome:	The proposed model improves on a scientific table-to-text generation dataset and surpasses specific LLMs.

Copied to clipboard

Fusheng Wang, Jianhao Yan, Fandong Meng, Jie Zhou

Stronger, Lighter, Better: Towards Life-Long Attribute Value Extraction for E-Commerce Products (2024.findings-acl)

Challenge:	Neural Machine Translation models achieve state-of-the-art performance on many translation benchmarks.
Approach:	They propose a protocol that analyzes different impacts of samples by comparing various samples’ partitions.
Outcome:	The proposed methods yield up to +1.28 and +0.89 BLEU points improvements over the Transformer baseline, respectively.

Copied to clipboard

Tao Zhang, Chenwei Zhang, Xian Li, Jingbo Shang, Hoang Nguyen, Philip Yu

Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMs (2024.lrec-main)

Challenge:	Existing models for attribute value extraction struggle for parameter efficiency and reliability due to data contamination and catastrophic forgetting.
Approach:	They propose to decouple product type and attribute to promote de-contamination and parameter efficiency while scaling up.
Outcome:	The proposed model achieves state-of-the-art performance with affordable parameter size, least historical knowledge forgetting, and greatest robustness against noises.

Copied to clipboard

Zhihong Sun, Chen Lyu, Bolun Li, Yao Wan, Hongyu Zhang, Ge Li, Zhi Jin

Text Generation

Challenge:	Large Language Models (LLMs) have made significant advances in code generation through the ‘Chain-of-Thought’ prompting technique.
Approach:	They propose a framework which aims to transfer LLMs’ reasoning capabilities to smaller models through distillation.
Outcome:	The proposed framework improves the smaller model's code generation performance by over 130% on the APPS benchmark.

Momentum Posterior Regularization for Multi-hop Dense Retrieval (2025.coling-main)

Copied to clipboard

Zehua Xia, Yuyang Wu, Yiyun Xia, Cam Tu Nguyen

A Strategic Coordination Framework of Small LMs Matches Large LMs in Data Synthesis (2025.acl-long)

Challenge:	Current methods for knowledge distillation in one-time retrieval are ineffective for multi-hop QA . posterior information is often defined as the response, which may not connect to the query without intermediate retrieval .
Approach:	They propose to distill knowledge from a posterior retrieval into a prior retrieval for multi-hop QA . they propose to use momentum moving average method to update posterior information along with prior retrievals .
Outcome:	Experiments on HotpotQA and StrategyQA show that MoPo outperforms baselines in retrieval and downstream QA tasks.

Copied to clipboard

Xin Gao, Qizhi Pei, Zinan Tang, Yu Li, Honglin Lin, Jiang Wu, Lijun Wu, Conghui He

Unlocking Recursive Thinking of LLMs: Alignment via Refinement (2025.findings-acl)

Challenge:	Large Language Models suffer from high computational costs and environmental inefficiency . smaller LMs are more accessible and sustainable, but their individual capabilities often fall short . a collaborative framework for small LM combines specialized roles to iterative refinement and quality control .
Approach:	They propose a framework that aggregates specialized roles across small LMs to iterative refinement and quality control typically achieved by a single large LM.
Outcome:	The proposed framework aggregates specialized roles across small LMs to iterative refinement and quality control typically achieved by large LM.

Copied to clipboard

Haoke Zhang, Xiaobo Liang, Cunxiang Wang, Juntao Li, Min Zhang

Challenge:	Existing methods for recursive reasoning are limited due to lack of expert-curated data.
Approach:	They propose a method that unlocks the potential of Large Language Models for recursive reasoning through long-form Chain of Thought.
Outcome:	The proposed method outperforms preference optimization methods on the openAI o1-series models by 20% on 3k synthetic samples.

Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering (2024.emnlp-main)

Copied to clipboard

Sungho Ko, Hyunjin Cho, Hyungjoo Chae, Jinyoung Yeo, Dongha Lee

Perturbed examples reveal invariances shared by language models (2024.findings-acl)

Challenge:	Existing methods for enhancing QA performance of Large Language Models (LLMs) have limitations, including duplicated entities or relations, reduced evidence density, and failure to highlight crucial evidence.
Approach:	They propose an Evidence-focused Fact Summarization framework for enhanced QA with knowledge-augmented Large Language Models (LLMs) that incorporates external knowledge into LLMs to improve QA performance.
Outcome:	The proposed framework improves LLM’s zero-shot QA performance especially when noisy facts are retrieved.

Copied to clipboard

Ruchit Rawal, Mariya Toneva

FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation (2024.emnlp-main)

Challenge:	Rapid growth in natural language processing (NLP) research has led to numerous new models outpacing our understanding of how they compare to established ones.
Approach:	They propose a framework to compare two NLP models by revealing their shared invariance to interpretable input perturbations targeting a specific linguistic capability.
Outcome:	The proposed framework can shed light on the types of invariances retained or emerging in new models.

Copied to clipboard

KaShun Shum, Minrui Xu, Jianshu Zhang, Zixin Chen, Shizhe Diao, Hanze Dong, Jipeng Zhang, Muhammad Raza

Small But Funny: A Feedback-Driven Approach to Humor Distillation (2024.acl-long)

Challenge:	Experimental results show that a well-calibrated model is more reliable than a fine-tuned model due to “tuning-induced mis-calibration”.
Approach:	They propose a method which utilizes a small portion of teacher’s knowledge to obtain a reliable language model in a cost-efficient way.
Outcome:	The proposed method reduces the computational burden by utilizing teacher's knowledge to obtain a reliable language model in a cost-efficient way.

Copied to clipboard

Sahithya Ravi, Patrick Huber, Akshat Shrivastava, Vered Shwartz, Arash Einolghozati

Latent Agents: A Post-Training Procedure for Internalized Multi-Agent Debate (2026.acl-long)

Challenge:	Large Language Models (LLMs) have been used to transfer knowledge from LLMs to smaller, smaller language models (SLMs).
Approach:	They propose to assign a dual role to the LLM as a “teacher” generating data, as well as evaluating the student’s performance.
Outcome:	The proposed approach narrows the performance gap between LLMs and larger models by incorporating feedback into the data.

Copied to clipboard

John Seon Keun Yi, Aaron Mueller, Dokyun Lee

mCLIP: Multilingual CLIP via Cross-lingual Transfer (2023.acl-long)

Challenge:	Multi-agent debate is compute-intensive and requires long transcripts before answering questions.
Approach:	They propose a framework that distills multi-agent debate into a single LLM by combining debate structure learning with internalization via dynamic reward scheduling and length clipping.
Outcome:	The proposed model matches or exceeds explicit multi-agent debate performance using 93% fewer tokens across multiple models and benchmarks.

Copied to clipboard

Guanhua Chen, Lu Hou, Yun Chen, Wenliang Dai, Lifeng Shang, Xin Jiang, Qun Liu, Jia Pan, Wenping Wang

Multilinguality

Challenge:	Existing multilingual vision-language pretrained models are biased towards English due to the lack of sufficient non-English image-text pairs.
Approach:	They propose to train a retrieval-efficient dual-stream multilingual VLP model by aligning CLIP model and a multilingual text encoder through a novel Triangle Cross-modal Knowledge Distillation method.
Outcome:	Empirical results show that mCLIP achieves new state-of-the-art performance for both zero-shot and finetuned multilingual image-text retrieval tasks.

Long-Chain Reasoning Distillation via Adaptive Prefix Alignment (2026.acl-long)

Copied to clipboard

Zhenghao Liu, Zhuoyang Wu, Xinze Li, Yukun Yan, Shuo Wang, Zulong Chen, Yu Gu, Ge Yu, Maosong Sun

A Comparative Study on the Impact of Model Compression Techniques on Fairness in Language Models (2023.acl-long)

Challenge:	Large Language Models (LLMs) have demonstrated remarkable reasoning capabilities, especially in solving complex mathematical problems.
Approach:	They propose a framework that exploits teacher CoTs for distillation through adaptive prefix alignment.
Outcome:	The proposed framework outperforms baseline models on multiple mathematical reasoning benchmarks by over 3%.

Copied to clipboard

Krithika Ramesh, Arnav Chavan, Shrey Pandit, Sunayana Sitaram

CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuning (2025.acl-long)

Challenge:	Existing literature demonstrates that compressing deep learning models could affect their fairness.
Approach:	They evaluate pruned, distilled, and quantized language models to assess their fairness . they also examine the impact of using multilingual models and evaluation measures .
Outcome:	The proposed methods can reduce the fairness of language models by reducing their complexity and reducing the cost of training and deployment.

Copied to clipboard

Yangfan Ye, Xiaocheng Feng, Zekun Yuan, Xiachong Feng, Libo Qin, Lei Huang, Weitao Ma, Yichong Huang, Zhirui Zhang, Yunfei Lu, Xiaohui Yan, Duyu Tang, Dandan Tu, Bing Qin

Multilinguality

Challenge:	Existing fine-tuning approaches that focus on English-centric training corpora often introduce implicit cross-lingual alignment, overlooking the potential for more profound, latent-level cross-linguistic interactions.
Approach:	They propose a multilingual fine-tuning paradigm that explicitly establishes a cross-lingual connection mechanism at the latent level.
Outcome:	The proposed model outperforms vanilla SFT and offers a strong latent-level alternative to data-level augmentation methods.

EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning for LLMs (2026.findings-acl)

Copied to clipboard

Huanyu Liu, Jia Li, Yihong Dong, Chang Yu, Taozhi Chen, Lecheng Wang, Yongding Tao, Bin Gu, Ge Li

Table-R1: Inference-Time Scaling for Table Reasoning Tasks (2025.emnlp-main)

Challenge:	Existing approaches to reinforcement learning with verifiable reward (RLVR) are limited by difficulty or lack of exploration.
Approach:	They propose a self-evolving curriculum learning framework based on chain-of-thought reasoning optimization that constrains exploration space by self-generating and verifying CoT trajectories.
Outcome:	The proposed framework enables LLMs to solve previously unsolved problems without external supervision and is compatible with various RL fine-tuning methods.

Copied to clipboard

Zheyuan Yang, Lyuhao Chen, Arman Cohan, Yilun Zhao

Structured Data in NLP

Challenge:	In this study, we explore inference-time scaling on table reasoning tasks.
Approach:	They propose a large-scale dataset of reasoning traces and a reinforcement learning with verifiable rewards approach to enable inference-time scaling on table reasoning tasks.
Outcome:	The proposed model matches or exceeds GPT-4.1 and DeepSeek-R1 models on diverse table reasoning tasks.

SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation (2024.emnlp-main)

Copied to clipboard

Abhishek Divekar, Greg Durrett

Information Retrieval

Challenge:	Prior approaches to synthesis use few-shot prompting, which relies on the LLM’s parametric knowledge to generate usable examples.
Approach:	They propose to use a dataset to generate examples of each label from the LLM.
Outcome:	The proposed model significantly improves lexical and semantic diversity, similarity to human-written text, and distillation performance, when compared to 32-shot prompting and four prior approaches.

Selective Knowledge Distillation: Fusing LLM Semantic Strengths with DNN Efficiency for Binary Code Similarity Detection (2026.acl-long)

Copied to clipboard

Shize Zhou, Peiyu Liu, Lirong Fu, Tong Ye, Wenhai Wang

Fast and Effective On-Policy Distillation from Reasoning Prefixes (2026.findings-acl)

Challenge:	BinSKD is a binary code similarity detection technique that can be used in bug detection, patch analysis, and malware detection.
Approach:	They propose to leverage an LLM-based BCSD method as the teacher model and transfer its knowledge of high-level program semantics to various DNN-based student models.
Outcome:	The proposed method yields Recall@1 improvements of 14.5%–91.2% for DNN-based BCSD methods and enables HermesSim to match the teacher’s performance with orders-of-magnitude efficiency.

Copied to clipboard

Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler II, Qian Qian, Gregory D Lyng, Sanjit Singh Batra, Robert E. Tillman

Rationalize and Align: Enhancing Writing Assistance with Rationale via Self-Training for Improved Alignment (2025.findings-acl)

Challenge:	On-policy distillation (OPD) requires expensive on-the-fly sampling of the student policy during training, which substantially increases training cost.
Approach:	They propose to use on-policy distillation to sample trajectories from student model . they propose to terminate the sampling early during distillation .
Outcome:	The proposed method matches the performance of full OPD in long reasoning outputs while reducing training FLOP by 2x–40x.

Copied to clipboard

Hannan Cao, Hai Ye, Hwee Tou Ng

Challenge:	Existing writing assistants rely on supervised fine-tuning to optimize models for multiple revisions.
Approach:	They propose a framework that enhances WA performance with rationale and alignment.
Outcome:	The proposed framework outperforms state-of-the-art WAs and the closed-source GPT-4o by 3.9 and 7.1 points on average across eight well-established writing-related test sets.

KAHAN: Knowledge-Augmented Hierarchical Analysis and Narration for Financial Data Narration (2025.findings-emnlp)

Copied to clipboard

Yajing Yang, Tony Deng, Min-Yen Kan

Cognitive-Uncertainty Guided Knowledge Distillation for Accurate Classification of Student Misconceptions (2026.findings-acl)

Challenge:	KAHAN leverages LLMs as domain experts to drive the analysis.
Approach:	They propose a knowledge-augmented hierarchical framework that extracts insights from raw tabular data.
Outcome:	KAHAN outperforms existing frameworks on financial reporting benchmarks on narrative quality and factuality.

Copied to clipboard

Qirui Liu, Hao Chen, Weijie Shi, Jiajie Xu, Jia Zhu

Information Retrieval

Challenge:	Existing methods for identifying student misconceptions overlook students' reasoning processes, authors report .
Approach:	They propose a knowledge distillation framework that mines high-value samples from existing data.
Outcome:	The proposed framework outperforms sota LLM and standard fine-tuned 72B models on cross-topic tests.

MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation (2026.acl-long)

Copied to clipboard

Pham Khanh Chi, Quoc Phong Dao, Thuat Nguyen, Linh Ngo Van, Trung Le, Thanh Hong Nguyen

Relevance to Utility: Process-Supervised Rewrite for RAG (2026.findings-acl)

Challenge:	Existing methods for reducing the computational cost of large language models (LLMs) focus on minimizing the divergence between the output probability distributions of the teacher and the student, which limits knowledge transfer.
Approach:	They propose a framework that aligns teacher and student representations along their layer-wise transformation trajectory.
Outcome:	The proposed framework outperforms state-of-the-art benchmarks on teacher–student layers.

Copied to clipboard

Jaeyoung Kim, Jongho Kim, Seung-won Hwang, Seoho Song, Young-In Song

Natural Language Interfaces

Challenge:	Existing bridge modules attempt to rewrite documents for better generation, but fail to capture "document utility".
Approach:	They propose to observe reasoning process as process supervision and scale this observation to enhance reliability in distillation.
Outcome:	The proposed method improves over strong bridging baselines on multiple open-domain question-answering benchmarks.

What Deserves Memory: Adaptive Memory Distillation for LLM Agents (2026.acl-long)

Copied to clipboard

Wenquan Ma, Jiayan Nan, WenLong Wu

ThinkSLM: Towards Reasoning in Small Language Models (2025.emnlp-main)

Challenge:	Existing memory systems rely on predefined heuristics such as importance scores, emotional tags, or factual templates, encoding designer intuition rather than learning from the data itself.
Approach:	They propose an adaptive memory distillation framework that casts the assessment of the experience’s future utility as a matter of predictability.
Outcome:	The proposed framework achieves strong performance, efficiency, and storage reduction.

Copied to clipboard

Gaurav Srivastava, Shuxiang Cao, Xuan Wang

MARD: Module-Aware Reasoning Distillation for Language Models with Adaptive Supervision (2026.acl-long)

Challenge:	Reasoning has long been viewed as an emergent property of large language models (LLMs), appearing at or above a certain scale (100B parameters).
Approach:	They propose a benchmark to evaluate the reasoning abilities of small language models (SLMs) using quantization, pruning, and distillation.
Outcome:	The proposed benchmark systematically evaluates reasoning abilities of small language models trained from scratch or derived from LLMs using quantization, pruning, and distillation.

Copied to clipboard

Wenqi Yang, Jianjun Li, Zhibo Zhang, Mingqian Ding, Yushen Fang

CoTEvol: Self-Evolving Chain-of-Thoughts for Data Synthesis in Mathematical Reasoning (2026.findings-acl)

Challenge:	Multi-step reasoning remains challenging for language models with limited capacity . et al., 2025) demonstrate remarkable reasoning capabilities across diverse tasks .
Approach:	They propose a module-aware reasoning distillation framework that explicitly targets key Transformer components for effective reasoning transfer.
Outcome:	The proposed framework targets key components for effective reasoning transfer . it adopts an offline distillation setting, where a strong teacher model provides reasoning trajectories in advance .

Copied to clipboard

Zhuo Wang, Zhuo Zhang, Yafu Li, Yu Cheng, Lizhen Qu, Zenglin Xu

SCURank: Ranking Multiple Candidate Summaries with Summary Content Units for Enhanced Summarization (2026.findings-acl)

Challenge:	Existing methods to improve LLMs' reasoning abilities suffer from diminishing returns or high computing overhead.
Approach:	They propose a genetic evolutionary framework that casts CoT generation as a population-based search over reasoning trajectories.
Outcome:	The proposed framework improves correct-CoT synthesis success by over 30% and enhances structural diversity with markedly improved efficiency.

Copied to clipboard

Bo-Jyun Wang, Ying-Jia Lin, Hung-Yu Kao

Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment (2026.acl-long)

Challenge:	Existing ranking strategies for large language models suffer from instability and lack of information content.
Approach:	They propose a framework that enhances summarization by leveraging Summary Content Units (SCUs) they investigate the effectiveness of SCURank in distilling summaries from multiple LLMs .
Outcome:	The proposed framework outperforms traditional metrics and LLM-based ranking methods in summarization tasks.

Copied to clipboard

Yuming Yang, Mingyoung Lai, Wanxu Zhao, Xiaoran Fan, Zhiheng Xi, Mingqi Wu, Chiyue Huang, Jun Zhao, Haijun Lv, Jian Tong, Yunhua Zhou, Yicheng Zou, Qipeng Guo, Tao Gui, Qi Zhang, Xuanjing Huang