FIDELITY: Fine-grained Interpretable Distillation for Effective Language Insights and Topic Yielding (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for topic modeling generate contextually specific and semantically intuitive topics, especially in dynamic environments and low-resource languages. |
| Approach: | They propose a hybrid method that combines topic modeling and text summarization to produce fine-grained, semantically rich, and contextually relevant output. |
| Outcome: | FIDELITY outperforms traditional models in topic diversity, similarity, and ability to process new, unseen documents. |
Similar Papers
Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Tasks (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to evaluate summary faithfulness are sub-optimal due to the granularity level considered for premises and hypotheses. |
| Approach: | They propose a novel approach that uses a variable premise size and simplifies summary sentences into shorter hypotheses. |
| Outcome: | The proposed model performs better on diverse summarisation tasks than existing models. |
EasyDistill: A Comprehensive Toolkit for Effective Knowledge Distillation of Large Language Models (2025.emnlp-demos)
Copied to clipboard
| Challenge: | In this paper, we present EasyDistill, a comprehensive toolkit designed for effective black-box and white-box knowledge distillation (KD) of large language models. |
| Approach: | They propose a toolkit for effective black-box and white-box knowledge distillation (KD) of large language models (LLMs). |
| Outcome: | The framework offers data synthesis, supervised fine-tuning, ranking optimization, and reinforcement learning techniques specifically tailored for KD scenarios. |
Contextualization Distillation from Large Language Model for Knowledge Graph Completion (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing knowledge graph completion models lack textual information, which limits their performance . a plug-in-and-play approach is needed to train small models in descriptive context . |
| Approach: | They propose a plug-in-and-play approach to knowledge graph completion that prompts LLMs to generate descriptive context. |
| Outcome: | The proposed method improves performance on Wikipedia articles and synset definitions. |
Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations (2022.acl-long)
Copied to clipboard
Wei Chen, Yeyun Gong, Can Xu, Huang Hu, Bolun Yao, Zhongyu Wei, Zhihao Fan, Xiaowu Hu, Bartuer Zhou, Biao Cheng, Daxin Jiang, Nan Duan
| Challenge: | Existing studies focus on coarse-grained response selection in retrieval-based dialogue systems. |
| Approach: | They propose a Contextual Fine-to-Coarse (CFC) distilled model for coarse-grained response selection in open-domain conversations. |
| Outcome: | The proposed model improves over baseline methods on two datasets based on the Reddit comments dump and Twitter corpus compared with baseline methods. |
Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model (2024.naacl-long)
Copied to clipboard
Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, Yejin Choi
| Challenge: | Impossible Distillation is a framework for paraphrasing and sentence summarization that can be trained from a low-quality teacher model. |
| Approach: | They propose a framework that distills a high-quality dataset from a low-quality teacher . they hypothesize and verify the paraphrastic proximity intrinsic to pre-trained LMs . |
| Outcome: | The proposed framework outperforms baseline models on unconstrained paraphrase generation and sentence summarization benchmarks. |
LLM-XTM: Enhancing Cross-Lingual Topic Models with Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing cross-lingual topic models depend on sparse bilingual resources and often yield incoherent or weakly aligned topics. |
| Approach: | They propose a framework that integrates LLM-guided topic refinement with self-consistency uncertainty quantification to enable black-box, stable, and scalable enhancement of cross-lingual topic models. |
| Outcome: | Experiments on multilingual corpora show that the proposed framework achieves superior topic coherence and alignment while reducing reliance on bilingual dictionaries and expensive LLM calls. |
Distillation with Explanations from Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) like ChatGPT and GPT-4 have made remarkable progress in various NLP tasks while also providing explanations alongside their answers. |
| Approach: | They propose to use Large language models (LLMs) to generate more accurate answers and corresponding free-text explanations by combining ground truth labels and answers-explanations generated by LLMs. |
| Outcome: | The proposed method achieves improved predictive performance and generates explanations that exhibit greater alignment with the model’s task outputs. |
Comprehensive and Efficient Distillation for Lightweight Sentiment Analysis Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent efforts to develop lightweight and practical sentiment analysis models are limited by manual instruction and large-scale user texts. |
| Approach: | They propose a framework for sentiment analysis that uses attribute-based instruction construction and difficulty-based data filtering to distill knowledge. |
| Outcome: | The proposed framework outperforms baseline methods in data efficiency and performance. |
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |
Improving Neural Topic Models using Knowledge Distillation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Current paradigms for transfer learning use general knowledge as a foundation for more specialized endeavors. |
| Approach: | They propose to combine probabilistic topic models and pretrained transformers to improve topic quality by using knowledge distillation. |
| Outcome: | The proposed framework improves topic quality over all estimated topics and in head-to-head comparisons of aligned topics. |