An Active Learning Framework for Inclusive Generation by Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit bias toward underrepresented groups, despite advances in active learning. |
| Approach: | They propose a clustering-based active learning framework enhanced with knowledge distillation that transforms the intermediate outputs of the learner model to yield more representative models without prior knowledge of underlying data distribution. |
| Outcome: | The proposed framework improves performance across data subgroups and lexical diversity, underscoring the model’s resilience to skewness in available data. |
Similar Papers
Evolving Knowledge Distillation with Large Language Models and Active Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the direct use of large language models for text generation and labeling, without fully exploring their potential to comprehend the target task and acquire valuable knowledge. |
| Approach: | They propose to distill the knowledge of large language models into smaller models by generating annotated data. |
| Outcome: | The proposed method improves the performance of small domain models while enhancing the ability of large language models. |
Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings (D19-61)
Copied to clipboard
| Challenge: | Recent research points to knowledge distillation as a potential solution for NLU tasks. |
| Approach: | They propose a training approach that distills large finetuned LMs into a small network using unlabeled training examples. |
| Outcome: | The proposed approach outperforms BERT training approaches while using 300 times fewer parameters. |
From Selection to Generation: A Survey of LLM-based Active Learning (2025.acl-long)
Copied to clipboard
Yu Xia, Subhojyoti Mukherjee, Zhouhang Xie, Junda Wu, Xintong Li, Ryan Aponte, Hanjia Lyu, Joe Barrow, Hongjie Chen, Franck Dernoncourt, Branislav Kveton, Tong Yu, Ruiyi Zhang, Jiuxiang Gu, Nesreen K. Ahmed, Yu Wang, Xiang Chen, Hanieh Deilamsalehy, Sungchul Kim, Zhengmian Hu, Yue Zhao, Nedim Lipka, Seunghyun Yoon, Ting-Hao Kenneth Huang, Zichao Wang, Puneet Mathur, Soumyabrata Pal, Koyel Mukherjee, Zhehao Zhang, Namyong Park, Thien Huu Nguyen, Jiebo Luo, Ryan A. Rossi, Julian McAuley
| Challenge: | Large Language Models (LLMs) have been used for selection and training of data for active learning. |
| Approach: | They propose an intuitive taxonomy that categorizes LLM-based active learning techniques and discuss the transformative roles they can play in the active learning loop. |
| Outcome: | The proposed model can generate entirely new data instances and provide more cost-effective annotations with fewer labeled data instances. |
Multi-Sense Embeddings for Language Models and Knowledge Distillation (2025.findings-acl)
Copied to clipboard
| Challenge: | Transformer-based large language models generate different representations for the same token depending on context . however, words and tokens typically have only a limited number of senses . a knowledge distillation method can be used to learn a smaller student model . |
| Approach: | They propose a multi-sense embedding method that uses a clustering algorithm to generate a sense embeddable dictionary. |
| Outcome: | The proposed method offers significant space and inference time savings while maintaining competitive performance. |
Distilling Rule-based Knowledge into Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have broadened their applicability across diverse realworld scenarios. |
| Approach: | They propose to encode rule-based knowledge into large language models by using strong in-context abilities to extract the knowledge from the textual rules and then explicitly encode it into the parameters of LLMs. |
| Outcome: | The proposed learning paradigm is much more efficient than example-based learning in both sample size and generalization ability. |
Improving Diversity of Commonsense Generation by Large Language Models via In-Context Learning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown proficiency in enhancing the generation quality across various tasks without the need for any fine-tuning. |
| Approach: | They propose a method that diversifies the LLM generations while preserving their quality. |
| Outcome: | The proposed method can be used as training data to improve diversity in existing commonsense generators. |
Teaching Small Language Models Reasoning through Counterfactual Distillation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance in a wide range of downstream tasks. |
| Approach: | They propose a counterfactual distillation framework that leverages LLMs to generate high-quality counterfacts and utilizes multi-view CoT to enhance the diversity of reasoning samples. |
| Outcome: | The proposed framework enhances reasoning capabilities of large language models and is more robust to OOD data. |
ELAD: Explanation-Guided Large Language Models Active Distillation (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are hindered by their memory inefficiency, computational demands, and the high costs of API inferences. |
| Approach: | They propose an Explanation-Guided LLMs Active Distillation framework that employs an active learning strategy to optimize the balance between annotation costs and model performance. |
| Outcome: | The proposed framework significantly improves the efficiency of LLMs knowledge distillation. |
Natural Language Generation for Effective Knowledge Distillation (D19-61)
Copied to clipboard
| Challenge: | Knowledge distillation can transfer knowledge from deep language representation models to shallow word embedding-based neural networks. |
| Approach: | They propose to build an unlabeled transfer dataset to enable effective knowledge transfer . they hypothesize that this principled, general approach outperforms rule-based techniques . |
| Outcome: | The proposed method outperforms OpenAI GPT on four datasets in sentiment classification, sentence similarity, and linguistic acceptability. |
Contextualization Distillation from Large Language Model for Knowledge Graph Completion (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing knowledge graph completion models lack textual information, which limits their performance . a plug-in-and-play approach is needed to train small models in descriptive context . |
| Approach: | They propose a plug-in-and-play approach to knowledge graph completion that prompts LLMs to generate descriptive context. |
| Outcome: | The proposed method improves performance on Wikipedia articles and synset definitions. |