Papers with sLLMs
RRADistill: Distilling LLMs’ Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Engine (2024.emnlp-industry)
Copied to clipboard
Nayoung Choi, Youngjune Lee, Gyu-Hwung Cho, Haeyu Jeong, Jungmin Kong, Saehun Kim, Keunchan Park, Sarah Cho, Inchang Jeong, Gyohee Nam, Sunghoon Han, Wonil Yang, Jaeho Choi
| Challenge: | Large Language Models excel at understanding the semantic relationships between queries and documents, even with lengthy and complex long-tail queries. |
| Approach: | They propose an efficient label generation pipeline and novel sLLM training methods for both encoder and decoder models. |
| Outcome: | The proposed method improves re-ranking for long-tail queries on a Korean-based search platform. |
Aggregating Crowd of LLMs for Cost-Effective Data Annotation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have shown promise for automated data annotation, yet reliance on expensive commercial models like GPT-4 limits accessibility. |
| Approach: | They propose to build a crowd of LLMs which aggregates annotations from multiple sLLMs using label aggregation algorithms. |
| Outcome: | The proposed approach outperforms individual sLLMs and human crowd labels yields superior results compared to either method alone. |
ixi-GEN: Efficient Industrial sLLMs through Domain Adaptive Continual Pretraining (2025.emnlp-industry)
Copied to clipboard
Seonwu Kim, Yohan Na, Kihun Kim, Hanhee Cho, Geun Lim, Mintae Kim, Seongik Park, Ki Hyun Kim, Youngsub Han, Byoung-Ki Jeon
| Challenge: | Domain Adaptive Continual Pretraining (DACP) is a method to mitigate performance degradation in small LLMs and enhance their effectiveness in target domains. |
| Approach: | They propose a continual pretraining methodology that optimizes sLLMs within service domains and enhances their effectiveness in target domains. |
| Outcome: | The proposed model achieves significant gains in target-domain performance while preserving general capabilities, offering a cost-efficient and scalable solution for enterprise-level deployment. |
Investigating Value-Reasoning Reliability in Small Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | sLLMs have been widely deployed in practical applications, but little attention has been paid to their value-reasoning abilities, particularly in terms of reasoning reliability. |
| Approach: | They propose a systematic evaluation framework for assessing the Value-Reasoning Reliability of small Large Language models (sLLMs) . framework includes three core tasks: Repetition Consistency task, Interaction Stability task, and Open-ended Expression Consistencies task. |
| Outcome: | The proposed framework incorporates self-reported confidence scores to evaluate the model’s value reasoning reliability from two perspectives: the model's self awareness of its values, and its value-based decision-making. |
EnSToM: Enhancing Dialogue Systems with Entropy-Scaled Steering Vectors for Topic Maintenance (2025.findings-acl)
Copied to clipboard
| Challenge: | Small large language models (sLLMs) are lightweight and efficient, but struggle to maintain topic consistency in task-oriented dialogue systems. |
| Approach: | They propose an approach to ensure topic consistency in task-oriented dialogue systems by manipulating internal activations during inference. |
| Outcome: | The proposed approach achieves significant performance gain with a relatively small data size compared to fine-tuning approaches. |