Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Prompting Large Language Models (LLMs) performs impressively in zero- and few-shot settings. |
| Approach: | They propose a framework that allows reducing calls to LLMs by caching previous LLM responses and using them to train a local inexpensive model on the SME side. |
| Outcome: | The proposed framework reduces calls to LLMs by caching previous LLM responses and using them to train a local inexpensive model on the SME side. |
Similar Papers
Cache Saver: A Modular Framework for Efficient, Affordable, and Reproducible LLM Inference (2025.findings-emnlp)
Copied to clipboard
Nearchos Potamitis, Lars Henning Klein, Bardia Mohammadi, Chongyang Xu, Attreyee Mukherjee, Niket Tandon, Laurent Bindschaedler, Akhil Arora
| Challenge: | Inference is the major cost throughout the lifecycle of a large language model (LLM). |
| Approach: | They propose a plug-and-play, asynchronous framework that facilitates high-level inference optimizations. |
| Outcome: | The proposed framework reduces cost and CO2 by 35% across methods, tasks, and LLMs. |
Cache & Distil: Optimising API Calls to Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models are expensive to run and expose the entire request stream to external providers. |
| Approach: | They propose to locally train a small private language model on the LLM's predictions to minimise the costs and data exposure associated with calling the API. |
| Outcome: | The proposed model can handle an increasing number of user requests independently and is able to perform better than other policies and baselines across tasks and budgets. |
Cost-Performance Optimization for Processing Low-Resource Language Tasks Using Commercial LLMs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit impressive zero/few-shot inference and generation quality for high-resource languages (HRLs). |
| Approach: | They propose to reduce the cost of processing LRLs by code-mixing, translation, and transliteration of LRL to HRLs to ensure that predictive and generative qualities are not compromised. |
| Outcome: | The proposed model reduces the cost of processing LRLs while ensuring that predictive and generative qualities are not compromised. |
Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems (2025.emnlp-industry)
Copied to clipboard
Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Hejian Sang, Shao Tang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh S. Pillai, Luke Simon, Rahul Mazumder
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications. |
| Approach: | They propose two techniques for training and deploying small language models that deliver high performance for a variety of industry use cases. |
| Outcome: | The proposed techniques retain much of the quality of larger models while reducing training/serving costs and latency. |
Decoding with Limited Teacher Supervision Requires Understanding When to Trust the Teacher (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated their tremendous capability to generate human-like text sentences that convey rich knowledge in various problem domains. |
| Approach: | They propose an algorithm to aggregate small-scale LLM and LLM predictions on initial tokens so that the generated tokens can more accurately condition the subsequent token generation by small-level LLM only. |
| Outcome: | The proposed method improves on the limited supervision scenario on a wide range of models and datasets. |
Unlearn What You Want to Forget: Efficient Unlearning for LLMs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be used to memorize a vast amount of data, but can suffer from privacy issues and data protection violations. |
| Approach: | They propose an efficient unlearning framework that could update LLMs without retraining them . they introduce lightweight unlearning layers learned with a selective teacher-student objective into transformers . |
| Outcome: | The proposed framework could update LLMs without having to retrain the whole model after data removals. |
Confidence-Calibrated Small-Large Language Model Collaboration for Cost-Efficient Reasoning (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have superior reasoning capabilities compared to small language models, but incur substantially higher inference costs. |
| Approach: | They propose a system that cascades an LLM with an SLM to achieve a balance between accuracy and cost in complex reasoning tasks. |
| Outcome: | The proposed system improves the SLM’s reasoning ability and confidence calibration across diverse datasets and model backbones. |
Learning with Less: Knowledge Distillation from Large Language Models via Unlabeled Data (2025.findings-naacl)
Copied to clipboard
Juanhui Li, Sreyashi Nag, Hui Liu, Xianfeng Tang, Sheikh Muhammad Sarwar, Limeng Cui, Hansu Gu, Suhang Wang, Qi He, Jiliang Tang
| Challenge: | Large Language Models (LLMs) have demonstrated superior language understanding abilities in many real-world NLP applications. |
| Approach: | They propose a learning-based sample selection method that incorporates signals from both teacher and student to enhance model performance. |
| Outcome: | The proposed method improves model performance across datasets with higher data efficiency. |
Soft Prompting for Unlearning in Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing ethical and safety considerations for large language models are important for deployment . however, some ethical concerns have been raised due to the presence of private, sensitive, or harmful information in the training data. |
| Approach: | They propose a framework that learns prompt tokens that are prepended to a query to induce unlearning in LLMs. |
| Outcome: | The proposed method improves the trade-off between utility and forgetting for text classification and question-answering. |
Teaching Language Models to Self-Improve through Interactive Demonstrations (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been shown to improve performance on downstream tasks by prompting them to analyze and revise their outputs. |
| Approach: | They propose a training algorithm that prompts large language models to analyze and revise their own outputs and uses this feedback to train the small model. |
| Outcome: | The proposed approach improves LLaMA-7B's performance on math and reasoning tasks by up to 7.13%. |