| Challenge: | Large language models (LLMs) have been gaining in performance but deployment in edge devices faces significant hurdles due to their high computational complexity. |
| Approach: | They propose a collaborative decoding system that allows small models to perform on-device inference while selectively consulting a cloud-based large model for critical token generation. |
| Outcome: | The proposed system achieves 60% performance gain on CommonsenseQA using a 0.5B model on an M1 MacBook, with under 7% of tokens generation uploaded to the large model in the cloud. |
Similar Papers
Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM (2026.findings-acl)
Copied to clipboard
Sravanth Kodavanti, Sowmya Vajrala, Srinivas Soumitri Miriyala, Utsav Tiwari, Uttam Kumar, Utkarsh Kumar Mahawar, Achal Pratap Singh, Arya D, Narendra Mutyala, Vikram Nelvoy Rajendiran, Sharan Kumar Allur, Euntaik Lee, Dohyoung Kim, HyeonSu Lee, Gyusung Cho, JungBae Kim
| Challenge: | a framework for efficient on-device inference of large language models is needed for smartphones . memory, latency, and runtime flexibility are constraints for large language model deployments. |
| Approach: | They propose a hardware-aware framework for efficient on-device inference of a LLaMA-based multilingual foundation model for Samsung Galaxy S24 and S25 devices with SM8650 and SM8750 chipsets respectively. |
| Outcome: | The proposed framework improves memory, latency and performance across 9 languages and 8 tasks. |
A Drop-In Solution for On-the-Fly Adaptation of Speculative Decoding in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are highly memory-intensive when performing real-time inference. |
| Approach: | They propose a technique that allows for speculative decoding to be run on the fly to maximize the efficiency of LLM inferences. |
| Outcome: | The proposed solution can lead to 3.55-16.48% speed improvement over the standard speculative decoding, and 1.2-3.4 over the default LLMs. |
LinguaLinked: Distributed Large Language Model Inference on Mobile Devices (2024.acl-demos)
Copied to clipboard
| Challenge: | Recent research shows that large language models demonstrate enhanced capabilities in various language tasks. |
| Approach: | They introduce a system for decentralized, distributed LLM inference on mobile devices . they use optimized model assignment technique to segment LLMs and linear optimization to align segments with each device . |
| Outcome: | The proposed system performs well on high-end to low-end Android devices. |
Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference (2025.findings-emnlp)
Copied to clipboard
Hao Mark Chen, Wayne Luk, Yiu Ka Fai Cedric, Rui Li, Konstantin Mishchenko, Stylianos Venieris, Hongxiang Fan
| Challenge: | Auto-regressive decoding of Large Language Models results in significant overheads in hardware performance . a novel parallel prompt decoding approach is proposed to overcome these limitations . |
| Approach: | They propose a parallel prompt decoding that uses a single model for speculation and verification. |
| Outcome: | The proposed approach speeds up auto-regressive decoding of large language models 2.49 times . it can be used on mobileLlama to Vicuna-13B on a wide range of benchmarks . |
Crayon: Customized On-Device LLM via Instant Adapter Blending and Edge-Server Hybrid Inference (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models are highly advanced to user-requested tasks. |
| Approach: | They propose a device-server hybrid inference strategy for on-device LLM customization . they construct a pool of diverse base adapters and then blend them into a customized adapter . |
| Outcome: | The proposed method can be used on a large scale without extra training . it can be applied to large-scale LLMs without sacrificing the benefits of on-device customization. |
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding (2024.findings-acl)
Copied to clipboard
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui
| Challenge: | Large Language Models (LLMs) have a high inference latency stemming from autoregressive decoding. |
| Approach: | They propose a novel decoding paradigm that drafts multiple tokens and verifies them in parallel . they aim to provide a catalyst for further research on Speculative Decoding . |
| Outcome: | The proposed method drafts multiple tokens and verifies them in parallel . it can be used to accelerate inference in large language models. |
Your Inference Request Will Become a Black Box: Confidential Inference for Cloud-based Large Language Models (2026.acl-long)
Copied to clipboard
Chung-ju Huang, Huiqiang Zhao, Yuanpeng He, Lijian Li, Wenpin Jiao, Zhi Jin, Peixuan Chen, Leye Wang
| Challenge: | Existing approaches fail to ensure privacy, maintain model performance, and preserve computational efficiency simultaneously. |
| Approach: | They propose a confidential inference framework that partitions the LLM pipeline between a client-verified Confidential Virtual Machine (CVM) and the public cloud to protect client data without compromising the cloud’s model intellectual property or inference quality. |
| Outcome: | The proposed framework can defend against state-of-the-art token inference attacks while preserving model privacy, performance, and efficiency. |
Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized natural language processing and are limited by high inference time in multilingual settings. |
| Approach: | They propose a training recipe for an assistant model in speculative decoding, which are leveraged to draft and-then its future tokens are verified by the target LLM. |
| Outcome: | The proposed model significantly speeds up inference time and out-of-domain speedup across various languages. |
Petals: Collaborative Inference and Fine-tuning of Large Models (2023.acl-demo)
Copied to clipboard
Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Maksim Riabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, Colin Raffel
| Challenge: | Recent studies show that pretrained language models can solve practical tasks using more than 100 billion parameters. |
| Approach: | They propose a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties. |
| Outcome: | The proposed system outperforms offloading for very large models running on consumer GPUs with 1 step per second, enough for many interactive LLM applications. |
Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems (2025.emnlp-industry)
Copied to clipboard
Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Hejian Sang, Shao Tang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh S. Pillai, Luke Simon, Rahul Mazumder
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications. |
| Approach: | They propose two techniques for training and deploying small language models that deliver high performance for a variety of industry use cases. |
| Outcome: | The proposed techniques retain much of the quality of larger models while reducing training/serving costs and latency. |