Papers by Penghan Wang
DiSCo: Device-Server Collaborative LLM-based Text Streaming Services (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have introduced significant cost and quality of experience (QoE) challenges in serving millions of daily requests. |
| Approach: | They propose a device-server cooperative scheduler that optimizes users’ QoE by adaptively routing requests and migrating response generation between endpoints while maintaining cost constraints. |
| Outcome: | Evaluations on real-world workloads show that the proposed scheduler can reduce tail TTFT (11-52%) and mean TTTT (6-78%) while maintaining comparable QoE levels. |