MTRouter: Cost-Aware Multi-Turn LLM Routing with History–Model Joint Embeddings (2026.acl-long)
Copied to clipboard
| Challenge: | Multi-turn, long-horizon tasks require dozens of sequential model calls per episode. |
| Approach: | They propose a cost-aware multi-turn LLM routing tool which encodes interaction history and candidate models into joint history–model embeddings and learns an outcome estimator from logged trajectories to predict turn-level model utility. |
| Outcome: | The proposed model reduces cost and performance by 58.7% on ScienceWorld and on Humanity’s Last Exam (HLE) and even reduces costs for held-out tasks. |
Similar Papers
MasRouter: Learning to Route LLMs for Multi-Agent Systems (2025.acl-long)
Copied to clipboard
| Challenge: | Multi-agent systems (MAS) powered by Large Language Models (LLMs) have been demonstrated to push the boundaries of LLM capabilities, yet they often face significant costs and challenges in dynamic LLM selection. |
| Approach: | They propose a multi-agent system routing solution that integrates all components of MAS into a unified routing framework. |
| Outcome: | The proposed solution is high-performing, cost-effective, and efficient . it reduces overhead by up to 52.07 compared to current methods on HumanEval . |
Generalising LLM Routing using Past Performance Retrieval: A Few-Shot Router is Sufficient (2026.eacl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) use a single LLM to perform tasks. |
| Approach: | They propose a meta-evaluation framework that predicts per-model performance for new queries by retrieving similar past queries and reweighting model scores with lightweight attention. |
| Outcome: | The proposed framework matches the quality–cost trade-offs of generalisable routers across five routing benchmarks. |
RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment (2026.acl-industry)
Copied to clipboard
Yingfeng Luo, Hongyu Liu, DingYang Lin, Kaiyan Chang, Chenglong Wang, Bei Li, Quan Du, Tong Xiao, JingBo Zhu
| Challenge: | Existing routing strategies rely on heuristics, external predictors, or absolute quality estimation to capture whether the large model provides a worthwhile improvement over the small one. |
| Approach: | They propose a budget allocation problem for routing large model to large model . they propose heuristics, external predictors, or absolute quality estimation to determine the optimal signal for budgeted decisions. |
| Outcome: | The proposed model outperforms heuristics, quality/difficulty estimation baselines and achieves a superior quality–budget Pareto frontier. |
MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models (2024.emnlp-main)
Copied to clipboard
Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong
| Challenge: | Existing evaluation frameworks focus on single-turn evaluations, overlooking the models’ capabilities in multi-turn interactions. |
| Approach: | They propose a benchmark to evaluate the multi-turn conversational abilities of large language models (LLMs) by analyzing human-LLM conversations and constructing multi-turned queries for each category using GPT-4. |
| Outcome: | The proposed model outperforms open-source models in multi-turn tasks while retaining and recalling historical information. |
MT-OSC: Path for LLMs that Get Lost in Multi-Turn Conversation (2026.findings-acl)
Copied to clipboard
Jyotika Singh, Fang Tu, Miguel Ballesteros, Weiyi Sun, Sandip Ghoshal, Michelle Yuan, Yassine Benajiba, Sujith Ravi, Dan Roth
| Challenge: | Large language models suffer performance degradation when user instructions and context are distributed over multiple conversational turns. |
| Approach: | They propose a framework that condenses chat history in the background without disrupting the user experience. |
| Outcome: | The proposed framework reduces token counts by up to 72% in 10-turn dialogues while remaining robust to distractors and irrelevant turns. |
TagRouter: Learning Route to LLMs through Tags for Open-Domain Text Generation Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing models with limited performance and limited training can be difficult to use in large-scale applications. |
| Approach: | They propose a training-free model routing method that optimizes synergy among multiple LLMs for open-domain text generation tasks. |
| Outcome: | The proposed method outperforms 13 baseline models and reduces costs by 17.20%. |
TensorOpera Router: A Multi-Model Router for Efficient LLM Inference (2024.emnlp-industry)
Copied to clipboard
Dimitris Stripelis, Zhaozhuo Xu, Zijian Hu, Alay Shah, Han Jin, Yuhang Yao, Jipeng Zhang, Tong Zhang, Salman Avestimehr, Chaoyang He
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable performance across a diverse set of domain-specific tasks. |
| Approach: | They propose a non-monolithic LLM querying system that seamlessly integrates various LLM experts into a single query interface and dynamically routes incoming queries to the most high-performant expert based on query’s requirements. |
| Outcome: | The proposed model improves query efficiency by 40% and costs by 30% while maintaining or enhancing model performance by 10%. |
MixLLM: Dynamic Routing in Mixed Large Language Models (2025.naacl-long)
Copied to clipboard
Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, Haifeng Chen
| Challenge: | Large Language Models (LLMs) exhibit potential artificial generic intelligence, however, their usage is costly with high response latency. |
| Approach: | They develop a dynamic contextual-bandit-based routing system for query-LLM assignment that leverages query tags to enhance query embeddings. |
| Outcome: | The proposed model maximizes response quality and minimizes cost and latency. |
LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing (2026.findings-acl)
Copied to clipboard
Hao Li, Yiqun Zhang, Zhaoyan Guo, Chenxu Wang, Shengji Tang, Qiaosheng Zhang, Yang Chen, Biqing Qi, Peng Ye, Lei Bai, Zhen Wang, Shuyue Hu
| Challenge: | Large language model (LLM) routing assigns each query to the best suitable model from an ensemble. |
| Approach: | They introduce a large-scale benchmark and unified framework for LLM routing . they find that many routing methods exhibit similar performance under unified evaluation . |
| Outcome: | The proposed benchmark provides comprehensive metrics for both performance-oriented and performance-cost trade-off routing. |
StitchLLM: Serving LLMs, One Block at a Time (2025.acl-long)
Copied to clipboard
Bodun Hu, Shuozhe Li, Saurabh Agarwal, Myungjin Lee, Akshay Jajoo, Jiamin Li, Le Xu, Geon-Woo Kim, Donghyun Kim, Hong Xu, Amy Zhang, Aditya Akella
| Challenge: | Existing techniques like distillation and pruning are not efficient for large language models. |
| Approach: | They propose a dynamic model routing framework that uses a powerful bottom model to process all queries and a lightweight routing mechanism to allocate computational resources appropriately. |
| Outcome: | The proposed framework improves system throughput while minimizing performance degradation. |