TurnBack: A Geospatial Route Cognition Benchmark for Large Language Models through Reverse Route (2025.emnlp-main)
Copied to clipboard
Hongyi Luo, Qing Cheng, Daniel Matos, Hari Krishna Gadi, Yanfeng Zhang, Lu Liu, Yongliang Wang, Niclas Zeller, Daniel Cremers, Liqiu Meng
| Challenge: | Existing studies on large language models have limited evaluation of their geospatial cognition . a unified framework for evaluating geospcial cognition in LLMs remains absent . |
| Approach: | They propose a benchmark to evaluate the geospatial route cognition of Large Language Models . they propose 'pathbuilder' tool for converting natural language instructions into navigation routes . |
| Outcome: | The proposed framework and metrics evaluate 9 state-of-the-art LLMs on route reversal task. |
Similar Papers
Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have extensive world knowledge, but often generate inaccurate geospatial knowledge. |
| Approach: | They propose a framework for evaluation of large language models to mitigate hallucinations . they use Kahneman-Tversky Optimization to align LLMs with their reality . |
| Outcome: | The proposed evaluation framework uncovers hallucinations in 20 advanced LLMs. |
TravelBehaviorQA: A Benchmark Dataset for Behavioral Interpretation of GPS Trajectories (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have enabled strong performance on reasoning, summarization, and dialogue tasks across diverse domains. |
| Approach: | They propose a large-scale benchmark dataset that reframes trajectory analysis as a language-based understanding task. |
| Outcome: | The proposed dataset compares GPS trajectories with human-grounded question-answering (QA) pairs. |
Reverse Modeling in Large Language Models (2025.naacl-short)
Copied to clipboard
| Challenge: | Using pre-trained LLMs with reversed text inputs can improve their performance across multiple languages. |
| Approach: | They propose a way to determine whether LLMs can understand reversed text inputs by reversing entire paragraphs or documents at the token level. |
| Outcome: | The proposed model can be used to improve understanding across multiple languages. |
RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a lack of comprehensive benchmarks for Routing large language models has hindered the development of routers. |
| Approach: | They propose a router-based benchmark to evaluate Routing large language models . the benchmark includes performance records for 12 popular LLM evaluations . |
| Outcome: | The proposed model-level scaling up phenomenon can surpass the best single model in the pool and many existing strong LLMs. |
UrbanGeoEval: A City-Scale Benchmark for Evaluating Large Language Models in Geospatial Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Extensive experiments on 18 widely used LLMs uncover critical insights: (1) models exhibit severe geographic biases and resolution gaps; (2) failures in complex multi-hop tasks stem from brittle foundational spatial skills rather than high-level logic deficits. |
| Approach: | They propose a dual-module framework that disentangles factual recall and spatial logic from the model's real capabilities in urban environments. |
| Outcome: | Extensive tests on 18 widely used LLMs reveal that models exhibit severe geographic biases and resolution gaps, and failures in complex multi-hop tasks often stem from brittle foundational spatial skills rather than high-level logic deficits. |
ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Recent work in language modeling has led to effective SLMs with impressive performance levels across various benchmarks. |
| Approach: | They propose a benchmark that introduces process-level evaluation for commonsense reasoning tasks. |
| Outcome: | The proposed benchmarks show that large language models provide correct answers despite flawed reasoning processes in a substantial portion of cases. |
LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing (2026.findings-acl)
Copied to clipboard
Hao Li, Yiqun Zhang, Zhaoyan Guo, Chenxu Wang, Shengji Tang, Qiaosheng Zhang, Yang Chen, Biqing Qi, Peng Ye, Lei Bai, Zhen Wang, Shuyue Hu
| Challenge: | Large language model (LLM) routing assigns each query to the best suitable model from an ensemble. |
| Approach: | They introduce a large-scale benchmark and unified framework for LLM routing . they find that many routing methods exhibit similar performance under unified evaluation . |
| Outcome: | The proposed benchmark provides comprehensive metrics for both performance-oriented and performance-cost trade-off routing. |
IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models have demonstrated exceptional performance across a wide range of tasks . however, selecting the optimal LLM to respond to a user query often necessitates a delicate balance between performance and cost. |
| Approach: | They propose a multi-LLM routing framework that efficiently routes user queries to the most suitable LLM. |
| Outcome: | The proposed framework outperforms baseline methods in terms of effectiveness and interpretability. |
How Controllable Are Large Language Models? A Unified Evaluation across Behavioral Granularities (2026.acl-long)
Copied to clipboard
Ziwen Xu, Kewei Xu, Haoming Xu, Haiwen Hong, Longtao Huang, Hui Xue, Ningyu Zhang, Yongliang Shen, Guozhou Zheng, Huajun Chen, Shumin Deng
| Challenge: | Large language models (LLMs) are increasingly deployed in socially sensitive domains, yet their unpredictable behaviors pose significant risks. |
| Approach: | They propose a hierarchical benchmark for evaluating LLM controllability across three domains: language features, sentiment, and personality. |
| Outcome: | The proposed framework offers a principled and interpretable framework for safe and controllable LLM behavior serving as a foundation for future research. |
TripTailor: A Real-World Benchmark for Personalized Travel Planning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics for travel planning rely on unrealistic simulated data . fewer than 10% of the itineraries generated by the latest state-of-the-art LLMs achieve human-level performance. |
| Approach: | They propose a benchmark for personalized travel planning in real-world scenarios . they identify several critical challenges in travel planning including feasibility and rationality . |
| Outcome: | The proposed benchmarks show that fewer than 10% of the itineraries generated by the latest state-of-the-art LLMs achieve human-level performance. |