LoCar: Localization-Aware Evaluation of In-Vehicle Assistants through Fine-Grained Sociolinguistic Control (2026.acl-industry)
Copied to clipboard
| Challenge: | Using Large Language Models (LLMs) is challenging due to lack of domain-specific evaluation standards . current LLMs prioritize reasoning or knowledge over sociolinguistic nuances vital for automotive settings . |
| Approach: | They propose a framework for evaluation of Korean-language in-vehicle assistants . they propose to evaluate fine-grained Korean honorific control and safetycritical response behavior . |
| Outcome: | The proposed evaluation framework evaluates fine-grained honorific control, safetycritical response behavior, and task efficiency in deployment-aligned settings. |
Similar Papers
Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advances, Resources, and Future Directions (2025.findings-emnlp)
Copied to clipboard
Yaozu Wu, Dongyuan Li, Yankai Chen, Renhe Jiang, Henry Peng Zou, Wei-Chieh Huang, Yangning Li, Liancheng Fang, Zhen Wang, Philip S. Yu
| Challenge: | Large Language Models (LLMs) are used to assist with driving decisions, but they face limitations in perception and computational demands. |
| Approach: | They propose a survey of LLM-based multi-agent ADSs and their applications . they analyze agent-human interactions in scenarios where LLM agents engage with humans . |
| Outcome: | The proposed approach reduces human intervention and improves safety and efficiency. |
CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for Large Language Model (LLM) agents focus on task completion under idealistic settings but overlook reliability in real-world, user-facing applications. |
| Approach: | They propose a benchmark to evaluate consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. |
| Outcome: | The proposed benchmarks evaluate consistency, uncertainty handling, and capability awareness in multi-turn, tool-using LLM agents in an in-car assistant domain. |
GEAR: A Scalable and Interpretable Evaluation Framework for RAG-Based Car Assistant Systems (2025.emnlp-industry)
Copied to clipboard
Niloufar Beyranvand, Hamidreza Dastmalchi, Aijun An, Heidar Davoudi, Winston Chan, Ron DiCarlantonio
| Challenge: | Large language models (LLMs) increasingly power car assistants, but evaluating response quality remains a challenge. |
| Approach: | They propose a framework that uses large language models as evaluators to compare assistant responses against ground-truth counterparts. |
| Outcome: | The proposed framework compares assistant responses against ground-truth counterparts, assessing coverage, correctness, and other dimensions of answer quality. |
Spoken Conversational Agents with Large Language Models (2025.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial focuses on the evolution of voice-native LLMs . it reviews the adaptation of text LLM to audio, cross-modal alignment, and joint speech–text training . |
| Approach: | This tutorial examines the evolution of voice-native LLMs in conversational agents . it compares cascaded and voice-based LLM systems to end-to-end retrieval-and vision-grounded systems . |
| Outcome: | This tutorial examines the evolution of voice-native LLMs . it compares the performance of voice assistants to current open-domain agents . |
Toward Beginner-Friendly LLMs for Language Learning: Controlling Difficulty in Conversation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Practicing conversations with large language models is a promising alternative to traditional in-person language learning. |
| Approach: | They propose a new token-level evaluation metric, Token Miss Rate, that measures the proportion of incomprehensible tokens per utterance and correlates strongly with human judgments. |
| Outcome: | The proposed methods improve comprehensibility for beginner speakers from 39.4% to 83.3%, compared with prompting alone and a token-level evaluation metric, Token Miss Rate (TMR). |
FAC2E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are evaluated by overall performance on various text understanding and generation tasks. |
| Approach: | They propose a framework for Fine-grAined and Cognition-grounded LLMs’ Capability Evaluation that dissociates the language-related capabilities from cognition-related ones. |
| Outcome: | The proposed framework dissociates the language-related capabilities from cognition-related ones and breaks down the process of applying a specific capability into three sub-steps: recalling relevant knowledge, utilizing knowledge, and solving problems. |
Evaluating Conversational Agents with Persona-driven User Simulations based on Large Language Models: A Sales Bot Case Study (2025.emnlp-industry)
Copied to clipboard
Justyna Gromada, Alicja Kasicka, Ewa Komkowska, Lukasz Krajewski, Natalia Krawczyk, Morgan Veyret, Bartosz Przybył, Lina M. Rojas-Barahona, Michał K. Szczerbak
| Challenge: | Recent advances in LLMs enable sophisticated user simulations that can replace traditional rule-based evaluations. |
| Approach: | They propose a persona-driven approach to conversational agent evaluation using Large Language Models (LLMs) they introduce a dataset of customer personas, which are then used to configure a single LLM-based user simulator. |
| Outcome: | The proposed model emulates nuanced customer roles and can implement cross-selling strategies with minimal impact on customer satisfaction, varying by customer type. |
Exploring the Potential of LLMs as Personalized Assistants: Dataset, Evaluation, and Analysis (2025.acl-long)
Copied to clipboard
| Challenge: | Personalized AI assistants are a challenging application that intertwines multiple problems in LLM research. |
| Approach: | They propose a Llama-3.2-based automated evaluation model that matches human preferences to a conversational dataset. |
| Outcome: | HiCUPID provides a conversational dataset tailored for personalization . the evaluation model closely mirrors human preferences, the researchers show . |
Harnessing and Evaluating the Intrinsic Extrapolation Ability of Large Language Models for Vehicle Trajectory Prediction (2025.naacl-long)
Copied to clipboard
| Challenge: | Emergent abilities of large language models (LLMs) have advanced their application in autonomous vehicle research. |
| Approach: | They propose a framework that leverages LLMs’ built-in extrapolation capabilities for vehicle trajectory prediction, enabling them to understand traffic agents' behavior and interactions over time. |
| Outcome: | The proposed framework enables off-the-shelf, frozen LLMs to achieve competitive trajectory prediction performance with qualitative analyses revealing their enhanced understanding of complex, multi-agent traffic scenarios. |
AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments (2025.acl-long)
Copied to clipboard
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang, Chenyang Liao, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang
| Challenge: | Large language models (LLMs) are promising foundations to build generally-capable agents . however, the community lacks a unified interactive framework that covers diverse environments for comprehensive evaluation of agents. |
| Approach: | They propose a framework that features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |
| Outcome: | The proposed framework features 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction. |