PEC-Home: Interpretation of Progressively Elliptical Commands in Smart Homes (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing home assistants struggle to interpret elliptical commands based on ellipine expressions . current assistants overlook the progressive omission that occurs in human dialogue as context accumulates - limiting their effectiveness in real-world applications . |
| Approach: | They propose a simulated home dataset specifically designed for interpreting progressively elliptical commands in smart homes. |
| Outcome: | The proposed dataset shows that existing home assistants struggle to execute user-intended operations based solely on elliptical commands. |
Similar Papers
HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices (2025.acl-long)
Copied to clipboard
| Challenge: | Existing state-of-the-art LLMs cannot perform well in situations where instructions are invalid or multiple devices are involved. |
| Approach: | They propose to integrate large language models into smart home assistants by enhancing their ability to accurately understand user needs and respond appropriately. |
| Outcome: | The proposed dataset is the first with valid and invalid instructions across devices . it achieves only 0.0% success rate in the scenario of invalid multi-device instructions . |
We Understand Elliptical Sentences, and Language Models should Too: A New Dataset for Studying Ellipsis and its Interaction with Thematic Fit (2023.acl-long)
Copied to clipboard
| Challenge: | ellipsis is a linguistic phenomenon characterized by the omission of one or more sentence elements. |
| Approach: | They investigated how prototypicality affects the ability of Language Models to handle elliptical sentences . they found that models were better suited to evaluating argument thematic fit . |
| Outcome: | The proposed dataset shows that the models perform better for typical events than for atypical ones in different elliptical contexts. |
Parse Me if You Can: Artificial Treebanks for Parsing Experiments on Elliptical Constructions (L18-1)
Copied to clipboard
| Challenge: | ellipsis is a phenomenon present in many natural languages, but it complicates syntactic parsing of the content that is not omitted. |
| Approach: | They analyze outputs of state-of-the-art parsers to learn about parsing accuracy and typical errors from the perspective of elliptical constructions. |
| Outcome: | The proposed treebank is a semi-artificially constructed treebank of ellipsis. |
GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing LLM leaderboards often reference scores reported in other papers without consistent settings and prompts, which may encourage cherry-picking favored settings and for better results. |
| Approach: | They propose an open-source and reproducible LLM evaluation suite built on top of OpenAI Evals that systematically evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
| Outcome: | The evaluation suite is built on top of OpenAI Evals and evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents (2025.acl-long)
Copied to clipboard
Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang
| Challenge: | Multimodal large language models (MLLMs) have enabled LLM-based agents to directly interact with application user interfaces (UIs), enhancing agents’ performance in complex tasks. |
| Approach: | They propose a novel agent framework that prioritizes actions through application programming interfaces over UI actions and facilitates the creation and expansion of APIs through automated exploration of applications. |
| Outcome: | The proposed framework reduces task completion time by 65%-70% and cognitive workload by 38%-53% while maintaining accuracy of 97%-98% compared to humans. |
Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework (2026.eacl-long)
Copied to clipboard
Clea Chataigner, Rebecca Ma, Prakhar Ganesh, Yuhao Chen, Afaf Taik, Elliot Creager, Golnoosh Farnadi
| Challenge: | Existing studies have studied prompt sensitivity by altering formatting or generating paraphrases with automated techniques. |
| Approach: | They propose a framework for generating controlled paraphrases grounded in user behaviors . they leverage linguistically informed rules and enforce quality through checks on instruction adherence . |
| Outcome: | The proposed framework is able to detect weaknesses in large language models . it leverages linguistically informed rules and enforces quality through checks on instruction adherence, semantic similarity, and realism. |
xLAM: A Family of Large Action Models to Empower AI Agent Systems (2025.naacl-long)
Copied to clipboard
Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Quoc Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao Feng, Tulika Manoj Awalgaonkar, Rithesh R N, Zeyuan Chen, Ran Xu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, Caiming Xiong
| Challenge: | Autonomous agents powered by large language models (LLMs) have attracted significant research interest, but there are few standards for developing specialized models for agent tasks. |
| Approach: | They propose a series of large action models with dense and mixture-of-expert architectures that unifies, augments, and synthesizes diverse datasets to enhance agent generalizability and performance. |
| Outcome: | The proposed models outperform GPT-4, Claude-3, and many other models in terms of tool use and outperformed GPT-based models on multiple agent ability benchmarks. |
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (2024.emnlp-demo)
Copied to clipboard
| Challenge: | EMNLP 2024 conference on Empirical Methods in Natural Language Processing received 153 submissions . 52 submissions were selected for inclusion in the program (acceptance rate of 34%) |
| Approach: | EMNLP 2024 conference on Empirical Methods in Natural Language Processing will take place in london on november 12-16, 2024 . |
| Outcome: | The EMNLP 2024 conference is a hybrid event with demonstration papers presented through pre-recorded talks and in presence during the poster sessions. |
Un-considering Contextual Information: Assessing LLMs’ Understanding of Indexical Elements (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in coreference resolution tasks, but previous studies only assessed performance with nouns and third person pronouns. |
| Approach: | They evaluate LLMs' performance on coreference resolution with indexicals like I, you, here and tomorrow which come with unique challenges due to their linguistic properties. |
| Outcome: | The proposed models perform well with some indexicals while struggling with others. |
TOAD: Task-Oriented Automatic Dialogs with Diverse Response Styles (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets for Task-Oriented Dialogs (TOD) lack consideration for adaptive response styles and neglect to simulate interactions with app contexts like calendars or alarms. |
| Approach: | They propose to generate an annotated task-oriented dialog dataset and an automatic pipeline to generate it. |
| Outcome: | The proposed dataset provides a variety of system response styles and provides verbose or non-verbal responses. |