| Challenge: | Large Language Models (LLMs) are used for behavior planning given natural language instructions from the user. |
| Approach: | They propose to use a textual dataset of ambiguous instructions addressed to a robot in a kitchen environment to compare them. |
| Outcome: | The proposed dataset includes 1000 pairs of ambiguous tasks and their unambiguous counterparts, with environment descriptions, clarifying questions and answers, user intents, and task plans. |
Similar Papers
AmbigNLG: Addressing Task Ambiguity in Instruction for NLG (2024.emnlp-main)
Copied to clipboard
| Challenge: | AmbigNLG is a novel task designed to tackle task ambiguity in instructions for NLG . ambiguous instructions often impede the performance of Large Language Models (LLMs) . |
| Approach: | They propose an ambiguity taxonomy that categorizes different types of instruction ambiguities and refines initial instructions with clearer specifications. |
| Outcome: | The proposed task improves alignment of generated text with user expectations, achieving 15.02-point increase in ROUGE scores. |
CHAmbi: A New Benchmark on Chinese Ambiguity Challenges for Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Ambiguity is an inherent feature of language, whose management is crucial for effective communication and collaboration. |
| Approach: | They propose a dataset to evaluate LLMs' ability to handle ambiguity in Chinese by using a specialized Chinese multi-label disambiguation dataset formatted in Natural Language Inference. |
| Outcome: | The CHAmbi dataset comprises 4,991 pairs of premises and hypotheses, including 824 examples featuring a wide range of ambiguities. |
CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models (2024.acl-long)
Copied to clipboard
Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, Tat-Seng Chua
| Challenge: | Large language models are used to meet user information needs, but their effectiveness in dealing with user queries that contain various types of ambiguity remains unknown. |
| Approach: | They propose a benchmark for evaluating large language models using a well-organized taxonomy. |
| Outcome: | The proposed model is based on a well-organized taxonomy and compares it with other models. |
Detecting Ambiguous Utterances in an Intelligent Assistant (2024.emnlp-industry)
Copied to clipboard
| Challenge: | ambiguous utterances can be interpreted as either chat or task intents in intelligent assistants . ambiguity of intent is particularly noticeable in intelligent devices where task-oriented and non-task-oriented utterrances are mixed and most utterations are short due to characteristics of devices. |
| Approach: | They propose to feed sentence embeddings developed from microblogs and search logs with a self-attention mechanism to detect ambiguous utterances robustly. |
| Outcome: | The proposed model outperforms baselines and a strong LLM-based model. |
Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems require large number of accurate annotations, such as image-level labels and location-level labeling. |
| Approach: | They propose a joint anaphora resolution and object localization dataset targeting visual-linguistic ambiguity. |
| Outcome: | The proposed framework improves visual-linguistic alignment and object localization with one joint model compared to a strong single-task baseline. |
Aligning Language Models to Explicitly Handle Ambiguity (2024.emnlp-main)
Copied to clipboard
Hyuhng Joon Kim, Youna Kim, Cheonbok Park, Junyeob Kim, Choonghyun Park, Kang Min Yoo, Sang-goo Lee, Taeuk Kim
| Challenge: | Large language models (LLMs) are not specifically trained to deal with ambiguous utterances . ambiguity can lead to varying interpretations of the same input based on different assumptions or background knowledge . |
| Approach: | They propose a pipeline that aligns large language models to manage ambiguous queries . they propose to use their own assessment of perceived ambiguity to detect and manage queries a . |
| Outcome: | Experimental results show that APA empowers LLMs to detect and manage ambiguous queries while retaining the ability to answer clear questions. |
We’re Afraid Language Models Aren’t Modeling Ambiguity (2023.emnlp-main)
Copied to clipboard
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith, Yejin Choi
| Challenge: | Ambiguity is an intrinsic feature of natural language, allowing us to anticipate misunderstandings and revise our interpretations as listeners. |
| Approach: | They use AmbiEnt to capture ambiguity in a sentence and analyze it to evaluate pretrained LMs. |
| Outcome: | The proposed model can flag political claims in the wild that are misleading due to ambiguity. |
Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | ambiguities in natural language can lead to outputs that seem correct but fail to reflect the speaker’s intent. |
| Approach: | They propose to identify and then resolve ambiguities in natural language and propose metrics to quantify them. |
| Outcome: | The proposed metrics better correlate with human annotations than uncertainty baselines. |
AmbigQA: Answering Ambiguous Open-domain Questions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing open-domain question answering systems assume questions have a single welldefined answer. |
| Approach: | They propose an open-domain question answering task which involves finding every plausible answer and rewriting the question for each one to resolve the ambiguity. |
| Outcome: | The proposed task is based on a dataset covering 14,042 open-domain questions . it shows that strong models benefit from weakly supervised learning . |
Language Models Identify Ambiguities and Exploit Loopholes (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that exploit loopholes identify and reason about ambiguity and conflicting goals, presenting a potential safety risk. |
| Approach: | They propose to study the responses of large language models to loopholes by examining ambiguity and pragmatics in LLMs. |
| Outcome: | The proposed models can identify ambiguities and exploit loopholes to satisfy their given goals as opposed to the goals of the user. |