Measuring and Mitigating Constraint Violations of In-Context Learning for Utterance-to-API Semantic Parsing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In task-oriented semantic parsing, the system aims to translate users’ utterances in natural language to machine-interpretable programs (API calls) However, Large Language Models (LLMs) are known to hallucinate and therefore pose a formidable challenge in constraining generated content. |
| Approach: | They propose to use large language models to translate user's utterances to machine-interpretable programs (API calls) they identify constraints violations in task-oriented utterrances and define fine-grained metrics that complement traditional ones. |
| Outcome: | The proposed methods reduce constraints violations and improve quality of the generated API calls, but require careful consideration given their implementation complexity and latency. |
Similar Papers
Learning API Functionality from In-Context Demonstrations for Tool-based Agents (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Documentation is often missing, outdated, privatized, or inconsistent in tool-based agents. |
| Approach: | They propose a task that allows agents to learn API functionality directly from in-context demonstrations. |
| Outcome: | The proposed model improves the task success rate of an API-based agent by providing explicit function calls and natural language critiques. |
Addressing Resource and Privacy Constraints in Semantic Parsing Through Data Augmentation (2022.findings-acl)
Copied to clipboard
| Challenge: | a low-resource task-oriented semantic parser is limited by privacy requirements for unlabeled natural utterances. |
| Approach: | They propose a setup for low-resource task-oriented semantic parsing based on user interactions . they use structured canonical utterances, then simulating corresponding natural language to improve performance. |
| Outcome: | The proposed setup improves on a low-resource task-oriented semantic parser using utterances collected through user interactions. |
Calibrated Interpretation: Confidence Estimation in Semantic Parsing (2023.tacl-1)
Copied to clipboard
| Challenge: | Sequence generation models are increasingly being used to translate natural language into programs . calibration of such models is a key component of safety, says aaron sagar . |
| Approach: | They investigate whether calibration of popular generation models varies across models and datasets . they find that calibration varies among models and data sets, and that it is important to include it in evaluations if it is included . |
| Outcome: | The calibration of popular generation models varies across models and datasets . the authors find that the accuracy of models is dependent on confidence . |
API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs (2024.acl-long)
Copied to clipboard
Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Vernon Austel, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, Luis Lastras
| Challenge: | Existing methods to train and test large language models that involve calls to tools and APIs are lacking. |
| Approach: | They propose a large corpora for training and systematic testing of tool-augmented LLMs. |
| Outcome: | The proposed datasets mimic real-world scenarios involving API-tasks and slot filling. |
Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programs (D19-1)
Copied to clipboard
| Challenge: | Semantic parsing aims to map natural language utterances onto machine interpretable meaning representations. |
| Approach: | They propose to instill an inductive bias in the parser to help it distinguish between spurious and correct programs. |
| Outcome: | The proposed model is highly tractable on WikiTableQuestions and WikiSQL datasets. |
Deconstructing In-Context Learning: Understanding Prompts via Corruption (2024.lrec-main)
Copied to clipboard
| Challenge: | Prior work examined how modifying different elements of the prompt can affect model performance, but this limited number of elements made replication challenging. |
| Approach: | They decompose the entire prompt into four components: task description, demonstration inputs, labels, and inline instructions provided for each demonstration. |
| Outcome: | The proposed model is robust to minor prompt modifications, but its underlying pre-trained backbone is brittle . previous studies focused on models with fewer than 15 billion parameters or exclusively examined black-box models like GPT-3 or PaLM, making replication challenging. |
Context Dependent Semantic Parsing: A Survey (2020.coling-main)
Copied to clipboard
| Challenge: | Semantic parsing is the task of translating natural language utterances into machine-readable meaning representations. |
| Approach: | They propose to use contextual information to translate natural language utterances into machine-readable meaning representations. |
| Outcome: | The proposed methods do not utilize contextual information, which could boost the semantic parsing systems. |
Learning to Generate Structured Output with Schema Reinforcement Learning (2025.acl-long)
Copied to clipboard
Yaxi Lu, Haolun Li, Xin Cong, Zhong Zhang, Yesai Wu, Yankai Lin, Zhiyuan Liu, Fangming Liu, Maosong Sun
| Challenge: | Recent advances in large language models have facilitated the development of intelligent applications like automatic web search (Qin et al., 2023) Several methods exist for generating JSON strings from LLMs, including Prompting but often miss certain schemas. |
| Approach: | They propose to use 40K different JSON schemas to assess models' ability to generate valid JSON outputs. |
| Outcome: | The proposed model improves both in generating JSON outputs and downstream tasks. |
A Thorough Examination of Decoding Methods in the Era of LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Decoding methods are essential for converting language models from next-token predictors into practical task solvers. |
| Approach: | They propose to evaluate decoding methods in general-purpose large language models . they find that decoding method performance is notably task-dependent . |
| Outcome: | The proposed methods perform task-dependently and are influenced by alignment, model size, and quantization. |
Diagnosing Transformers in Task-Oriented Semantic Parsing (2021.findings-acl)
Copied to clipboard
| Challenge: | a recent study shows transformer-based parsers struggle with disambiguating intents/slots and producing syntactically valid frames. |
| Approach: | They propose to use seq2seq transformers to map textual utterances to semantic frames . they propose to model transformer-based parsers across monolingual and multilingual settings . |
| Outcome: | The proposed parsers struggle with disambiguating intents/slots and produce syntactically valid frames. |