| Challenge: | Experimental results show that neural semantic parsers are difficult to interpret due to their complexity. |
| Approach: | They propose to use confidence models to estimate predictions for neural semantic parsers . they outline three major causes of uncertainty and use metrics to quantify them . |
| Outcome: | The proposed model outperforms a widely used method that relies on posterior probability and improves interpretation quality. |
Similar Papers
Calibrated Interpretation: Confidence Estimation in Semantic Parsing (2023.tacl-1)
Copied to clipboard
| Challenge: | Sequence generation models are increasingly being used to translate natural language into programs . calibration of such models is a key component of safety, says aaron sagar . |
| Approach: | They investigate whether calibration of popular generation models varies across models and datasets . they find that calibration varies among models and data sets, and that it is important to include it in evaluations if it is included . |
| Outcome: | The calibration of popular generation models varies across models and datasets . the authors find that the accuracy of models is dependent on confidence . |
AdaNSP: Uncertainty-driven Adaptive Decoding in Neural Semantic Parsing (P19-1)
Copied to clipboard
| Challenge: | Semantic parsing (SP) maps a natural language utterance into a formal language . standard Seq2Seq models ignore underlying grammars and may give ill-formed results. |
| Approach: | They propose an end-to-end model for semantic parsing that transduces a natural language sentence to the formal semantic representation. |
| Outcome: | The proposed model outperforms the state-of-the-art models and does not need expertise like predefined grammar or sketches in the meantime. |
Neural Semantic Parsing (P18-5)
Copied to clipboard
| Challenge: | Semantic parsing is the study of translating natural language utterances into machine-executable programs. |
| Approach: | They will describe the various approaches researchers have taken to translate natural language into a formal language . they will also discuss why much recent work has chosen to use standard programming languages instead of more linguistically-motivated representations. |
| Outcome: | This paper will describe the various approaches researchers have taken to translate natural language into a formal language. |
Coarse-to-Fine Decoding for Neural Semantic Parsing (P18-1)
Copied to clipboard
| Challenge: | Experimental results show that semantic parsing is more efficient than using simple decoders. |
| Approach: | They propose a structure-aware neural architecture which decomposes the semantic parsing process into two stages. |
| Outcome: | The proposed architecture consistently improves performance on four datasets characteristic of different domains and meaning representations. |
Semantic Accuracy in Natural Language Generation: A Thesis Proposal (2023.acl-srw)
Copied to clipboard
| Challenge: | Using large pre-trained language models, it is essential to research their reliability . if a human does not know the answer to a question, the socially acceptable behavior is to say 'I do not know' failing to fulfill this expectation can lead to distrust, or spread of misinformation. |
| Approach: | They propose a method for evaluating semantic accuracy and a benchmark for NLG metrics. |
| Outcome: | The proposed method evaluates semantic accuracy and provides a benchmark for NLG metrics. |
Context Dependent Semantic Parsing: A Survey (2020.coling-main)
Copied to clipboard
| Challenge: | Semantic parsing is the task of translating natural language utterances into machine-readable meaning representations. |
| Approach: | They propose to use contextual information to translate natural language utterances into machine-readable meaning representations. |
| Outcome: | The proposed methods do not utilize contextual information, which could boost the semantic parsing systems. |
Modeling Input Uncertainty in Neural Network Dependency Parsing (D18-1)
Copied to clipboard
| Challenge: | Recent advances in neural network parsers address data sparsity issues by modeling character level information and exploiting raw data in semi-supervised settings. |
| Approach: | They investigate whether lexical normalization provides similar functionality to lexiconal normalization . they show that a separate normalization component improves performance of a neural network parser . |
| Outcome: | The proposed approaches improve performance even with access to character level information and word embeddings. |
Reranking for Neural Semantic Parsing (P19-1)
Copied to clipboard
| Challenge: | Semantic parsing is the task of transducing natural language utterances into machine executable meaning representations (e.g., Python code). |
| Approach: | They propose to rerank an n-best list of predicted MRs and use features to fix observed problems with baseline models to improve parser performance. |
| Outcome: | The proposed method outperforms the best published neural parser on four datasets and improves the baseline parsing performance by 5.7% and 2.9%. |
Towards Collaborative Neural-Symbolic Graph Semantic Parsing via Uncertainty (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent work in task-independent graph semantic parsing has shifted from symbolic approaches to neural models, showing strong performance on different types of meaning representations. |
| Approach: | They propose a framework that incorporates prior knowledge from a symbolic parser into a decision criterion for beam search to address these limitations. |
| Outcome: | The proposed framework improves on the in-distribution test set but degrades significantly on long-tail situations while the symbolic parser performs more robustly. |
Improving the Calibration of Confidence Scores in Text Generation Using the Output Distribution’s Characteristics (2025.acl-short)
Copied to clipboard
| Challenge: | Existing methods for estimating confidence in text generation do not account for many valid answers in generation tasks. |
| Approach: | They propose task-agnostic confidence metrics that rely solely on model probabilities without the need for further fine-tuning or heuristics. |
| Outcome: | The proposed models improve the accuracy of BART and Flan-T5 on summarization, translation, and question answering datasets. |