Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, Dragomir Radev
| Challenge: | Current evaluations of text-to-SQL systems are limited by the way they divide data into training and test sets. |
| Approach: | They propose to standardize and improve existing and new text-to-SQL datasets . they propose a template-based slot-filling baseline that cannot generalize to new queries . |
| Outcome: | The proposed system is competitive with prior work on multiple datasets and can be used on training and test sets. |
Similar Papers
Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expect (2022.coling-1)
Copied to clipboard
| Challenge: | text-to-SQL is a language processing and database-based language processing (NLP) task is to convert natural utterances into SQL queries and its practical application is to build natural language interfaces to database systems. |
| Approach: | They propose to conduct a systematic survey of text-to-SQL to examine the challenges and potential future directions. |
| Outcome: | The proposed system converts natural utterances into SQL queries and is a representative task in semantic parsing. |
Exploring Schema Generalizability of Text-to-SQL (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing text-to-SQL models are limited in their generalizability, despite their performance being over-estimated. |
| Approach: | They propose a framework to generate novel text-to-SQL data via automatic and synchronous (DS, SQL) pair altering. |
| Outcome: | The proposed framework generates text-to-SQL data via automatic and synchronous (DS, SQL) pair altering. |
Evaluating Cross-Domain Text-to-SQL Models and Benchmarks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Text-to-SQL benchmarks are used to evaluate progress made in the field . however, matching a model-generated SQL query to a reference SQL query fails due to various reasons. |
| Approach: | They conduct an extensive evaluation of text-to-SQL benchmarks and re-evaluate some of the top-performing models. |
| Outcome: | The results show that a recent model surpasses the gold standard reference queries in the Spider benchmark in human evaluation. |
On the Effectiveness of Automated Metrics for Text Generation Systems (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation methods lack a sound theoretical foundation for evaluation campaigns . imperfect automated metrics and insufficiently sized test sets are some of the factors that cause uncertainty. |
| Approach: | They propose a theoretical framework that incorporates different sources of uncertainty, such as imperfect automated metrics and insufficiently sized test sets. |
| Outcome: | The proposed model can be leveraged to improve evaluation protocols regarding reliability, robustness, and significance of the evaluation outcome. |
Importance of Synthesizing High-quality Data for Text-to-SQL Parsing (2023.findings-acl)
Copied to clipboard
Yiqun Hu, Yiyun Zhao, Jiarong Jiang, Wuwei Lan, Henghui Zhu, Anuj Chauhan, Alexander Hanbo Li, Lin Pan, Jun Wang, Chung-Wei Hang, Sheng Zhang, Jiang Guo, Mingwen Dong, Joseph Lilien, Patrick Ng, Zhiguo Wang, Vittorio Castelli, Bing Xiang
| Challenge: | Existing text-to-SQL parsers lack the data to perform well with augmented synthetic data. |
| Approach: | They propose a framework that imposes strong typing constraints and incorporates key relationships from schema. |
| Outcome: | The proposed framework improves on the high-quality synthesized SQL and natural language question (NLQ) models have significant accuracy boosts and achieve new state-of-the-art performance on spider. |
SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gaps (2025.findings-emnlp)
Copied to clipboard
| Challenge: | SQLSpace is a representation for text-to-SQL examples derived with minimal human intervention. |
| Approach: | They introduce SQLSpace, a human-interpretable, generalizable, compact representation for text-to-SQL examples . they demonstrate that SQLSpace enables analysis that would be difficult with raw examples alone . |
| Outcome: | The proposed representations are human-interpretable and generalizable . they are used to evaluate models with a granularity beyond overall accuracy scores . |
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications (2022.naacl-main)
Copied to clipboard
| Challenge: | Evaluating natural language generation systems is difficult, as there are many ways to express similar things in text. |
| Approach: | They combine interviews with NLG practitioners to examine ethical considerations and their implications for NLG evaluation. |
| Outcome: | The findings of the study surface goals, community practices, assumptions, and constraints that shape NLG evaluations, and examine their implications and how they embody ethical considerations. |
Know What I don’t Know: Handling Ambiguous and Unknown Questions for Text-to-SQL (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing text-to-SQL parsers generate a plausible SQL query for arbitrary user questions, thereby failing to handle problematic user questions. |
| Approach: | They propose a weakly supervised DTE model for error detection, localization, and explanation. |
| Outcome: | The proposed model achieves the best result on real-world examples and generated examples compared with baselines. |
Interactive Text-to-SQL Generation via Editable Step-by-Step Explanations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to generate SQL from natural language are still making many mistakes . a new interaction mechanism allows users to edit a step-by-step explanation of a query to fix errors. |
| Approach: | They propose a mechanism that allows users to edit a step-by-step explanation of a query to fix errors. |
| Outcome: | The proposed approach can achieve better performance than multiple SOTA approaches on multiple datasets and 24 participants. |
Evaluating Historical Text Normalization Systems: How Well Do They Generalize? (N18-2)
Copied to clipboard
| Challenge: | Historical text normalization systems aim to convert historical wordforms to their modern equivalents . many of these systems have been developed and tested on a single language . |
| Approach: | They propose to use a nave baseline system to evaluate historical text normalization systems . they show that the models generalize well to unseen words in tests on five languages . |
| Outcome: | The proposed models generalize well to unseen words on five languages, but provide no clear benefit over the nave baseline. |