Papers with Text-to-
Semantic Evaluation for Text-to-SQL with Distilled Test Suites (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate semantic accuracy of Text-to-SQL models are not accurate. |
| Approach: | They propose a test suite accuracy method to approximate semantic accuracy for Text-to-SQL models. |
| Outcome: | The proposed method evaluates 21 models submitted to the Spider leader board and manually checks on 100 examples. |
Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing Text-to-SQL parsers are vulnerable to perturbations in NL questions . we propose the Adversarial Table Perturbation (ATP) as a new attacking paradigm . |
| Approach: | They propose to use the Adversarial Table Perturbation to measure robustness of Text-to-SQL parsers against adversarial perturbations. |
| Outcome: | The proposed approach outperforms baseline methods in robustness evaluations on ADVETA and can be used in future projects. |
StatBot.Swiss: Bilingual Open Data Exploration in Natural Language (2024.findings-acl)
Copied to clipboard
Farhad Nooralahzadeh, Yi Zhang, Ellery Smith, Sabine Maennel, Cyril Matthey-Doret, Raphaël De Fondeville, Kurt Stockinger
| Challenge: | StatBot.Swiss dataset is the first bilingual benchmark for evaluating Text-to-SQL systems based on real-world applications. |
| Approach: | They propose to use a bilingual dataset to evaluate LLMs in Text-to-SQL systems. |
| Outcome: | The proposed dataset contains 455 natural language/SQL-pairs over 35 big databases with varying level of complexity for English and German. |
BiBL: AMR Parsing and Generation with Bidirectional Bayesian Learning (2022.coling-1)
Copied to clipboard
| Challenge: | Existing approaches to AMR focus on one-side improvements despite the duality of the two tasks . instead, we propose data-efficient Bidirectional Bayesian learning (BiBL) to facilitate bidirectional information transition. |
| Approach: | They propose a data-efficient bidirectional Bayesian learning approach to facilitate bidirectional information transition by adopting a single-stage multitasking strategy. |
| Outcome: | The proposed model outperforms existing models on benchmark datasets without extra training data. |
In-Context Reinforcement Learning with Retrieval-Augmented Generation for Text-to-SQL (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods of synthetic query generation generate mostly simple queries which might not be sufficiently representative of complex, real world queries. |
| Approach: | They propose to use large language models to fine tune query generation to produce complex queries that practitioners may pose during inference. |
| Outcome: | The proposed framework achieves 15-20% higher recall in database/table retrieval task compared to the existing state-of-the-art models for schema identification and upto 2% higher execution accuracy for SQL generation. |
Uncovering and Categorizing Social Biases in Text-to-SQL (2023.acl-long)
Copied to clipboard
| Challenge: | Existing Text-to-SQL models are trained on clean, neutral datasets, such as Spider and WikiSQl, but these models contain social bias at different rates. |
| Approach: | They propose to use data to map natural language utterances to SQL queries. |
| Outcome: | The proposed model can contain social bias at different rates in the downstream Text-to-SQL task. |
Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL translation (2024.findings-acl)
Copied to clipboard
Federico Ranaldi, Elena Sofia Ruzzetti, Dario Onorati, Leonardo Ranaldi, Cristina Giannone, Andrea Favalli, Raniero Romagnoli, Fabio Massimo Zanzotto
| Challenge: | Large Language Models (LLMs) understand textual description to generate code in zero-shot scenarios, but there is a possibility that this ability may be influenced by having seen target textual descriptions and the related code. |
| Approach: | They propose a method to detect Data Contamination in Large Language Models (LLMs) and analyze their results on Termite and Spider Datasets to investigate their method. |
| Outcome: | The proposed method detects data contamination in GPTs and analyzes its performance on unfamiliar datasets. |
Enhancing Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies (2023.findings-emnlp)
Copied to clipboard
Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, Dragomir Radev
| Challenge: | In-context learning (ICL) is a new approach to natural language processing tasks that rely on large language models to make predictions based on context . recent studies have shown that neural symbolic design is the preferred choice for question answering systems because of its limited working memory and unreliable long-term memory. |
| Approach: | They propose to extend in-context learning to question answering tasks that utilize structured knowledge sources and to explore various prompt design strategies for employing LLMs. |
| Outcome: | The proposed approach outperforms the state-of-the-art system by 2.5 points and the best fine-tuned system by 5.1 points on the Spider dataset. |
LatentRefusal: Latent-Signal Refusal for Unanswerable Text-to-SQL Queries (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing refusal strategies for unanswerable and underspecified user queries are brittle due to model hallucinations or add complexity and overhead. |
| Approach: | They propose a latent-signal refusal mechanism that predicts query answerability from hidden activations of an LLM. |
| Outcome: | The proposed scheme reduces schema noise and sparse, localized question–schema mismatch cues that indicate unanswerability. |