Semantic Evaluation for Text-to-SQL with Distilled Test Suites (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate semantic accuracy of Text-to-SQL models are not accurate. |
| Approach: | They propose a test suite accuracy method to approximate semantic accuracy for Text-to-SQL models. |
| Outcome: | The proposed method evaluates 21 models submitted to the Spider leader board and manually checks on 100 examples. |
Similar Papers
FLEX: Expert-level False-Less EXecution Metric for Text-to-SQL Benchmark (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluation methods for text-to-SQL systems show many false positives and negatives . however, the Execution Accuracy (EX) metric is flawed and can diverge from human experts. |
| Approach: | They propose a method to evaluate text-to-SQL systems using large language models to emulate human expert-level evaluation of SQL queries. |
| Outcome: | The proposed metric improves agreement with human experts with comprehensive context and sophisticated criteria. |
Evaluating Cross-Domain Text-to-SQL Models and Benchmarks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Text-to-SQL benchmarks are used to evaluate progress made in the field . however, matching a model-generated SQL query to a reference SQL query fails due to various reasons. |
| Approach: | They conduct an extensive evaluation of text-to-SQL benchmarks and re-evaluate some of the top-performing models. |
| Outcome: | The results show that a recent model surpasses the gold standard reference queries in the Spider benchmark in human evaluation. |
A Review of Cross-Domain Text-to-SQL Models (2020.aacl-srw)
Copied to clipboard
| Challenge: | WikiSQL and Spider are cross-domain text-to-SQl datasets that have attracted much attention from the research community. |
| Approach: | They propose to divide top models into two paradigms and evaluate their models for schema linking, pretrained word embeddings, reasoning assistance modules. |
| Outcome: | The proposed models have over 90% execution accuracy, the authors show . the proposed models are more complex and more complex than the proposed ones . |
TS-SQL: Test-driven Self-refinement for Text-to-SQL (2025.findings-emnlp)
Copied to clipboard
| Challenge: | null |
| Approach: | null |
| Outcome: | null |
Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale, pre-trained language models achieve human-level and superhuman accuracy on existing language understanding tasks, but statistical bias in benchmark data and probing studies has recently called into question their true capabilities. |
| Approach: | They propose to evaluate systems through a measure of prediction coherence by using two existing language understanding benchmarks with different properties to demonstrate its versatility. |
| Outcome: | The proposed evaluation framework is quick, effective, and versatile to provide insight into the coherence of machines’ predictions. |
Improving Text-to-SQL Evaluation Methodology (P18-1)
Copied to clipboard
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, Dragomir Radev
| Challenge: | Current evaluations of text-to-SQL systems are limited by the way they divide data into training and test sets. |
| Approach: | They propose to standardize and improve existing and new text-to-SQL datasets . they propose a template-based slot-filling baseline that cannot generalize to new queries . |
| Outcome: | The proposed system is competitive with prior work on multiple datasets and can be used on training and test sets. |
CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation protocols for text generation suffer from rating inconsistencies . lexical overlap-based metrics align poorly with human judgments . |
| Approach: | They propose a checklist-based evaluation framework that improves rating reliability via decomposed binary questions. |
| Outcome: | The proposed framework improves rating reliability by decomposing binary questions . it improves agreement across evaluator models by 0.45 and reduces score variance . human evaluation remains the gold standard, but it #, Equal contribution. |
Leveraging Adjective-Noun Phrasing Knowledge for Comparison Relation Prediction in Text-to-SQL (D19-1)
Copied to clipboard
| Challenge: | Existing models for text-to-SQL do not explicitly introduce common knowledge to address comparison relations. |
| Approach: | They propose to leverage adjective-noun phrasing knowledge mined from the web to predict comparison relations in text-to-SQL. |
| Outcome: | The proposed approach improves on the original and re-split Spider datasets on comparison relation prediction. |
Semantic-Eval : A Semantic Comprehension Evaluation Framework for Large Language Models Generation without Training (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have emerged as key drivers of progress in the field of natural language processing. |
| Approach: | They propose a framework that assesses LLM-generated text based on semantic understanding. |
| Outcome: | The proposed framework surpasses traditional evaluation metrics and lags behind GPT-4. |
DTS-SQL: Decomposed Text-to-SQL with Small Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | relying on proprietary Large Language Models poses privacy and cost implications for models. |
| Approach: | They propose a two-stage fine-tuning approach that breaks down the task into two simpler tasks. |
| Outcome: | The proposed method achieves 60.31% execution accuracy on Bird hold-out test set . it is the highest performance among methods using 7B parameter models . |