Revisiting Automated Evaluation for Long-form Table Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing automated metrics for long-form table question answering (LFTQA) are poorly correlated with human judgments and fail to distinguish between factually accurate responses and those that are factual incorrect. |
| Approach: | They propose to use a meta-evaluation dataset to assess the effectiveness of LLM-based LFTQA systems. |
| Outcome: | The proposed meta-evaluation dataset includes 2,988 human-annotated examples. |
Similar Papers
A Critical Evaluation of Evaluations for Long-form Question Answering (2023.acl-long)
Copied to clipboard
| Challenge: | Long-form question answering (LFQA) is an emerging research area within QA . however, its flexibility poses enormous challenges for evaluation . |
| Approach: | They conduct the first targeted study of the evaluation of long-form answers, covering both human and automatic evaluation practices. |
| Outcome: | The proposed evaluations cover human and automatic evaluations. |
Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Table Question Answering (TQA) aims to answer natural language questions using tabular data. |
| Approach: | They propose a systematic overview of TQA research using large language models and summarize available benchmarks based on task features. |
| Outcome: | The proposed framework provides a comprehensive overview of the current state of the art in the field of Table Question Answering. |
PEDANTS: Cheap but Effective and Interpretable Answer Equivalence (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Current short-form QA evaluations lack diverse styles of evaluation data and rely on expensive and slow LLMs. |
| Approach: | They propose a rubric for machine QA that is more stable than an exact match and neural methods. |
| Outcome: | The proposed evaluations improve on the existing short-form QA evaluations using the Trivia community. |
TableEval: A Real-World Benchmark for Complex, Multilingual, and Multi-Structured Table Question Answering (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing TableQA benchmarks focus on simple flat tables and suffer from data leakage . current benchmarks are monolingual and fail to capture cross-lingual variability . |
| Approach: | They propose a table-based TableQA benchmark to evaluate LLMs on real-world tasks. |
| Outcome: | The proposed benchmarks show that they achieve high agreement with human judgment . the proposed framework improves on the alignment between model responses and reference answers . |
On Evaluating the Integration of Reasoning and Action in LLM Agents with Database Question Answering (2024.findings-naacl)
Copied to clipboard
| Challenge: | a new study evaluates how Large Language Models interact with a SQL interpreter . the model is limited in context and is stochastic, making it less suited for tasks requiring high precision and extensive computations. |
| Approach: | They propose and evaluate two interaction strategies to evaluate how LLMs interact with a SQL interpreter. |
| Outcome: | The proposed framework improves the accuracy and reliability of the evaluations. |
ELI5: Long Form Question Answering (P19-1)
Copied to clipboard
| Challenge: | Existing question answering datasets provide extractive or short answers, but less attention has been paid to open-ended questions that require explanations. |
| Approach: | They present a large-scale corpus for long form question answering . they use a Reddit forum to provide elaborate answers to open-ended questions . |
| Outcome: | The proposed model outperforms Seq2Seq, language modeling, and other models in human evaluations. |
Question Answering over Tabular Data with DataBench: A Large-Scale Empirical Evaluation of LLMs (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are showing emerging abilities, but they are not large enough to assess their capabilities. |
| Approach: | They propose a benchmark that compares large language models with open and closed source models. |
| Outcome: | The proposed benchmark compares open and closed-source models with open-source and closed source models. |
Hurdles to Progress in Long-form Question Answering (2021.naacl-main)
Copied to clipboard
| Challenge: | Long-form question answering (LFQA) involves retrieving documents relevant to a given question and using them to generate a paragraph-length answer. |
| Approach: | They propose a long-form question answering system that relies on sparse attention and contrastive retriever learning to achieve state-of-the-art performance on the ELI5 LFQA dataset. |
| Outcome: | The proposed system tops the public leaderboard on the ELI5 LFQA dataset, but it has several troubling issues. |
DebateQA: Evaluating Question Answering on Debatable Knowledge (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing QA benchmarks that provide fixed answers to debatable questions are inadequate for evaluating their performance. |
| Approach: | They propose to use a dataset of 2,941 debatable questions to assess their ability to provide comprehensive answers to inherently debatably asked questions. |
| Outcome: | The proposed model performs well on 2,941 debatable questions accompanied by human-annotated partial answers that capture a variety of perspectives. |
FinLFQA: Evaluating Attributed Text Generation of LLMs in Financial Long-Form Question Answering (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks focus on simple attribution that retrieves textual evidence as references. |
| Approach: | They propose a benchmark to evaluate the ability of large language models to generate reliable attributions. |
| Outcome: | The proposed benchmark evaluates the ability of LLMs to generate long-form answers with reliable and nuanced attributions. |