RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations (2023.acl-long)
Copied to clipboard
Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi, Dragomir Radev
| Challenge: | Existing Table QA models are vulnerable to task-specific perturbations, such as replacing key question entities or shuffling table columns. |
| Approach: | They propose to use large language models to generate adversarial examples to enhance training, which significantly improves the robustness of Table QA models. |
| Outcome: | The proposed model significantly improves on existing Table QA models against human-annotated adversarial perturbations. |
Similar Papers
Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing Text-to-SQL parsers are vulnerable to perturbations in NL questions . we propose the Adversarial Table Perturbation (ATP) as a new attacking paradigm . |
| Approach: | They propose to use the Adversarial Table Perturbation to measure robustness of Text-to-SQL parsers against adversarial perturbations. |
| Outcome: | The proposed approach outperforms baseline methods in robustness evaluations on ADVETA and can be used in future projects. |
Beyond Static Synthetic Noise: Assessing the Robustness of Large Language Models to Natural Context Variation in the Real World (2026.findings-acl)
Copied to clipboard
| Challenge: | Current robustness evaluation methods rely on static synthetic perturbations to stress-test models. |
| Approach: | They propose a framework for automatically evaluating QA models under naturally occurring textual perturbations by replacing context passages with revised Wikipedia edit histories. |
| Outcome: | The proposed framework replaces context passages with revised Wikipedia edit histories to improve model performance. |
FREB-TQA: A Fine-Grained Robustness Evaluation Benchmark for Table Question Answering (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for Table Question Answering lack robustness . current systems generate inconsistent responses by performing coarse-grained changes in tables and questions. |
| Approach: | They formalize three major desiderata for a fine-grained evaluation of robustness of TQA systems . they aim to evaluate TQA responses on the content of relevant cells rather than on biases . |
| Outcome: | The proposed benchmarks show that state-of-the-art TQA systems fail in three aspects . the benchmarks are released publicly and are expected to improve over time . |
A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Using large language models, we evaluated their robustness on multiple datasets. |
| Approach: | They propose a new metric for assessing model robustness by empirical evaluation of several models on multiple datasets. |
| Outcome: | The proposed metric is based on a set of datasets that are constructed by introducing naturally-occurring, non-malicious perturbations or by generating semantically equivalent paraphrases of input questions or statements. |
RobustQA: A Framework for Adversarial Text Generation Analysis on Question Answering Systems (2023.emnlp-demo)
Copied to clipboard
Yasaman Boreshban, Seyed Morteza Mirbostani, Seyedeh Fatemeh Ahmadi, Gita Shojaee, Fatemeh Kamani, Gholamreza Ghassem-Sani, Seyed Abolghasem Mirroshandel
| Challenge: | Question answering (QA) systems have reached human-level accuracy, but they are not robust enough and vulnerable to adversarial examples. |
| Approach: | They modified the attack algorithms widely used in text classification to fit them for QA systems. |
| Outcome: | The proposed framework is the first open-source toolkit for investigating textual adversarial attacks in QA systems. |
Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation (2026.acl-long)
Copied to clipboard
| Challenge: | Table Question Answering (TQA) aims to answer natural language questions using tabular data. |
| Approach: | They propose a systematic overview of TQA research using large language models and summarize available benchmarks based on task features. |
| Outcome: | The proposed framework provides a comprehensive overview of the current state of the art in the field of Table Question Answering. |
Tougher Text, Smarter Models: Raising the Bar for Adversarial Defence Benchmarks (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have highlighted the vulnerability of deep learning models to adversarial attacks. |
| Approach: | They propose a benchmark for textual adversarial defence that evaluates state-of-the-art defence mechanisms across diverse datasets, models, and tasks. |
| Outcome: | The proposed benchmark incorporates a wide range of datasets and evaluates state-of-the-art defence mechanisms. |
Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialness (2025.naacl-long)
Copied to clipboard
| Challenge: | As models evolve, datasets can become outdated. |
| Approach: | They propose a human-grounded evaluation metric that assesses adversarialness by capturing models’ and humans’ varying abilities, while also identifying poor examples. |
| Outcome: | The proposed evaluation metric measures the accuracy of an adversarial question answering dataset and determines whether models are performing well on the dataset. |
Robustness Evaluation of the German Extractive Question Answering Task (2025.coling-main)
Copied to clipboard
| Challenge: | Existing evaluation benchmarks for Question Answering systems only include EM and F1 scores, but they overlook critical factors for the deployment of QA systems. |
| Approach: | They propose to define an evaluation method specifically tailored to the German language to evaluate the robustness of German QA models. |
| Outcome: | The proposed method extends existing methods to German language . it shows that all models are vulnerable to character-level perturbations . |
How Robust Are the QA Models for Hybrid Scientific Tabular Data? A Study Using Customized Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing tabular QA models are lacking in understanding their robustness on scientific information. |
| Approach: | They propose a dataset to assess the robustness of tabular QA models on scientific hybrid tabular data. |
| Outcome: | The proposed model performs well on scientific tables and text, while the best score is 0.462. |