| Challenge: | Existing datasets for reading comprehension tasks have been used to test the generalization of natural language understanding systems. |
| Approach: | They propose a diagnostic benchmark suite to clarify key issues related to the robustness and systematicity of NLU systems. |
| Outcome: | The proposed benchmark suite clarifies key issues related to the robustness and systematicity of NLU systems. |
Similar Papers
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021.naacl-main)
Copied to clipboard
| Challenge: | Evaluation for many natural language understanding (NLU) tasks is broken due to unreliable and biased systems scoring so high on standard benchmarks. |
| Approach: | They argue that current benchmarks fail at four criteria for evaluation . they argue that adversarial data collection does not address the causes of failures . |
| Outcome: | The proposed frameworks fail at four criteria, and adversarial data collection does not address the causes of these failures, the authors argue . restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, reliability with which they are annotated, their size, and the ways they handle social bias. |
Adversarial NLI: A New Benchmark for Natural Language Understanding (2020.acl-main)
Copied to clipboard
| Challenge: | a new large-scale NLI benchmark dataset is presented to test models on a variety of popular NLIs. |
| Approach: | They propose a large-scale NLI benchmark dataset that is iteratively compared with a human-and-model-in-the-loop procedure. |
| Outcome: | The proposed method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate. |
KLEJ: Comprehensive Benchmark for Polish Language Understanding (2020.acl-main)
Copied to clipboard
| Challenge: | Recent introduction of robust, general-purpose models for fine-tuning has enabled improvements in general natural language understanding (NLU) but such benchmarks are only available for a handful of languages. |
| Approach: | They propose a multi-task benchmark for the Polish language understanding with an online leaderboard . they also propose GLUE, a task for named entity recognition and sentiment analysis . |
| Outcome: | The proposed model performs best on three out of nine tasks in the Polish language . the proposed model is also used in an e-commerce domain to analyze the sentiments of users . |
Probing Linguistic Systematicity (2020.acl-main)
Copied to clipboard
| Challenge: | Existing evidence that deep natural language understanding models do not learn systematically is lacking. |
| Approach: | They examine whether deep natural language understanding models exhibit systematicity . they find that network architectures can generalize non-systematically . |
| Outcome: | The proposed model generalizes non-systematically, but is unsatisfactory, the authors argue . they show that the current state-of-the-art models do not generalize systematically . |
NLU++: A Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialogue (2022.findings-naacl)
Copied to clipboard
| Challenge: | NLU++ provides a more challenging evaluation environment for dialogue NLU models . Typical ToD systems still rely on a modular design . |
| Approach: | They propose to use NLU++ to provide a more challenging evaluation environment for dialogue NLU models. |
| Outcome: | The proposed dataset improves existing datasets and provides a much more challenging evaluation environment for dialogue NLU models. |
Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages (2023.acl-long)
Copied to clipboard
Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, Pratyush Kumar
| Challenge: | Recent advances in Natural Language Understanding are driven by pretrained multilingual models, which can potentially reduce the performance gap between high-resource languages through zero-shot knowledge transfer. |
| Approach: | They propose to create a human-supervised benchmark for Indic languages, IndicXTREME, with nine diverse NLU tasks covering 20 languages. |
| Outcome: | The proposed model improves on the monolingual corpora, IndicCorp, and IndicBERT in Indic languages with 105 evaluation sets across languages and tasks. |
Statistically Profiling Biases in Natural Language Reasoning Datasets and Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to evaluate NLP models' weaknesses are limited by “hypothesis-only” tests and CheckLists. |
| Approach: | They propose a lightweight general statistical profiling framework that automatically identifies potential biases in multiple-choice NLU datasets without requiring additional test cases. |
| Outcome: | The proposed framework assesses the extent to which models exploit these biases through black-box testing, confirming prior findings and revealing new insights. |
MENLI: Robust Evaluation Metrics from Natural Language Inference (2023.tacl-1)
Copied to clipboard
| Challenge: | Recent proposed BERT-based evaluation metrics for text generation are vulnerable to adversarial attacks, e.g., relating to information correctness. |
| Approach: | They propose to use BERT-based evaluation metrics for text generation to evaluate text for semantic similarity but are vulnerable to adversarial attacks using Natural Language Inference. |
| Outcome: | The proposed metrics outperform existing summarization metrics but perform below SOTA MT metrics on standard benchmarks. |
FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding (2022.acl-long)
Copied to clipboard
Yanan Zheng, Jing Zhou, Yujie Qian, Ming Ding, Chonghua Liao, Li Jian, Ruslan Salakhutdinov, Jie Tang, Sebastian Ruder, Zhilin Yang
| Challenge: | Existing evaluation protocols for few-shot natural language understanding (NLU) tasks are inconsistent and hinder fair comparison and measuring progress. |
| Approach: | They propose an evaluation framework that improves previous evaluation procedures in three key aspects, i.e., test performance, dev-test correlation, and stability. |
| Outcome: | The proposed framework improves evaluation procedures in three key aspects, i.e., performance, dev-test correlation, and stability. |
A MISMATCHED Benchmark for Scientific Natural Language Inference (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets for scientific NLI are derived from various computer science domains, whereas non-CS domains are completely ignored. |
| Approach: | They propose a scientific natural language inference benchmark called MisMatched that incorporates sentence pairs having an implicit scientific NLI relation into model training. |
| Outcome: | The proposed benchmark covers three non-CS domains and contains 2,700 human annotated sentence pairs. |