Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark (P19-1)
Copied to clipboard
| Challenge: | GLUE is a suite of language understanding tasks that has seen dramatic progress in the past year . average performance on the benchmark is 83.9, state of the art at the time of writing . |
| Approach: | They use crowdsourcing to measure human performance on a set of language understanding tasks and 20 examples to determine whether there is room for improvement. |
| Outcome: | The GLUE benchmark outperforms state-of-the-art models on six of the nine tasks and achieves an average score of 87.1. |
Similar Papers
Proxy Indicators for the Quality of Open-domain Dialogues (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for evaluation of open-domain dialogues are expensive and require human annotators to evaluate their quality. |
| Approach: | They propose to use a deep-learning model trained on the general language understanding evaluation benchmark to serve as a quality indication of open-domain dialogues. |
| Outcome: | The proposed model can infer various quality metrics and derive a component-based overall score. |
GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-Distribution Generalization Perspective (2023.findings-acl)
Copied to clipboard
Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, Yue Zhang
| Challenge: | Pre-trained language models (PLMs) have improved generalization performance but the out-of-distribution (OOD) generalization problem remains a challenge in many NLP tasks. |
| Approach: | They propose to create a benchmark for evaluating out-of-distribution (OOD) generalization in NLP models. |
| Outcome: | The proposed benchmarks highlight the importance of OOD robustness and provide insights on how to measure it and improve it. |
MuLD: The Multitask Long Document Benchmark (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing benchmarks for NLP focus on tasks for one or two sentences, but efficient techniques are needed for processing much longer sequences. |
| Approach: | They propose to modify existing NLP tasks to create a long document benchmark which requires models to successfully model long-term dependencies in the text. |
| Outcome: | The proposed benchmark is much more challenging than its ‘short document’ equivalents. |
Adversarial NLI: A New Benchmark for Natural Language Understanding (2020.acl-main)
Copied to clipboard
| Challenge: | a new large-scale NLI benchmark dataset is presented to test models on a variety of popular NLIs. |
| Approach: | They propose a large-scale NLI benchmark dataset that is iteratively compared with a human-and-model-in-the-loop procedure. |
| Outcome: | The proposed method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate. |
Abstraction not Memory: BERT and the English Article System (2022.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained models are the state of the art in linguistics. |
| Approach: | They compare the performance of pre-trained and native English language models on the task of article prediction set up as a three way choice (a/an, the, zero) they argue that BERT captures a high level generalisation of article use akin to human intuition. |
| Outcome: | The proposed model outperforms humans on the linguistically interesting task of article prediction. |
HuLU: Hungarian Language Understanding Benchmark Kit (2024.lrec-main)
Copied to clipboard
Noémi Ligeti-Nagy, Gergő Ferenczi, Enikő Héja, László János Laki, Noémi Vadász, Zijian Győző Yang, Tamás Váradi
| Challenge: | The Hungarian Language Understanding (HuLU) benchmark is a comprehensive assessment framework designed to evaluate the performance of neural language models on Hungary language tasks. |
| Approach: | They propose to use a framework to evaluate the performance of neural language models on Hungarian language tasks. |
| Outcome: | The framework evaluates models against Hungarian language tasks using a web service and a leaderboard. |
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021.naacl-main)
Copied to clipboard
| Challenge: | Evaluation for many natural language understanding (NLU) tasks is broken due to unreliable and biased systems scoring so high on standard benchmarks. |
| Approach: | They argue that current benchmarks fail at four criteria for evaluation . they argue that adversarial data collection does not address the causes of failures . |
| Outcome: | The proposed frameworks fail at four criteria, and adversarial data collection does not address the causes of these failures, the authors argue . restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, reliability with which they are annotated, their size, and the ways they handle social bias. |
TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in generative language models have enabled machines to generate realistic texts. |
| Approach: | They propose a benchmark environment to test the 'Turing Test' problem for neural text generation methods. |
| Outcome: | The proposed benchmark environment is based on 200K human- or machine-generated samples across 20 labels Human, GPT-1, GTP-2_small, GTT-2_medium, GPG-2_large, GGT-2_PyTorch, GGP-3, GROVER_base, griover_large and GRover_mega. |
NLPre: A Revised Approach towards Language-centric Benchmarking of Natural Language Preprocessing Systems (2024.lrec-main)
Copied to clipboard
| Challenge: | GLUE benchmarking system enables ongoing evaluation of multiple NLPre tools while credibly tracking their performance. |
| Approach: | They propose a language-centric benchmarking system that enables ongoing evaluation of multiple NLPre tools while credibly tracking their performance. |
| Outcome: | The proposed system is configured for Polish and integrated with the thoroughly assembled NLPre-PL benchmark. |
How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Transferability estimation has been a topic of great interest in computer vision fields . a lack of a comprehensive comparison between these estimation methods is a problem . |
| Approach: | They conduct a thorough survey of existing methods to find the most suitable model . they also outline difficulties of consideration of training details and applicability to text generation . |
| Outcome: | The proposed methods perform well with superiorities in effectiveness and efficiency. |