Targeting the Benchmark: On Methodology in Current Natural Language Processing Research (2021.acl-short)
Copied to clipboard
| Challenge: | a language benchmark is a task devised that is restricted enough to be managable with current methods, but is deemed challenging enough to serve as a benchmark. |
| Approach: | They propose to use a language task as a benchmark and a baseline model to argue it is challenging enough to be a good one. |
| Outcome: | The proposed language benchmarks are based on a dataset and a language task . the proposed benchmarks can be used to measure progress towards the goal of the research . |
Similar Papers
High Performance Natural Language Processing (2020.emnlp-tutorials)
Copied to clipboard
| Challenge: | a tutorial on scaling natural language processing will recapitulate the state-of-the-art in the field . |
| Approach: | This cutting-edge tutorial recapitulates the state-of-the-art in natural language processing with scale in perspective. |
| Outcome: | This cutting-edge tutorial recapitulates the state-of-the-art in natural language processing with scale in perspective. |
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021.naacl-main)
Copied to clipboard
| Challenge: | Evaluation for many natural language understanding (NLU) tasks is broken due to unreliable and biased systems scoring so high on standard benchmarks. |
| Approach: | They argue that current benchmarks fail at four criteria for evaluation . they argue that adversarial data collection does not address the causes of failures . |
| Outcome: | The proposed frameworks fail at four criteria, and adversarial data collection does not address the causes of these failures, the authors argue . restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, reliability with which they are annotated, their size, and the ways they handle social bias. |
Navigating the Modern Evaluation Landscape: Considerations in Benchmarks and Frameworks for Large Language Models (LLMs) (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | General-purpose Language Models have changed the world of Natural Language Processing, if not the world itself. |
| Approach: | This tutorial will lay the foundations and explain the basics of evaluation and compare traditional methods to newly developed methods. |
| Outcome: | The tutorial assumes little familiarity with metrics, datasets, prompts and benchmarks . it will compare traditional methods to newly developed methods . |
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track (2024.emnlp-industry)
Copied to clipboard
| Challenge: | EMNLP has received a record 334 submissions for the Industry Track this year . the track is aimed at highlighting key insights and new research challenges . |
| Approach: | EMNLP 2024 Industry Track 2024 is the third edition of the track . the track is aimed at highlighting key insights and new research challenges . submissions are more than double that of last year . |
| Outcome: | EMNLP 2024 Industry Track 2024 received 334 submissions, more than double last year . the track aims to be the premier forum for knowledge sharing between academia and industry . |
In Benchmarks We Trust ... Or Not? (2025.emnlp-main)
Copied to clipboard
Ine Gevers, Victor De Marez, Jens Van Nooten, Jens Lemmens, Andriy Kosar, Ehsan Lotfi, Nikolay Banar, Pieter Fivez, Luna De Bruyne, Walter Daelemans
| Challenge: | Existing benchmarks for Large Language Models (LLMs) are inadequate and lack a clear solution. |
| Approach: | They propose checklists to cover all aspects of benchmarking issues, both for benchmark creation and usage. |
| Outcome: | The proposed checklists cover all aspects of benchmarking issues, both for benchmark creation and usage. |
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track (2023.emnlp-industry)
Copied to clipboard
| Challenge: | . - (EN) |
| Approach: | . - (EN) |
| Outcome: | . - (EN) |
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track (2022.emnlp-industry)
Copied to clipboard
| Challenge: | . - (EN) |
| Approach: | . - (EN) |
| Outcome: | . - (EN) |
Defining a New NLP Playground (2023.findings-emnlp)
Copied to clipboard
Sha Li, Chi Han, Pengfei Yu, Carl Edwards, Manling Li, Xingyao Wang, Yi Fung, Charles Yu, Joel Tetreault, Eduard Hovy, Heng Ji
| Challenge: | Recent explosion of performance of large language models (LLMs) has changed the field more abruptly and seismically than any other shift in the field’s 80 year history. |
| Approach: | They propose 20+ PhD-dissertation-worthy research directions to define a new NLP playground by combining theoretical analysis, new and challenging problems, learning paradigms and interdisciplinary applications. |
| Outcome: | The proposed research will cover theoretical analysis, new and challenging problems, learning paradigms and interdisciplinary applications. |
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop (2021.acl-srw)
Copied to clipboard
| Challenge: | ACL-IJCNLP 2021 Student Research Workshop is a forum for student researchers in computational linguistics and natural language processing. |
| Approach: | the ACL-IJCNLP 2021 Student Research Workshop is a forum for student researchers in computational linguistics and natural language processing. |
| Outcome: | the ACL-IJCNLP 2021 Student Research Workshop is a forum for student researchers . the workshop has received 114 submissions including 109 research papers and 5 thesis proposals . |