Papers with meta-analysis
It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance (2023.findings-acl)
Copied to clipboard
| Challenge: | a meta-analysis and survey of practitioners reveal that benchmarks suffer from operationalization disagreements. |
| Approach: | They propose a taxonomy of disagreement to explain disagreements in NLP benchmarks . they propose defining how tasks are conceptualized and operationalizing benchmarks to document their limitations. |
| Outcome: | The proposed taxonomy identifies two types of disagreements among NLP practitioners . it shows that benchmarks are not clearly conceptualized and suffer from operationalization disagreements . |
Let’s Stop Incorrect Comparisons in End-to-end Relation Extraction! (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature on Relation Extraction (RE) uses multiple evaluation setups to compare performance. |
| Approach: | They propose to quantify the most common comparison mistake and evaluate it leads to overestimating the final RE performance by around 5% on ACE05. |
| Outcome: | The proposed meta-analysis overestimates the final RE performance by around 5% on ACE05. |
What is ”Typological Diversity” in NLP? (2024.emnlp-main)
Copied to clipboard
| Challenge: | linguistic typology is commonly used to motivate language selections, but there are no set definitions or criteria for such claims. |
| Approach: | They propose to use linguistic typology to motivate language selections on the basis that a broad typological sample ought to imply generalization across a wide range of languages. |
| Outcome: | The proposed measures show that skewed language selection can lead to overestimated multilingual performance. |
BUST: Benchmark for the evaluation of detectors of LLM-Generated Text (2024.naacl-long)
Copied to clipboard
| Challenge: | Using the benchmark, we evaluated 5 detectors and found substantial performance variance across tasks. |
| Approach: | They propose to evaluate detectors of texts generated by instruction-tuned large language models (LLMs) using a benchmark dataset, they evaluated 5 detectors and found substantial performance variance across tasks. |
| Outcome: | The proposed benchmarks evaluated 5 detectors and found substantial performance variance across tasks. |
On Learning to Summarize with Large Language Models as References (2024.naacl-long)
Copied to clipboard
Yixin Liu, Kejian Shi, Katherine He, Longtian Ye, Alexander Fabbri, Pengfei Liu, Dragomir Radev, Arman Cohan
| Challenge: | Recent studies have found that summaries generated by large language models (LLMs) are favored by human annotators when compared to reference summary from widely used summarization datasets. |
| Approach: | They propose to use large language models (LLMs) as reference learning settings for smaller text summarization models to investigate whether their performance can be substantially improved. |
| Outcome: | The proposed model outperforms standard supervised fine-tuning and human evaluations while retaining human-level performance. |
A Call for Standardization and Validation of Text Style Transfer Evaluation (2023.findings-acl)
Copied to clipboard
| Challenge: | Text style transfer (TST) evaluation is inconsistent in practice. |
| Approach: | They conduct a meta-analysis on human and automated TST evaluation and experimentation . they find a standardization gap and a validation gap in the field . |
| Outcome: | The authors find that evaluation procedures are inconsistent and that they need to improve on them. |
What is the Real Intention behind this Question? Dataset Collection and Intention Classification (2023.acl-long)
Copied to clipboard
| Challenge: | Using the Wikipedia discussions, we identified positive/neutral and negative intentions in questions . questions can also reflect implicit offenses such as highlighting one’s lack of knowledge or bolstering an alleged superior knowledge, which can lead to conflict in conversations. |
| Approach: | They propose to use a dataset to identify questions with positive/neutral and negative intentions and the underlying intention categories within each group to highlight tacit and apparent intents. |
| Outcome: | The proposed method highlights tacit and apparent intents and uses Transformers augmented by TF-IDF-based features to classify the main intention categories. |
Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Downstream scaling laws aim to predict task performance at larger scales from the model’s performance at smaller scales. |
| Approach: | They conduct a meta-analysis of existing data on downstream scaling laws and find that predictable scaling only occurs in a minority of cases: 39% of the time. |
| Outcome: | The proposed scaling laws only occur in a minority of cases, and seemingly benign changes to the experimental setting can completely change the scaling behavior. |