OpenTuringBench: An Open-Model-based Benchmark and Framework for Machine-Generated Text Detection and Attribution (2025.emnlp-main)
Copied to clipboard
| Challenge: | Open Large Language Models (OLLMs) are increasingly leveraged in generative AI applications, posing new challenges for detecting their outputs. |
| Approach: | They propose a benchmark to train and evaluate machine-generated text detectors on Turing Test and Authorship Attribution problems. |
| Outcome: | The proposed detector outperforms existing detectors in varying degrees of difficulty and relevance across tasks. |
Similar Papers
TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in generative language models have enabled machines to generate realistic texts. |
| Approach: | They propose a benchmark environment to test the 'Turing Test' problem for neural text generation methods. |
| Outcome: | The proposed benchmark environment is based on 200K human- or machine-generated samples across 20 labels Human, GPT-1, GTP-2_small, GTT-2_medium, GPG-2_large, GGT-2_PyTorch, GGP-3, GROVER_base, griover_large and GRover_mega. |
IMGTB: A Framework for Machine-Generated Text Detection Benchmarking (2024.acl-demos)
Copied to clipboard
| Challenge: | MGTD methods are needed in many areas, such as prevention of disinformation spreading, plagiarism, impersonation and identity theft. |
| Approach: | They propose a framework for machine-generated text detection that integrates custom methods and evaluation datasets into existing frameworks. |
| Outcome: | The proposed framework simplifies the benchmarking of machine-generated text detection methods by easy integration of custom (new) methods and evaluation datasets. |
LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing (2026.eacl-long)
Copied to clipboard
| Challenge: | a single prompt can inspire countless valid stories, making objective verification impossible. |
| Approach: | They propose a large-scale benchmark for creative writing evaluation using a reddit corpus and a 2,480-pair test set. |
| Outcome: | The proposed model outperforms existing OTS judges and generative reward models in the evaluation of creative writing. |
AttributionBench: How Hard is Automatic Attribution Evaluation? (2024.findings-acl)
Copied to clipboard
| Challenge: | generative search engines enhance the reliability of large language model responses by providing cited evidence. |
| Approach: | They propose to use a benchmark to evaluate whether a large language model supports the generated responses or not . |
| Outcome: | The proposed benchmark shows that even a fine-tuned GPT-3.5 only achieves around 80% macro-F1 under a binary classification formulation. |
AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Assertions have been the de facto collateral for hardware for over a decade. |
| Approach: | They propose a benchmark to evaluate LLMs’ effectiveness for assertion generation quantitatively. |
| Outcome: | The proposed benchmark compares state-of-the-art LLMs with existing benchmarks and shows that they generate higher fractions of functionally correct assertions. |
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models (2025.naacl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo
| Challenge: | a recent study evaluated language models using abstract evaluation criteria that lack the flexibility and granularity of human assessment. |
| Approach: | They propose a benchmark to evaluate nine distinct language models' capabilities . they use instance-specific evaluation criteria to mirror human evaluation . |
| Outcome: | The proposed benchmark evaluates nine distinct capabilities of language models across 77 tasks. |
Omni-RewardBench: Toward a Comprehensive Evaluation of Generative Reward Models Across Modalities (2026.acl-long)
Copied to clipboard
Chi-Min Chan, Yujin Zhou, Pengcheng Wen, Boqin Yin, Jiaming Ji, Juntao Dai, Wei Xue, Sirui Han, Yike Guo
| Challenge: | Existing evaluation benchmarks for ORMs are largely text-centric or limited to bimodal tasks . a new study examines the effectiveness of Omni-RewardBench for ORms across modalities . |
| Approach: | They propose a hybrid automatic-annotation and human-verification pipeline to construct high-quality evaluation data. |
| Outcome: | The proposed model is the first benchmark for comprehensive evaluation of ORMs across modalities. |
IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show pre-trained LMs store linguistic and relational knowledge . pre-training LM models can answer "fill-in-the-blank" questions based on pre-defined relations . |
| Approach: | They propose an open information extraction benchmark for pre-trained language models . they turn pre-trained LMs into zero-shot OIE systems to examine open relational information . |
| Outcome: | The proposed benchmark outperforms state-of-the-art methods on factual OIE datasets without training sets. |
MCiteBench: A Multimodal Benchmark for Generating Text with Citations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work focuses on generating citations for text-only content . experimental results reveal MLLMs struggle to ground outputs reliably when handling multimodal input . |
| Approach: | They propose a benchmark to assess the ability of MLLMs to generate text with citations in multimodal contexts. |
| Outcome: | The proposed benchmark assesses the ability of MLLMs to generate text with citations in multimodal contexts. |
Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmark (2025.acl-long)
Copied to clipboard
| Challenge: | Existing MLLMs rely on commercial models such as GPT-4o for evaluations, but they are not universally accessible. |
| Approach: | They propose a task decomposition evaluation framework based on GPT-4o to automatically construct a specialized training dataset to break down the multifaceted evaluation process into simpler sub-tasks. |
| Outcome: | The proposed framework outperforms the current state-of-the-art GPT-4o evaluation framework with over 4.6% improvement in Spearman and Kendall correlations with human judgments. |