LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing (2026.eacl-long)
Copied to clipboard
| Challenge: | a single prompt can inspire countless valid stories, making objective verification impossible. |
| Approach: | They propose a large-scale benchmark for creative writing evaluation using a reddit corpus and a 2,480-pair test set. |
| Outcome: | The proposed model outperforms existing OTS judges and generative reward models in the evaluation of creative writing. |
Similar Papers
WaterBench: Towards Holistic Evaluation of Watermarks for Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent studies have developed watermarking algorithms which restrict the generation process to leave an invisible trace for watermark detection. |
| Approach: | They propose a benchmarking procedure that compares different methods to ensure consistent watermarking strength and jointly evaluates their generation and detection performance. |
| Outcome: | The proposed benchmark compares 4 open-source watermarks on 2 LLMs under 2 watermarking strengths and observes the common struggles for current methods on maintaining the generation quality. |
TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in generative language models have enabled machines to generate realistic texts. |
| Approach: | They propose a benchmark environment to test the 'Turing Test' problem for neural text generation methods. |
| Outcome: | The proposed benchmark environment is based on 200K human- or machine-generated samples across 20 labels Human, GPT-1, GTP-2_small, GTT-2_medium, GPG-2_large, GGT-2_PyTorch, GGP-3, GROVER_base, griover_large and GRover_mega. |
CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used to judge code, but their reliability remains poorly understood. |
| Approach: | They propose a benchmark to evaluate Large Language Models as code judges . they find that small reasoning models outperform larger non-reasoning models . |
| Outcome: | The proposed benchmark evaluates LLM-as-a-Judge models across three coding tasks. |
HoWToBench: Holistic Evaluation for LLM’s Capability in Human-level Writing using Tree of Writing (2026.acl-long)
Copied to clipboard
Andrew Zhuoer Feng, Cunxiang Wang, Yu Luo, Lin Fan, Irene Zhou, Zikang Wang, Xiaotao Gu, Jie Tang, Hongning Wang, Minlie Huang
| Challenge: | Evaluating the writing capabilities of large language models remains a significant challenge due to the multidimensional nature of writing skills and the limitations of existing metrics. |
| Approach: | They propose to model the aggregation weights of sub-features in a tree-structured workflow and propose a Chinese writing benchmark that mitigates biases. |
| Outcome: | The proposed tree-of-writing (ToW) measures the writing capabilities of large language models (LLMs) in Chinese and shows that it mitigates biases and achieves a *0.93* Pearson correlation with human judgments. |
AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Assertions have been the de facto collateral for hardware for over a decade. |
| Approach: | They propose a benchmark to evaluate LLMs’ effectiveness for assertion generation quantitatively. |
| Outcome: | The proposed benchmark compares state-of-the-art LLMs with existing benchmarks and shows that they generate higher fractions of functionally correct assertions. |
Towards A “Novel” Benchmark: Evaluating Literary Fiction with Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) context windows have enabled them to process inputs over 100K tokens and generate outputs of up to 10K token. |
| Approach: | They propose a multi-level evaluation framework that incorporates ten metrics across the Macro, Meso, and Micro levels and an annotated fiction dataset. |
| Outcome: | The proposed framework incorporates ten metrics across the Macro, Meso, and Micro levels and is based on a human-human-AI dataset. |
ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are transforming diverse fields and gaining increasing influence as human proxies. |
| Approach: | They propose a psychometric evaluation pipeline grounded in realistic human-AI interactions to probe value orientations and novel tasks for evaluating value understanding in an open-ended value space. |
| Outcome: | The proposed evaluation pipeline is grounded in realistic human-AI interactions and performs tasks that approximate expert conclusions in value-related extraction and generation tasks. |
RubricBench: Aligning Model-Generated Rubrics with Human Standards (2026.acl-long)
Copied to clipboard
Junyi Zhou, Qiyuan Zhang, Yufei Wang, Fuyuan Lyu, Yidong Ming, Can Xu, Qingfeng Sun, Kai Zheng, Peng Kang, Xue Liu, Chen Ma
| Challenge: | Existing benchmarks lack discriminative complexity and ground-truth rubric annotations required for rigorous evaluation. |
| Approach: | They propose a curated benchmark with 1,147 pairwise comparisons to assess the reliability of rubric-based evaluation. |
| Outcome: | The proposed benchmarks show that they support diverse domains, exhibit discriminative ability, provide high-quality annotations, and include human-authored rubrics. |
CriticBench: Benchmarking LLMs for Critique-Correct Reasoning (2024.findings-acl)
Copied to clipboard
| Challenge: | CriticBench is a benchmark designed to assess LLMs’ abilities to critique and refine their reasoning across a variety of tasks. |
| Approach: | They propose a benchmark to assess LLMs' ability to critique and correct reasoning across a variety of tasks. |
| Outcome: | The proposed benchmark examines the performance of 17 large language models in generation, critique, and correction reasoning. |
WebNovelBench: Placing LLM Novelists on the Web Novel Distribution (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing benchmarks for long-form novel generation lack scale, diversity, or objective measures. |
| Approach: | They propose a framework that assesses long-form novel generation using an LLM-as-Judge approach. |
| Outcome: | The proposed framework differentiates between human-written masterpieces, popular web novels, and LLM-generated content. |