WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics (2026.acl-long)
Copied to clipboard
| Challenge: | Web applications (web apps) are a key arena for large language models to demonstrate their code generation capabilities and commercial potential. |
| Approach: | a new benchmark for large language models (LLMs) is designed to provide real-world user requirements and generalizable evaluation metrics. |
| Outcome: | a new benchmark for large language models (LLMs) provides a real-world, generalizable, and interpretable evaluation score . the benchmark measures user requirements, expression styles and human-preference-aligned weights . a web application can be used to demonstrate its commercial potential, authors say . |
Similar Papers
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models (2025.naacl-long)
Copied to clipboard
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo
| Challenge: | a recent study evaluated language models using abstract evaluation criteria that lack the flexibility and granularity of human assessment. |
| Approach: | They propose a benchmark to evaluate nine distinct language models' capabilities . they use instance-specific evaluation criteria to mirror human evaluation . |
| Outcome: | The proposed benchmark evaluates nine distinct capabilities of language models across 77 tasks. |
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2024.acl-long)
Copied to clipboard
Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, Shuiguang Deng
| Challenge: | Existing benchmarks for evaluating the code understanding and generation capacities of Large Language Models are insufficient . existing benchmarks focus on a narrow range of popular programming languages and specific tasks . |
| Approach: | They propose an execution-based, multilingual, multitask evaluation benchmark for LLMs . they evaluate coding performance from three dimensions: length, difficulty, efficiency . |
| Outcome: | The proposed benchmark covers 43 programming languages and eight coding tasks. |
WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models focus on webpage generation outcomes. |
| Approach: | They propose a multi-view evaluation framework to evaluate MLLMs in four key areas: WebUI Perception, HTML Programming, WebUI-HTML Understanding, and WebUI to code. |
| Outcome: | The proposed framework evaluates MLLMs in four key areas: WebUI Perception, HTML Programming, WebUI-HTML Understanding, and WebUI to code. |
CodeJudge: Evaluating Code Generation with Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown promising performance in code generation, but how to reliably evaluate code generated by LLMs remains a challenging problem. |
| Approach: | They propose a framework that leverages Large Language Models to evaluate the semantic correctness of generated code without the need for test cases. |
| Outcome: | The proposed framework outperforms existing methods on four code generation datasets and five programming languages. |
XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have shown impressive abilities in generating codes from natural language descriptions, repairing buggy codes, translating codes between languages, and retrieving relevant code segments. |
| Approach: | They propose to use a multilingual multitask benchmark to evaluate large language models that can generate codes from natural language descriptions, repair buggy codes, and translate between languages. |
| Outcome: | The proposed model performs 7 tasks covering up to 11 languages with execution-level parallelism and 25 M document-level coding examples (16.5 B tokens) |
CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used to judge code, but their reliability remains poorly understood. |
| Approach: | They propose a benchmark to evaluate Large Language Models as code judges . they find that small reasoning models outperform larger non-reasoning models . |
| Outcome: | The proposed benchmark evaluates LLM-as-a-Judge models across three coding tasks. |
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code (2022.emnlp-demos)
Copied to clipboard
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo F . R. Ribeiro, Lewis Tunstall, Li Zhang, Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, Yufang Hou
| Challenge: | Evaluations in machine learning rarely use the latest metrics, datasets, or human evaluation in favor of remaining compatible with prior work. |
| Approach: | They propose to use the Generation, Evaluation, and Metrics Benchmark to integrate new evaluation methods into existing evaluations. |
| Outcome: | The proposed evaluation infrastructure bridges the gap between the advantages of leaderboards and in-depth and evolving evaluations by allowing model developers to benefit from each other's work. |
WaterBench: Towards Holistic Evaluation of Watermarks for Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent studies have developed watermarking algorithms which restrict the generation process to leave an invisible trace for watermark detection. |
| Approach: | They propose a benchmarking procedure that compares different methods to ensure consistent watermarking strength and jointly evaluates their generation and detection performance. |
| Outcome: | The proposed benchmark compares 4 open-source watermarks on 2 LLMs under 2 watermarking strengths and observes the common struggles for current methods on maintaining the generation quality. |
A Tutorial on Evaluation Metrics used in Natural Language Generation (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
| Approach: | This tutorial presents the evolution of automatic evaluation metrics to their current state . it aims to assess the extent of scientific progress made and identify areas/components that need improvement . |
| Outcome: | This tutorial presents the evolution of automatic evaluation metrics to their current state along with emerging trends in this field. |
WebNovelBench: Placing LLM Novelists on the Web Novel Distribution (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing benchmarks for long-form novel generation lack scale, diversity, or objective measures. |
| Approach: | They propose a framework that assesses long-form novel generation using an LLM-as-Judge approach. |
| Outcome: | The proposed framework differentiates between human-written masterpieces, popular web novels, and LLM-generated content. |