LLMs are Brittle to Simple Code Transformations: Introducing CETBench – A Benchmark for Code-Equivalence Checking (2026.findings-acl)
Copied to clipboard
| Challenge: | a new benchmarking tool for code equivalence checks the performance of LLMs. |
| Approach: | They propose a code-equivalence with transformations benchmark built from a repository of programs that may solve the same or different tasks. |
| Outcome: | The proposed approach boosts performance on the transformed pairs of programs. |
Similar Papers
What can Large Language Models Capture about Code Functional Equivalence? (2025.findings-naacl)
Copied to clipboard
| Challenge: | SeqCoBench is a benchmark to assess how Code-LLMs can capture code semantics. |
| Approach: | They propose a benchmark to assess how Code-LLMs capture code semantics . they use seqCoBench to evaluate whether they can discern semantically equivalent or different pairs of programs . |
| Outcome: | The proposed benchmarks show that they can capture code semantics better than classical match-based retrieval scores. |
EquiBench: Benchmarking Large Language Models’ Reasoning about Program Semantics via Equivalence Checking (2025.emnlp-main)
Copied to clipboard
Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang, Yuan Liu, Thiago S. F. X. Teixeira, Diyi Yang, Ke Wang, Alex Aiken
| Challenge: | EquiBench is a new benchmark to evaluate large language models' ability to reason about program semantics . Unlike natural language, code is executable. |
| Approach: | They propose a benchmark to evaluate large language models through equivalence checking . EquiBench consists of 2400 program pairs across four languages and six categories . |
| Outcome: | The proposed benchmark consists of 2400 program pairs across four languages and six categories. |
CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used to judge code, but their reliability remains poorly understood. |
| Approach: | They propose a benchmark to evaluate Large Language Models as code judges . they find that small reasoning models outperform larger non-reasoning models . |
| Outcome: | The proposed benchmark evaluates LLM-as-a-Judge models across three coding tasks. |
CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models (LLMs) are limited by their narrow language pairs and tasks, failing to adequately assess their code-mixing abilities. |
| Approach: | They propose a benchmark to assess large language models' (LLMs) code-mixing abilities that covers eight tasks and 18 languages from seven language families. |
| Outcome: | The proposed method combines word substitution with GPT-4 prompting to generate large-scale synthetic code-mixed texts. |
Metric Calculating Benchmark: Code-Verifiable Complicate Instruction Following Benchmark for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent frontier-level LLMs have saturated many previously difficult benchmarks, leaving little room for further differentiation. |
| Approach: | They propose a benchmark to evaluate whether LLMs can execute string-matching NLP metrics by strictly following step-by-step instructions. |
| Outcome: | The proposed benchmarks show that they can perform step-by-step execution, instruction adherence, numerical computation, and long-range consistency in handling intermediate results. |
CRUXEVAL-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution (2025.acl-long)
Copied to clipboard
Ruiyang Xu, Jialun Cao, Yaojie Lu, Ming Wen, Hongyu Lin, Xianpei Han, Ben He, Shing-Chi Cheung, Le Sun
| Challenge: | Existing code benchmarks focus on code generation, while those for code reasoning are insufficient. |
| Approach: | They propose a multi-lingual code reasoning benchmark that contains 19 programming languages and at least 600 subjects for each language. |
| Outcome: | The proposed model trains on Python and achieves 34.4% Pass@1 in other languages, revealing the cross-language generalization of LLMs. |
LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking (2024.eacl-demo)
Copied to clipboard
Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali
| Challenge: | Recent development and success of Large Language Models necessitate evaluation of their performance across diverse NLP tasks in different languages. |
| Approach: | They propose a framework that can be customized to evaluate LLMs for any NLP task, regardless of language. |
| Outcome: | The LLMeBench framework can be customized to evaluate LLMs for any NLP task, regardless of language. |
CoreCodeBench: Decoupling Code Intelligence via Fine-Grained Repository-Level Tasks (2026.acl-long)
Copied to clipboard
Lingyue Fu, Hao Guan, Bolun Zhang, Haowei Yuan, Yaoming Zhu, Lin Qiu, ZongYu Wang, Xuezhi Cao, Xunliang Cai, Weiwen Liu, Weinan Zhang, Yong Yu
| Challenge: | Existing large language models for software engineering rely on coarse-grained pass rates obscuring specific cognitive bottlenecks. |
| Approach: | They propose a repository-level benchmark that dissects coding capabilities through atomized tasks. |
| Outcome: | The proposed framework achieves a 78.55% validity yield, surpassing the 31.7% retention rate of SWE-bench-Verified. |
RiddleBench: A New Generative Reasoning Benchmark for LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) show remarkable capabilities, but complex reasoning skills require deeper investigation. |
| Approach: | They propose a benchmark of 1,737 puzzles to test reasoning beyond simple pattern matching. |
| Outcome: | The proposed model performs poorly when faced with reordered constraints or irrelevant information. |
CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing code translation datasets focus on a single pair of programming languages . early software systems are developed using programming languages such as Fortran and COBOL . |
| Approach: | They propose a large-scale comprehensive benchmark that supports the largest variety of programming languages for code translation. |
| Outcome: | The proposed framework supports translations between multiple programming languages and a cross-framework dataset for deep learning code across different frameworks. |