DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches focus on syntactic correctness through synthetic micro-benchmarks or subjective human ratings, despite semantic fidelity and usability. |
| Approach: | They propose a framework that enables effective evaluation of decompilers in reverse engineering workflows . they compare six industrial-strength decompils and six recent LLM-powered approaches . |
| Outcome: | The proposed framework outperforms commercial tools in code understandability despite lower functionality correctness . it shows that it can transform human-centric reverse engineering workflows . |
Similar Papers
CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used to judge code, but their reliability remains poorly understood. |
| Approach: | They propose a benchmark to evaluate Large Language Models as code judges . they find that small reasoning models outperform larger non-reasoning models . |
| Outcome: | The proposed benchmark evaluates LLM-as-a-Judge models across three coding tasks. |
SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks have highlighted character-level tasks as lacking practical relevance . many real-world applications rely heavily on precise sub-token understanding . |
| Approach: | They propose a benchmark that assesses sub-token understanding through practical tasks . they examine the impact of test-time scaling on sub-word reasoning . |
| Outcome: | The proposed benchmark assesses sub-token understanding through practical tasks . it includes ten tasks across four domains and isolates tokenization-related failures . |
Are LLM-based Evaluators Confusing NLG Quality Criteria? (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that LLMs confuse evaluation criteria, which reduces their reliability. |
| Approach: | They propose a hierarchical classification system for 11 common aspects with corresponding different evaluation criteria. |
| Outcome: | The proposed system is based on 11 common aspects with different evaluation criteria. |
EntroBench: Evaluating LLM Watermarking Under Multi-Entropy Scenarios and Practical User Operations (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluations of large language models (LLMs) watermarking are limited to fixed entropy settings. |
| Approach: | They propose a benchmark for LLM watermarking that systematically covers three entropy levels and seven representative tasks. |
| Outcome: | The proposed framework covers three entropy levels and seven representative tasks. |
When Benchmarks Leak: Inference-Time Decontamination for LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | a large number of large language models (LLMs) are being evaluated for their performance, but their reliability is threatened by test set contamination. |
| Approach: | They propose a framework that decontaminates large language models by applying small perturbations to the input embedding space. |
| Outcome: | The proposed framework achieves strong decontamination effectiveness while incurring minimal degradation in benign utility. |
RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models fail to capture complex interplay between functionality and security. |
| Approach: | They propose a benchmark for secure code generation constructed from real-world, high-risk Java repositories. |
| Outcome: | The proposed benchmarks highlight the gap between functional and secure code generation in LLMs. |
FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents (2024.findings-emnlp)
Copied to clipboard
| Challenge: | LLM-based agents are susceptible to undesired planning hallucinations when lacking specific knowledge for expertise-intensive tasks. |
| Approach: | They propose a benchmark to evaluate the efficacy of workflow-guided agent planning by formalizing different formats of workflow knowledge. |
| Outcome: | The proposed benchmark aims to improve the planning reliability of LLM-based agents by incorporating external workflow-related knowledge. |
WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics (2026.acl-long)
Copied to clipboard
| Challenge: | Web applications (web apps) are a key arena for large language models to demonstrate their code generation capabilities and commercial potential. |
| Approach: | a new benchmark for large language models (LLMs) is designed to provide real-world user requirements and generalizable evaluation metrics. |
| Outcome: | a new benchmark for large language models (LLMs) provides a real-world, generalizable, and interpretable evaluation score . the benchmark measures user requirements, expression styles and human-preference-aligned weights . a web application can be used to demonstrate its commercial potential, authors say . |
A Thorough Examination of Decoding Methods in the Era of LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Decoding methods are essential for converting language models from next-token predictors into practical task solvers. |
| Approach: | They propose to evaluate decoding methods in general-purpose large language models . they find that decoding method performance is notably task-dependent . |
| Outcome: | The proposed methods perform task-dependently and are influenced by alignment, model size, and quantization. |
CoreCodeBench: Decoupling Code Intelligence via Fine-Grained Repository-Level Tasks (2026.acl-long)
Copied to clipboard
Lingyue Fu, Hao Guan, Bolun Zhang, Haowei Yuan, Yaoming Zhu, Lin Qiu, ZongYu Wang, Xuezhi Cao, Xunliang Cai, Weiwen Liu, Weinan Zhang, Yong Yu
| Challenge: | Existing large language models for software engineering rely on coarse-grained pass rates obscuring specific cognitive bottlenecks. |
| Approach: | They propose a repository-level benchmark that dissects coding capabilities through atomized tasks. |
| Outcome: | The proposed framework achieves a 78.55% validity yield, surpassing the 31.7% retention rate of SWE-bench-Verified. |