What can Large Language Models Capture about Code Functional Equivalence? (2025.findings-naacl)
Copied to clipboard
| Challenge: | SeqCoBench is a benchmark to assess how Code-LLMs can capture code semantics. |
| Approach: | They propose a benchmark to assess how Code-LLMs capture code semantics . they use seqCoBench to evaluate whether they can discern semantically equivalent or different pairs of programs . |
| Outcome: | The proposed benchmarks show that they can capture code semantics better than classical match-based retrieval scores. |
Similar Papers
LLMs are Brittle to Simple Code Transformations: Introducing CETBench – A Benchmark for Code-Equivalence Checking (2026.findings-acl)
Copied to clipboard
| Challenge: | a new benchmarking tool for code equivalence checks the performance of LLMs. |
| Approach: | They propose a code-equivalence with transformations benchmark built from a repository of programs that may solve the same or different tasks. |
| Outcome: | The proposed approach boosts performance on the transformed pairs of programs. |
EquiBench: Benchmarking Large Language Models’ Reasoning about Program Semantics via Equivalence Checking (2025.emnlp-main)
Copied to clipboard
Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang, Yuan Liu, Thiago S. F. X. Teixeira, Diyi Yang, Ke Wang, Alex Aiken
| Challenge: | EquiBench is a new benchmark to evaluate large language models' ability to reason about program semantics . Unlike natural language, code is executable. |
| Approach: | They propose a benchmark to evaluate large language models through equivalence checking . EquiBench consists of 2400 program pairs across four languages and six categories . |
| Outcome: | The proposed benchmark consists of 2400 program pairs across four languages and six categories. |
LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking (2024.eacl-demo)
Copied to clipboard
Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali
| Challenge: | Recent development and success of Large Language Models necessitate evaluation of their performance across diverse NLP tasks in different languages. |
| Approach: | They propose a framework that can be customized to evaluate LLMs for any NLP task, regardless of language. |
| Outcome: | The LLMeBench framework can be customized to evaluate LLMs for any NLP task, regardless of language. |
Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly being used to automate programming tasks. |
| Approach: | They propose a benchmark to evaluate LLMs' reasoning abilities on program semantics. |
| Outcome: | The proposed benchmark shows that LLMs perform well with simple control flows but struggle with more complex structures, especially loops, even with advanced prompting. |
CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models (LLMs) are limited by their narrow language pairs and tasks, failing to adequately assess their code-mixing abilities. |
| Approach: | They propose a benchmark to assess large language models' (LLMs) code-mixing abilities that covers eight tasks and 18 languages from seven language families. |
| Outcome: | The proposed method combines word substitution with GPT-4 prompting to generate large-scale synthetic code-mixed texts. |
Can Language Models Replace Programmers for Coding? REPOCOD Says ‘Not Yet’ (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for code generation use short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks. |
| Approach: | They propose a Python code-generation benchmark that contains 980 whole-function generation tasks with realistic dependencies from 11 popular projects. |
| Outcome: | The proposed benchmarks are short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks. |
API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs (2024.acl-long)
Copied to clipboard
Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Vernon Austel, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, Luis Lastras
| Challenge: | Existing methods to train and test large language models that involve calls to tools and APIs are lacking. |
| Approach: | They propose a large corpora for training and systematic testing of tool-augmented LLMs. |
| Outcome: | The proposed datasets mimic real-world scenarios involving API-tasks and slot filling. |
Out-of-Context Reasoning in Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a lightweight technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
| Approach: | They propose a lightweight technique that trains only new token embeddings on axioms . they train only new embeddables and evaluate them on unseen tasks . |
| Outcome: | The proposed technique trains only new token embeddings on axioms and evaluates them on unseen tasks. |
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are impressive in solving tasks, but they can quickly be outdated after deployment. |
| Approach: | They provide a review of recent advances in aligning deployed large language models with the ever-changing world knowledge. |
| Outcome: | The proposed models can be used to perform various tasks directly through in-context learning or for further fine-tuning for domain-specific uses. |
CODEMENV: Benchmarking Large Language Models on Code Migration (2025.findings-acl)
Copied to clipboard
Keyuan Cheng, Xudong Shen, Yihao Yang, TengyueWang TengyueWang, Yang Cao, Muhammad Asif Ali, Hanbin Wang, Lijie Hu, Di Wang
| Challenge: | Large language models (LLMs) have demonstrated remarkable proficiency in handling a wide range of tasks within the software engineering domain, but their ability to perform code migration—adapting code to different environments—remains underexplored. |
| Approach: | They propose a benchmark to evaluate large language models’ performance in handling code migration tasks. |
| Outcome: | The proposed benchmark comprises 922 data points across 19 Python and Java packages and offers three tasks to systematically evaluate code migration: identifying version-incompatible functions, determining function changes, and adapting code to target environments. |