PYTHON

GR-NLP-TOOLKIT: An Open-Source NLP Toolkit for Modern Greek (2025.coling-demos)

Challenge:	Existing evaluations focus on functional correctness or syntactic validity, overlooking how LLMs make critical design choices such as which library or programming language to use.
Approach:	They conduct the first systematic study of LLMs’ preferences for libraries and programming languages when generating code, covering eight different LLM.
Outcome:	The proposed benchmarks show that LLMs prioritize familiarity and popularity over suitability and task-specific optimality.

Copied to clipboard

Lefteris Loukas, Nikolaos Smyrnioudis, Chrysa Dikonomaki, Spiros Barbakos, Anastasios Toumazatos, John Koutsikakis, Manolis Kyriakakis, Mary Georgiou, Stavros Vassos, John Pavlopoulos, Ion Androutsopoulos

Syntactic Text Processing

Challenge:	GR-NLP-TOOLKIT is an open-source natural language processing toolkit for modern Greek.
Approach:	They present GR-NLP-TOOLKIT, an open-source natural language processing toolkit for Greek.
Outcome:	The toolkit provides state-of-the-art performance in five core NLP tasks . it can be easily installed in Python and is accessible through a demonstration platform on HuggingFace .

The Larger they are, the Harder they Fail: Language Models do not Recognize Identifier Swaps in Python (2023.findings-acl)

Copied to clipboard

Antonio Valerio Miceli Barone, Fazl Barez, Shay B. Cohen, Ioannis Konstas

DistaLs: a Comprehensive Collection of Language Distance Measures (2025.emnlp-demos)

Challenge:	Large Language Models (LLMs) are used for programming tasks but lack a deep understanding of the content they manipulate.
Approach:	They show that LLMs fail to correctly generate correct Python code when default function names are swapped . they also show that they become more confident in their incorrect predictions as the model size increases .
Outcome:	The proposed models fail to generate correct Python code when default function names are swapped, and become more confident in their incorrect predictions as the model size increases.

Copied to clipboard

Rob Van Der Goot, Esther Ploeger, Verena Blaschke, Tanja Samardzic

Challenge:	Existing work on how to measure distances between languages has focused on intuition and typological distance.
Approach:	They propose a toolkit that provides users with easy access to language distance measures.
Outcome:	The proposed toolkit provides easy access to a wide variety of language distance measures.

string2string: A Modern Python Library for String-to-String Algorithms (2024.acl-demos)

Copied to clipboard

Mirac Suzgun, Stuart Shieber, Dan Jurafsky

LOA: Logical Optimal Actions for Text-based Interaction Games (2021.acl-demo)

Challenge:	Notable algorithms include the Smith-Waterman algorithm for pairwise local alignment, the Hirschberg algorithm for global alignment, and the Wagner-Fischer algorithm for edit distance.
Approach:	string2string is an open-source library that offers efficient algorithms for string-to-string problems.
Outcome:	string2string is an open-source library that offers efficient algorithms for string-to-string problems.

Copied to clipboard

Daiki Kimura, Subhajit Chaudhury, Masaki Ono, Michiaki Tatsubori, Don Joven Agravante, Asim Munawar, Akifumi Wachi, Ryosuke Kohita, Alexander Gray

Knowledge Representation

Challenge:	et al., 2019) have proposed a neuro-symbolic approach for reinforcement learning in non-simultaneous environments.
Approach:	They propose an action decision architecture with a neuro-symbolic framework for natural language interaction games.
Outcome:	The proposed framework provides an open-source implementation in Python for the reinforcement learning environment to facilitate an experiment for studying neuro-symbolic agents.

PyOpenDial: A Python-based Domain-Independent Toolkit for Developing Spoken Dialogue Systems with Probabilistic Rules (D19-3)

Copied to clipboard

Youngsoo Jang, Jongmin Lee, Jaeyoung Park, Kyeng-Hun Lee, Pierre Lison, Kee-Eung Kim

Natural Language Interfaces

Challenge:	a recent development of spoken dialogue systems has enabled deep learning to achieve state-of-the-art performance.
Approach:	They propose a Python-based domain-independent, open-source toolkit for spoken dialogue systems.
Outcome:	The proposed toolkit extends OpenDial's Java-based architecture and provides new functions for neural dialogue state tracking and action planning.

StRuCom: A Novel Dataset of Structured Code Comments in Russian (2025.acl-srw)

Copied to clipboard

Maria Dziuba, Valentin Malykh

Multilinguality

Challenge:	Existing machine learning models for code comment generation are poorly suited for Russian . existing datasets that contain simple comments and docstrings in English are not suitable for function-level documentation generation.
Approach:	They propose a dataset specifically designed for Russian code documentation.
Outcome:	The first large-scale dataset specifically designed for Russian code documentation is based on human-written comments from GitHub repositories with synthetically generated ones.

Meta-Reasoning: Semantics-Symbol Deconstruction for Large Language Models (2024.findings-acl)

Copied to clipboard

Yiming Wang, Zhuosheng Zhang, Pei Zhang, Baosong Yang, Rui Wang

PyMarian: Fast Neural Machine Translation and Evaluation in Python (2024.emnlp-demo)

Challenge:	Existing methods rely on syntactically mapping natural languages to complete formal languages like Python and SQL.
Approach:	They propose to deconstruct reasoning-independent semantic information into generic symbolic representations, thereby efficiently capturing more generalized reasoning knowledge.
Outcome:	The proposed method improves in-context reasoning accuracy, learning efficiency, out-of-domain generalization, and output stability compared to the Chain-of thought technique.

Copied to clipboard

Thamme Gowda, Roman Grundkiewicz, Elijah Rippeth, Matt Post, Marcin Junczys-Dowmunt

Machine Translation

Challenge:	a Python interface to Marian NMT is available in PyPI via pip install pymarian . the interface provides a speedup factor of up to 7.8 the existing implementations .
Approach:	They propose a Python interface to Marian NMT, a C++-based training and inference toolkit for sequence-to-sequence models.
Outcome:	The proposed interface enables models trained with Marian to be connected to Python tools with a speedup factor of up to 7.8 the existing implementations.

FORG3D: Flexible Object Rendering for Generating Vision-Language Spatial Reasoning Data from 3D Scenes (2025.acl-demo)

Copied to clipboard

Oscar Pang, Freda Shi

Modeling Hierarchical Syntax Structure with Triplet Position for Source Code Summarization (2022.acl-long)

Challenge:	FORG3D synthesizes vision-language data for use in cognitive experiments . it provides precise control over object placement, orientation, and camera configurations .
Approach:	They propose a 3D rendering toolkit that synthesizes vision-language data with Blender and Python.
Outcome:	The toolkit synthesizes vision-language data for two primary purposes: supporting cognitive experiments and improving visual reasoning capabilities of large vision-linguistic models.

Copied to clipboard

Juncai Guo, Jin Liu, Yao Wan, Li Li, Pingyi Zhou

LM-Polygraph: Uncertainty Estimation for Language Models (2023.emnlp-demo)

Challenge:	Existing approaches to describe the syntax structure of code are lacking in retaining the semantic structure of source code.
Approach:	They propose to use a triplet position to model hierarchical syntax structure of code by introducing a graph neural network and Transformer to preserve the structural and sequential information of code.
Outcome:	The proposed model preserves the structural and sequential information of code and a pointer-generator network that pays attention to both the structure and sequential tokens of code for a better summary generation.

Copied to clipboard

Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Artem Shelmanov

MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoning (2024.naacl-long)

Challenge:	Large language models often "hallucinate" i.e., fabricate facts without providing users an apparent means to discern the veracity of their statements.
Approach:	They propose a framework with implementations of state-of-the-art UE methods for LLMs with unified program interfaces in Python.
Outcome:	The proposed framework implements state-of-the-art UE methods for LLMs with unified program interfaces in Python and an extendable benchmark for consistent evaluation by researchers.

Copied to clipboard

Debrup Das, Debopriyo Banerjee, Somak Aditya, Ashish Kulkarni

Reasoning

Challenge:	TALMs have been successfully employed in question-answering benchmarks, but their efficacy on complex mathematical reasoning benchmarks are open research questions.
Approach:	They propose a tool-augmented large language model for mathematical reasoning that enhances the skillset of large language models (LLMs) by 13.5%.
Outcome:	The proposed model achieves better accuracy and better knowledge retrieval performance than existing tools.

Valet: Rule-Based Information Extraction for Rapid Deployment (2022.lrec-1)

Copied to clipboard

Dayne Freitag, John Cadigan, Robert Sasseen, Paul Kalmar

Information Extraction & Text Mining

Challenge:	a number of machine learning models can be trained to perform sentence-level information extraction at accuracies ranging from strong to adequate.
Approach:	They propose a Python framework for rule-based information extraction that allows for complex matching.
Outcome:	The proposed framework can be used to perform rule-based information extraction on examples.

Detect-Localize-Repair: A Unified Framework for Learning to Debug with CodeT5 (2022.findings-emnlp)

Copied to clipboard

Nghi Bui, Yue Wang, Steven C.H. Hoi

Evaluating Tokenizer Adaptation Methods for Large Language Models on Low-Resource Programming Languages (2025.acl-srw)

Challenge:	Automated software debugging is crucial for improving productivity of software developers . many neural-based techniques focus only on one or the other, ignoring mutual benefits .
Approach:	They propose a framework to adapt a pretrained programming language model to automate debugging . they propose three objectives: bug detection, bug localization, program repair .
Outcome:	The proposed framework outperforms baselines from both NLP and software engineering domains on two new datasets.

Copied to clipboard

Georgy Andryushchenko, Vladimir V. Ivanov

StructVAE: Tree-structured Latent Variable Models for Semi-supervised Semantic Parsing (P18-1)

Challenge:	Large language models (LLMs) trained on high-resource programming languages perform sub-optimally for low-resourced programming languages (LRPLs).
Approach:	They evaluate the impact of tokenizer adaptation methods on improving code generation for LRPLs.
Outcome:	The proposed methods outperform the original models and fine-tuned models in LRPLs, but performance declines in non-target languages like Python after tokenizer adaptation.

Copied to clipboard

Pengcheng Yin, Chunting Zhou, Junxian He, Graham Neubig

AgentFactory: A Self-Evolving Framework Through Executable Subagent Accumulation and Reuse (2026.acl-demo)

Challenge:	Semantic parsing is the task of transducing natural language (NL) utterances into formal meaning representations (MRs), commonly represented as tree structures.
Approach:	They propose a variational auto-encoding model for semi-supervised semantic parsing which learns from limited amounts of parallel data and readily-available unlabeled NL utterances.
Outcome:	Experiments on ATIS domain and Python show that with extra unlabeled data, StructVAE outperforms strong supervised models.

Copied to clipboard

Zhang Zhang, Shuqi Lu, Hongjin Qian, Di He, Zheng Liu

TableCoder: Table Extraction from Text via Reliable Code Generation (2025.acl-industry)

Challenge:	Existing frameworks for building LLM-based agents treat agent behavior as static-knowledge gained during execution is not preserved for future use.
Approach:	They propose a new paradigm that preserves successful task solutions as executable subagent code rather than textual experience.
Outcome:	The proposed agent-based agent-driven paradigm preserves successful tasks as executable subagent code rather than textual experience.

Copied to clipboard

Haoyu Dong, Yue Hu, Huailiang Peng, Yanan Cao

Dataset of Student Solutions to Algorithm and Data Structure Programming Assignments (2022.lrec-1)

Challenge:	Structured table extraction from unstructured text is critical for automating data processing tasks across industries where accuracy and reliability are paramount.
Approach:	They propose a natural language-based method for extracting structured tables from text . they use Python classes or SQL statements to explicitly construct table structures .
Outcome:	The proposed method improves F1 scores and mitigates hallucinations . it integrates with standard SQL databases and Python workflows, ensuring seamless deployment .

Copied to clipboard

Fynn Petersen-Frey, Marcus Soll, Louis Kobras, Melf Johannsen, Peter Kling, Chris Biemann

ConCodeEval: Evaluating Large Language Models for Code Constraints in Domain-Specific Languages (2025.acl-industry)

Challenge:	a dataset containing source code solutions to algorithmic programming exercises solved by students at the University of Hamburg is available under the permissive CC BY-NC 4.0 license.
Approach:	They present a dataset containing source code solutions to algorithmic programming exercises solved by students at the University of Hamburg.
Outcome:	The proposed dataset contains solutions to 21 programming tasks written in Java and Python and over 1500 individual solutions.

Copied to clipboard

Mehant Kammakomati, Sameer Pimparkhede, Srikanth G. Tamilselvam, Prince Kumar, Pushpak Bhattacharyya

Text-to-SQL Error Correction with Language Models of Code (2023.acl-short)

Challenge:	Large Language Models (LLMs) have demonstrated potential in code generation and natural language understanding, but they struggle with code constraints.
Approach:	They propose to use Large Language Models to handle constraints represented in code . they use JSON, YAML, XML, Python, and natural language to test their effectiveness .
Outcome:	The proposed benchmark shows that LLMs can handle code constraints better than natural language . the results suggest that conscious choice of representations can lead to optimal use of LLM in enterprise use cases involving code constraints.

Copied to clipboard

Ziru Chen, Shijie Chen, Michael White, Raymond Mooney, Ali Payani, Jayanth Srinivasa, Yu Su, Huan Sun

MPL: Multiple Programming Languages with Large Language Models for Information Extraction (2025.findings-acl)

Challenge:	Existing semantic parsers are not accurate enough for use in text-to-SQL parsing tasks.
Approach:	They propose to build clause-level edit models to correct SQL queries instead of token-level ones.
Outcome:	The proposed model improves the exact set match accuracy of different parsers by 2.4-6.5 and obtains up to 4.3 point absolute improvement over two strong baselines.

Copied to clipboard

Bo Li, Gexiang Fang, Wei Ye, Zhenghua Xu, Jinglei Zhang, Hao Cheng, Shikun Zhang

Program-Aided Reasoners (Better) Know What They Know (2024.naacl-long)

Challenge:	Existing research focuses on Python for code-style simulation, overlooking the potential of other widely-used PLs during the supervised fine-tuning phase.
Approach:	They propose a framework that incorporates programming languages into IE tasks . they introduce function-prompt with virtual running to simulate code-style inputs .
Outcome:	The proposed framework exploits the potential of different programming languages during the supervised fine-tuning phase.

Copied to clipboard

Anubha Kabra, Sanketh Rangreji, Yash Mathur, Aman Madaan, Emmy Liu, Graham Neubig

CODEMENV: Benchmarking Large Language Models on Code Migration (2025.findings-acl)

Challenge:	Prior work shows that program-aided reasoning improves accuracy but also requires reasoners to "know what they know".
Approach:	They compare the calibration of program-aided language models (PAL) and text-based Chain-of-thought (COT) prompting techniques over 5 datasets and 2 model types .
Outcome:	The proposed methods improve accuracy and calibrate the models over 5 datasets and 2 model types.

Copied to clipboard

Keyuan Cheng, Xudong Shen, Yihao Yang, TengyueWang TengyueWang, Yang Cao, Muhammad Asif Ali, Hanbin Wang, Lijie Hu, Di Wang

TRACE: Evaluating Execution Efficiency of LLM-Based Code Translation (2026.acl-long)

Challenge:	Large language models (LLMs) have demonstrated remarkable proficiency in handling a wide range of tasks within the software engineering domain, but their ability to perform code migration—adapting code to different environments—remains underexplored.
Approach:	They propose a benchmark to evaluate large language models’ performance in handling code migration tasks.
Outcome:	The proposed benchmark comprises 922 data points across 19 Python and Java packages and offers three tasks to systematically evaluate code migration: identifying version-incompatible functions, determining function changes, and adapting code to target environments.

Copied to clipboard

Zhihao Gong, Zeyu Sun, Dong Huang, Qingyuan Liang, Jie M. Zhang, Dan Hao

AVATAR: A Parallel Corpus for Java-Python Program Translation (2023.findings-acl)

Challenge:	Large Language Models (LLMs) have improved the functional correctness of code translation, but execution efficiency remains overlooked.
Approach:	They propose a benchmark to explicitly assess execution efficiency in LLM-translated code.
Outcome:	The proposed benchmark identifies that execution efficiency is an essential dimension of code translation . the results highlight that correctness and efficiency are often misaligned .

Copied to clipboard

Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, Kai-Wei Chang

CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course (2022.naacl-main)

Challenge:	Program translation is a time-consuming and costly process that requires expertise in both the source and target languages.
Approach:	They present a collection of 9,515 programming problems and their solutions written in Java and Python.
Outcome:	The proposed model lacks in generating functionally accurate code.

Copied to clipboard

Changyoon Lee, Yeon Seonwoo, Alice Oh

Natural Language Interfaces

Challenge:	CS1QA is a dataset for code-based question answering in the programming education domain.
Approach:	They propose a dataset for code-based question answering in the programming education domain.
Outcome:	The proposed model can be used as a benchmark for source code comprehension and question answering in the educational setting.

Investigating the Transferability of Code Repair for Low-Resource Programming Languages (2025.findings-naacl)

Copied to clipboard

Kyle Wong, Alfonso Amayuelas, Liangming Pan, William Yang Wang

Nkululeko: A Tool For Rapid Speaker Characteristics Detection (2022.lrec-1)

Challenge:	Large language models (LLMs) have shown remarkable performance on code generation tasks.
Approach:	They investigate the benefits of distilling the ability to repair code for both high and low resource languages to determine if the techniques are also applicable in low resource settings.
Outcome:	The proposed techniques are effective in high- and low-resource languages, but weak in low-level languages.

Copied to clipboard

Felix Burkhardt, Johannes Wagner, Hagen Wierstorf, Florian Eyben, Björn Schuller

Challenge:	Nkululeko is a software tool that lets users perform semi-supervised machine learning experiments in the speaker characteristics domain.
Approach:	They propose a software tool called Nkululeko that lets users perform semi-supervised machine learning experiments in the speaker characteristics domain.
Outcome:	The proposed tool is based on audformat, a speech database metadata description . it supports best practise and fast setup of experiments without programming skills .

Marking Code Without Breaking It: Code Watermarking for Detecting LLM-Generated Code (2026.findings-eacl)

Copied to clipboard

Jungin Kim, Shinwoo Park, Yo-Sub Han

Novel Natural Language Summarization of Program Code via Leveraging Multiple Input Representations (2021.findings-emnlp)

Challenge:	Existing methods for identifying LLM-generated code are limited by syntax-critical tokens, which can introduce syntax errors.
Approach:	They propose a syntax-aware watermarking method that embeds watermarks only in non-syntactic tokens and preserves code integrity.
Outcome:	The proposed method outperforms baseline methods on Python, C++, and Java.

Copied to clipboard

Fuxiang Chen, Mijung Kim, Jaegul Choo

SolEval: Benchmarking Large Language Models for Repository-level Solidity Smart Contract Generation (2025.emnlp-main)

Challenge:	Existing work on code summarization shows that code descriptions are difficult to generate for developers unfamiliar with the code base.
Approach:	They propose a multi-task approach that trains two similar tasks to generate code descriptions for each line of code.
Outcome:	The proposed model improves over baselines and achieves the new state-of-the-art.

Copied to clipboard

Zhiyuan Peng, Xin Yin, Rui Qian, Peiqin Lin, YongKang Liu, Hao Zhang, Chenhao Ying, Yuan Luo

MojoBench: Language Modeling and Benchmarks for Mojo (2025.findings-naacl)

Challenge:	Existing methods focus on Python and Java, neglecting Solidity, the programming language for Ethereum smart contracts.
Approach:	They construct a repository-level benchmark for Solidity to evaluate the performance of LLMs on Ethereum.
Outcome:	The proposed benchmarks show that the best performing LLM achieves only 26.29% Pass@10, highlighting room for improvement in Solidity code generation.

Copied to clipboard

Md Nishat Raihan, Joanna C. S. Santos, Marcos Zampieri

Retrieval Augmented Code Generation and Summarization (2021.findings-emnlp)

Challenge:	Mojo is a programming language that has been praised for its speed and performance over Python.
Approach:	They propose a framework for Mojo code generation that evaluates code Large Language Models (LLMs) they propose 'mojo-Coder' which is the first LLM pretrained and fine-tuned for MoJO code generation .
Outcome:	MojoBench is the first framework for mojo code generation . it achieves a 30-35% performance improvement over leading models like GPT-4o and Claude-3.5-Sonnet .

Copied to clipboard

Md Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang

CORES: Code-Oriented Reasoning for Complex Text-to-SQL and Generalizable TableQA (2026.findings-acl)

Challenge:	Software developers often recall parts of source code or code summaries that they had written in the past while implementing software or documenting them.
Approach:	They propose a retrieval augmented framework that retrieves relevant code or summaries from a database and provides them as a supplement to code generation or summarization models.
Outcome:	The proposed framework can search for relevant code or summaries from retrieval databases and can work with unimodal (only code or natural language description) or bimodal instances (code-description pairs).

Copied to clipboard

Meng Zhang, Ruochun Jin, Yuanxi Peng, Wenjing Yang, Haotian Wang, Liting Sun, Kun Hu, Silin Yang, Zhang Ke-di

DebugBench: Evaluating Debugging Capability of Large Language Models (2024.findings-acl)

Challenge:	Text-to-SQL models struggle with complex analytical tasks such as generating simple SQL queries.
Approach:	They propose a text-to-sql model that leverages Python as a procedural reasoning pivot to enhance both complex SQL generation and tabular reasoning.
Outcome:	The proposed model outperforms baseline models on six text-to-SQL benchmarks by 6.44% on average while maintaining good capability on three tableQA benchmarks.

Copied to clipboard

Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Hui Haotian, Liu Weichuan, Zhiyuan Liu, Maosong Sun

Natural Language Embedded Programs for Hybrid Language Symbolic Reasoning (2024.findings-naacl)

Challenge:	Large language models (LLMs) have demonstrated exceptional coding capabilities, but their debugging capabilities remain relatively unexplored.
Approach:	They propose a debugging benchmark consisting of 4,253 LLMs with four major bug categories and 18 minor types in C++, Java, and Python.
Outcome:	The proposed benchmark covers four major bug categories and 18 minor types in C++, Java, and Python.

Copied to clipboard

Tianhua Zhang, Jiaxin Ge, Hongyin Luo, Yung-Sung Chuang, Mingye Gao, Yuan Gong, Yoon Kim, Xixin Wu, Helen Meng, James Glass

Quevedo: Annotation and Processing of Graphical Languages (2022.lrec-1)

Challenge:	Existing methods for surfacing symbolic reasoning capabilities are limited to narrow tasks . arithmetic computations are unnatural to perform in pure language space, and hence present difficulties for LLMs.
Approach:	They propose a natural language embedded program framework for solving symbolic reasoning tasks.
Outcome:	The proposed framework improves on strong baselines across math and symbolic reasoning, text classification, question answering, and instruction following tasks.

Copied to clipboard

Antonio F. G. Sevilla, Alberto Díaz Esteban, José María Lahoz-Bengoechea

Leveraging Code to Improve In-Context Learning for Semantic Parsing (2024.naacl-long)

Challenge:	graphical languages use images to convey meaning, but they require specialized computational processing . graphical systems use visual features and exploit the two dimensions of the page as a fundamental feature for codifying meaning .
Approach:	They present a software tool for automatic processing of graphical languages . they use a command line application and library to collect and manage image datasets .
Outcome:	a new software tool is developed for the processing of graphical languages . the tool provides features for the collection and management of image datasets .

Copied to clipboard

Ben Bogin, Shivanshu Gupta, Peter Clark, Ashish Sabharwal

NLG-Metricverse: An End-to-End Library for Evaluating Natural Language Generation (2022.coling-1)

Challenge:	In-context learning is an attractive approach for semantic parsing, but learning to parse to rare domain-specific languages from a few demonstrations is challenging.
Approach:	They propose to use Python instead of DSLs to augment prompts with a structured domain description.
Outcome:	The proposed approach improves accuracy and generalization across three datasets.

Copied to clipboard

Giacomo Frisoni, Antonella Carbonaro, Gianluca Moro, Andrea Zammarchi, Marco Avagnano

Diverse Retrieval-Augmented In-Context Learning for Dialogue State Tracking (2023.findings-acl)

Challenge:	Natural language generation models are a key component of deep learning, says aaron eliott . he says it is crucial to develop and apply better metrics for NLG evaluation .
Approach:	a new open-source library for NLG evaluation is created to facilitate researchers to judge the effectiveness of their models. the framework provides a living collection of NLG metrics in a unified and easy-to-use environment.
Outcome:	a new open-source library for NLG evaluation aims to improve performance of models . the framework provides tools to apply, analyze, compare, and visualize the metrics .

Copied to clipboard

Brendan King, Jeffrey Flanigan

Low-Resource NLP

Challenge:	Recent work has demonstrated that in-context learning for dialogue state tracking outperforms training methods in the few-shot setting.
Approach:	They propose a method for in-context learning for dialogue state tracking that takes into account probabilities of competing surface forms and produces a more accurate dialogue state prediction.
Outcome:	The proposed method outperforms trained methods in the few-shot setting and requires little data and zero parameter updates.

Verification-Aware Planning for Multi-Agent Systems (2026.eacl-long)

Copied to clipboard

Tianyang Xu, Dan Zhang, Kushan Mitra, Estevam Hruschka

Challenge:	Large language model (LLM) agents are increasingly deployed to tackle complex tasks . multi-agent collaboration introduces new challenges in planning, coordination, and verification .
Approach:	They propose a framework for multi-agent collaboration with verification-aware planning . the framework decomposes tasks, models subtask dependencies, and encodes planner-defined passing criteria as subtask verification functions (VFs)
Outcome:	The proposed framework outperforms baselines on diverse datasets while improving system robustness and interpretability.

CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow (2024.findings-acl)

Copied to clipboard

Nathanaël Beau, Benoit Crabbé

Low-resource Post Processing of Noisy OCR Output for Historical Corpus Digitisation (L18-1)

Challenge:	Comprising 3,402 crafted examples, our dataset is designed for both model finetuning and standalone evaluation.
Approach:	They propose a dataset that provides examples that include a clarified intent, code snippets associated, and an average of three related unit tests.
Outcome:	The proposed dataset includes 3,402 hand-written examples and 3,121 unrefined examples.

Copied to clipboard

Caitlin Richter, Matthew Wickes, Deniz Beser, Mitch Marcus

Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy (2021.emnlp-main)

Challenge:	7.6% of the words in the original OCR text contain an error; fully manual correction would take thousands of hours due to the size of the corpus.
Approach:	They propose a post-processing system to efficiently correct OCR errors in a 2.7 million word Faroese corpus.
Outcome:	The proposed method reduces the word error rate to 1.3% with around 65 hours of human annotator work.

Copied to clipboard

Colin Clement, Shuai Lu, Xiaoyu Liu, Michele Tufano, Dawn Drain, Nan Duan, Neel Sundaresan, Alexey Svyatkovskiy

Python is Not Always the Best Choice: Embracing Multilingual Program of Thoughts (2024.emnlp-main)

Challenge:	Statistical language modeling and translation with transformers have found many successful applications in program understanding and generation tasks.
Approach:	They propose an architecture-independent approach for leveraging syntactic hierarchies of source code . they use syntax trees to extract syntak hierarchical structures and integrate them into context window .
Outcome:	The proposed approach achieves state-of-the-art in code completion and summarization for Python in the CodeXGLUE benchmark.

Copied to clipboard

Xianzhen Luo, Qingfu Zhu, Zhiming Zhang, Libo Qin, Xuanyu Zhang, Qing Yang, Dongliang Xu, Wanxiang Che

RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository (2026.acl-long)

Challenge:	Program of Thoughts (PoT) is an approach characterized by its executable intermediate steps, which ensure the accuracy of the logical calculations in the reasoning process.
Approach:	They propose a task and model agnostic approach which harnesses strength and diversity from various languages to achieve better performance across all tasks.
Outcome:	The proposed approach outperforms Python Self-Consistency in almost all tasks and models and achieves comparable or superior performance on ChatGPT.

Copied to clipboard

Zhiyuan Peng, Xin Yin, Pu Zhao, Fangkai Yang, Lu Wang, Ran Jia, Xu Chen, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

Challenge:	Existing benchmarks focus on isolated function/class-level generation, neglecting complete microservice repository generation.
Approach:	They propose a multilingual benchmark for repository-level end-to-end web microservice generation that reflects real-world development workflows.
Outcome:	The benchmark compared 106 repositories across 18 domains and 11 frameworks and 1,258 API endpoints and 2,335 test cases.

Epitran: Precision G2P for Many Languages (L18-1)

Copied to clipboard

David R. Mortensen, Siddharth Dalmia, Patrick Littell

Challenge:	Epitran is a multilingual, multi-back-end system for grapheme-to-phoneme transduction . it supports 61 languages and is open source under an MIT license .
Approach:	Epitran is a multilingual back-end system for grapheme-to-phoneme transduction . it takes word tokens in the orthography of a language and outputs a phonemic representation . Epitran's efficacy has been demonstrated in multiple research projects .
Outcome:	Epitran is a multilingual, multi-backend system for grapheme-to-phoneme transduction . it supports 61 languages and is open source under MIT license .

ReACC: A Retrieval-Augmented Code Completion Framework (2022.acl-long)

Copied to clipboard

Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, Alexey Svyatkovskiy

The slurk Interaction Server Framework: Better Data for Better Dialog Models (2022.lrec-1)

Challenge:	Recent work has shown that statistical language modeling with transformers can greatly improve the performance in code completion tasks.
Approach:	They propose a retrieval-augmented code completion framework that combines a source code retriever and an auto-regressive language model for programming language.
Outcome:	The proposed framework achieves state-of-the-art on CodeXGLUE benchmark.

Copied to clipboard

Jana Götze, Maike Paetzel-Prüsmann, Wencke Liermann, Tim Diekmann, David Schlangen

Dialogue Systems & Conversational Agents

Challenge:	slurk is a lightweight dialog data collection and testing tool for crowdsourcing platforms.
Approach:	They present a lightweight dialog server that allows to set up dialog data collections and run experiments.
Outcome:	The slurk software allows to set up dialog data collections and run experiments with no limitations on the number of participants.

NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Queries (2024.findings-acl)

Copied to clipboard

Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng, Zehan Qi, Xiaotao Gu, Yuxiao Dong, Jie Tang

Green & Sustainable NLP

Challenge:	Large language models (LLMs) generate code for productive activities, but current benchmarks for code synthesis are oriented towards introductory tasks on algorithm and data science.
Approach:	They propose a code benchmark to mirror the complexity and variety of scenarios in real-world coding tasks.
Outcome:	The proposed benchmark improves on 39 large language models with close HumanEval scores and achieves an efficiency increase of more than 4 times.

Scaling Laws for Code: Every Programming Language Matters (2026.findings-acl)

Copied to clipboard

Jian Yang, Shuyue Guo, Linzheng Chai, Wei Zhang, Aishan Liu, Chuan Hao, Zhoujun Li, Xin Zhao, Xianglong Liu, Weifeng Lv, Bryan Dai

VulLibGen: Generating Names of Vulnerability-Affected Packages via a Large Language Model (2024.acl-long)

Challenge:	Existing studies focus on language-agnostic settings, neglecting the inherently multilingual nature of modern software development.
Approach:	They propose a proportion-dependent scaling law that prioritizes high-utility languages . they propose PLs to have varying effects during pre-training that affect model performance .
Outcome:	The proposed scaling law is based on 1000+ experiments across multiple languages and models.

Copied to clipboard

Tianyu Chen, Lin Li, ZhuLiuchuan ZhuLiuchuan, Zongyang Li, Xueqing Liu, Guangtai Liang, Qianxiang Wang, Tao Xie

DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale (2025.findings-acl)

Challenge:	Existing work on affected package identification is limited by large language models . a recent study shows that 84% third-party packages contain security vulnerabilities .
Approach:	They propose a method to use LLM to generate the affected package . they propose supervised fine-tuning, retrieval augmented generation and a local search algorithm .
Outcome:	The proposed method has an average precision of 0.806 for identifying vulnerable packages in four most popular ecosystems in GitHub Advisory.

Copied to clipboard

Linghao Zhang, Junhao Wang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Jiaheng Wen, Chengxing Xie, Maoquan Wang, Yufan Huang, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

Incorporating External Knowledge through Pre-training for Natural Language to Code Generation (2020.acl-main)

Challenge:	Existing studies highlight that dependency-related issues cause over 40% of observed runtime errors on the generated repository.
Approach:	They propose a large-scale benchmark and evaluation framework specifically designed to assess LLMs’ capability on dependency inference.
Outcome:	The proposed model achieves only a 48% execution pass rate on Python, indicating room for improvement.

Copied to clipboard

Frank F. Xu, Zhengbao Jiang, Pengcheng Yin, Bogdan Vasilescu, Graham Neubig

Bridge-Coder: Transferring Model Capabilities from High-Resource to Low-Resource Programming Language (2025.findings-acl)

Challenge:	Existing work on open-domain code generation focuses on limited domains or domain-specific languages with limited set of operators.
Approach:	They incorporate external knowledge into NL-to-code generation by combining StackOverflow and programming language API documentation with data augmentation and retrieval-based data re-sampling.
Outcome:	The proposed approach improves the current state-of-the-art by up to 2.2% absolute BLEU score on the code generation testbed CoNaLa.

Copied to clipboard

Jipeng Zhang, Jianshu Zhang, Yuanzhe Li, Renjie Pi, Rui Pan, Runtao Liu, Zheng Ziqiang, Tong Zhang

FARMI: A FrAmework for Recording Multi-Modal Interactions (L18-1)

Challenge:	Large Language Models (LLMs) excel at generating code for high-resource programming languages (HRPLs) however, they struggle significantly with low-resourced programming languages such as D, exacerbating the digital divide.
Approach:	They propose a method to generate LRPL data using LLM's general knowledge, HRPL proficiency, and in-context learning capabilities.
Outcome:	The proposed method improves on R, D, Racket, and Bash, while maintaining the same quality.

Copied to clipboard

Patrik Jonell, Mattias Bystedt, Per Fallgren, Dimosthenis Kontogiorgos, José Lopes, Zofia Malisz, Samuel Mascarenhas, Catharine Oertel, Eran Raveh, Todd Shore

MarkQA: A large scale KBQA dataset with numerical reasoning (2023.emnlp-main)

Challenge:	a new framework for recording multi-modal data is needed to capture multi-party, richly recorded corpora and perform real-time processing of such data.
Approach:	They propose an open-source processing architecture for corpora and real-time processing . they deploy the architecture in a multi-party deception game with six humans and one robot .
Outcome:	The proposed architecture is agnostic to hardware and programming languages, although it's mostly written in Python.

Copied to clipboard

Xiang Huang, Sitao Cheng, Yuheng Bao, Shanshan Huang, Yuzhong Qu

Reasoning

Challenge:	Existing KBQA datasets are insufficient for numerical reasoning . existing KBqa datasets lack multi-hop reasoning and numerical reasoning.
Approach:	They propose a task that necessitates the ability to perform multi-hop reasoning and numerical reasoning.
Outcome:	The proposed task necessitates the ability to perform multi-hop reasoning and numerical reasoning.

A Black-Box Attack on Code Models via Representation Nearest Neighbor Search (2023.findings-emnlp)

Copied to clipboard

Jie Zhang, Wei Ma, Qiang Hu, Shangqing Liu, Xiaofei Xie, Yves Le Traon, Yang Liu

The Lexometer: A Shiny Application for Exploratory Analysis and Visualization of Corpus Data (2022.lrec-1)

Challenge:	Existing methods for generating adversarial code examples face challenges such as limted availability of substitute variables and the creation of adversarials with noticeable perturbations.
Approach:	They propose a search seed based on historical attacks to find adversarial substitutes . they employ a pre-trained variable name encoder to map the search seed to a continuous vector space .
Outcome:	The proposed approach outperforms baseline methods in terms of ASR and QT.

Copied to clipboard

Oufan Hai, Matthew Sundberg, Katherine Trice, Rebecca Friedman, Scott Grimm

CodeFusion: A Pre-trained Diffusion Model for Code Generation (2023.emnlp-main)

Challenge:	Lexometer is a data science application that integrates data analysis and visualization functions into an easy-to-use graphical user interface.
Approach:	They propose a Shiny application that integrates data analysis and visualization functions into an easy-to-use graphical user interface.
Outcome:	The Lexometer integrates numerous data analysis and visualization functions into an easy-to-use graphical user interface.

Copied to clipboard

Mukul Singh, José Cambronero, Sumit Gulwani, Vu Le, Carina Negreanu, Gust Verbruggen

DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models (2024.emnlp-main)

Challenge:	Existing models for code generation from natural language do not allow reconsidering earlier tokens . prior work has explored grouped beam search or nucleus sampling to generate diverse text.
Approach:	They propose a diffusion code generation model that iteratively denoises a program conditioned on the encoded natural language.
Outcome:	The proposed model outperforms state-of-the-art models in accuracy and diversity compared to existing models.

Copied to clipboard

Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, Kang Liu

Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation (2025.acl-long)

Challenge:	DA-Code is a code generation benchmark designed to assess LLMs on agent-based data science tasks.
Approach:	They propose a code generation benchmark specifically designed for LLMs on agent-based data science tasks.
Outcome:	The benchmark performs better than existing frameworks, but lacks accuracy . it is based on real-world data, and includes examples that cover a wide range of tasks .

Copied to clipboard

Yue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta, Luca Weihs, Andrew Head, Mark Yatskar, Chris Callison-Burch, Ranjay Krishna, Aniruddha Kembhavi, Christopher Clark

ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness? (2024.emnlp-main)

Challenge:	Vision-language models struggle to understand text-rich images due to the scarcity of diverse text-only large language data.
Approach:	They propose a framework that leverages the coding capabilities of text-only large language models to create synthetic text-rich multimodal data.
Outcome:	The proposed framework can generate high-quality instruction-tuning data using Python, HTML, LaTeX and other languages.

Copied to clipboard

Siddhant Waghjale, Vishruth Veerendranath, Zhiruo Wang, Daniel Fried

CAMeL Tools: An Open Source Python Toolkit for Arabic Natural Language Processing (2020.lrec-1)

Challenge:	Current methods for optimizing program efficiency improve performance measured by execution time, but they often come at the cost of severely decreasing the functional correctness.
Approach:	They propose a reproducible benchmark for evaluating program efficiency via two paradigms: natural language (NL) based code generation and history-based code editing.
Outcome:	The proposed approach improves performance while maintaining correctness while adding execution information.

Copied to clipboard

Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, Nizar Habash

Challenge:	CAMeL Tools provides utilities for pre-processing, morphological modeling, Dialect Identification, Named Entity Recognition and sentiment analysis.
Approach:	They present CAMeL Tools, an open-source Python toolkit for Arabic natural language processing . CAMeleL Tools provides utilities for pre-processing, morphological modeling, Dialect Identification, Named Entity Recognition and sentiment analysis.
Outcome:	The proposed tools are based on CAMeL Tools, an open-source Python toolkit for Arabic natural language processing.

API-Assisted Code Generation for Question Answering on Varied Table Structures (2023.emnlp-main)

Copied to clipboard

Yihan Cao, Shuyi Chen, Ryan Liu, Zhiruo Wang, Daniel Fried

Data Augmentation for Code Translation with Comparable Corpora and Multiple References (2023.findings-emnlp)

Challenge:	Existing approaches to table question answering have been limited to specific table structures.
Approach:	They propose a unified TableQA framework that uses Python as a querying language and few-shot prompting to translate NL questions into Python programs.
Outcome:	The proposed framework provides a unified representation for structured tables as multi-index Pandas data frames and uses Python as a powerful querying language to translate NL questions into Python programs.

Copied to clipboard

Yiqing Xie, Atharva Naik, Daniel Fried, Carolyn Rose

Aligned Multi-View Scripts for Universal Chart-to-Code Generation (2026.acl-long)

Challenge:	Existing methods for translating code between programming languages are limited by parallel training data.
Approach:	They propose a data augmentation technique that builds comparable corpora and augments existing parallel data with multiple reference translations.
Outcome:	The proposed techniques improve CodeT5 translation between Java, Python, and C++ by an average of 7.5% Computational Accuracy (CA@1) .

Copied to clipboard

Zhihan Zhang, Lizi Liao

TReMu: Towards Neuro-Symbolic Temporal Reasoning for LLM-Agents with Memory in Multi-Session Dialogues (2025.findings-acl)

Challenge:	Existing methods for chart-to-code generation are largely Python-centric, limiting practical use and overlooking a critical source of supervision.
Approach:	They propose a chart-to-code generation tool that converts a graph image into an executable plotting script.
Outcome:	The proposed method outperforms existing systems and is competitive with proprietary systems.

Copied to clipboard

Yubin Ge, Salvatore Romeo, Jason Cai, Raphael Shu, Yassine Benajiba, Monica Sunkara, Yi Zhang

Natural Language Interfaces

Challenge:	Temporal reasoning in multi-session dialogues presents a significant challenge which has been under-studied in previous temporal reasoning benchmarks.
Approach:	They propose to augment LoCoMo dialogues and create multi-choice QAs to construct a temporal reasoning evaluation task and a framework to enhance temporal thinking capabilities of LLM-agents.
Outcome:	The proposed framework significantly improves temporal reasoning performance compared to baseline methods, raising from 29.83 on GPT-4o via standard prompting to 77.67 via the proposed framework.

PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs (2024.findings-emnlp)

Copied to clipboard

Ankit Yadav, Himanshu Beniwal, Mayank Singh

Which Programming Language and What Features at Pre-training Stage Affect Downstream Logical Inference Performance? (2024.emnlp-main)

Challenge:	HumanEval and MBPP are two popular benchmarks for Python code generation.
Approach:	They propose a large-scale human evaluation of two popular Python benchmarks . they propose 185 hand-crafted prompts in a balanced representation of 38 programming concepts across diverse difficulty levels.
Outcome:	The proposed benchmarks show a critical bias towards a limited set of programming concepts, neglecting most of the other concepts entirely.

Copied to clipboard

Fumiya Uchiyama, Takeshi Kojima, Andrew Gambardella, Qi Cao, Yusuke Iwasawa, Yutaka Matsuo

DocCGen: Document-based Controlled Code Generation (2024.emnlp-main)

Challenge:	Recent large language models (LLMs) have demonstrated remarkable generalization abilities in mathematics and reasoning tasks.
Approach:	They pre-trained decoder-based language models from scratch using ten programming languages and three natural language datasets.
Outcome:	The proposed models outperform natural languages on logical reasoning tasks.

Copied to clipboard

Sameer Pimparkhede, Mehant Kammakomati, Srikanth Tamilselvam, Prince Kumar, Ashok Kumar, Pushpak Bhattacharyya

MdEval: Massively Multilingual Code Debugging (2026.findings-acl)

Challenge:	Large language models (LLMs) produce state-of-the-art performance on natural language to code generation for resource-rich general-purpose languages like C++, Java, and Python.
Approach:	They propose a framework that breaks the NL-to-Code generation task into two steps . they use library documentation to detect the correct libraries and schema rules extracted from the documentation to constrain the decoding .
Outcome:	The proposed framework improves different sized language models across all six evaluation metrics, reducing syntactic and semantic errors in structured code.

Copied to clipboard

Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Jin Ke, Wei Zhang, Hualei Zhu, Shuyue Guo, Tao Sun, Jiaheng Liu, Yunlong Duan, Yu Hao, Liqun Yang, Guanglin Niu, Ge Zhang, Zhoujun Li

Generating Plausible Distractors for Multiple-Choice Questions via Student Choice Prediction (2025.acl-long)

Challenge:	Existing benchmarks primarily focus on Python and are limited in terms of language diversity.
Approach:	They propose a multilingual debugging benchmark that includes 3.9K test samples of 20 programming languages and introduces the debug instruction corpora MdEval-Instruct by injecting bugs into the correct multilingual queries and solutions.
Outcome:	The proposed benchmark includes 3.9K test samples of 20 programming languages and covers the automated program repair task, bug localization task, and bug identification task.

Copied to clipboard

Yooseop Lee, Suin Kim, Yohan Jo

Challenge:	Multiple-choice questions (MCQs) are critical for identifying misconceptions and gaps in knowledge and accurately assessing students' understanding.
Approach:	They propose to train a model to generate distractors that are more likely to be selected by students by a pairwise ranker and a distractor generator via Direct Preference Optimization.
Outcome:	The proposed model outperforms baseline models and performs comparable to humans in various metrics including pairwise rank accuracy and distractor plausibility.

CRUXEVAL-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution (2025.acl-long)

Copied to clipboard

Ruiyang Xu, Jialun Cao, Yaojie Lu, Ming Wen, Hongyu Lin, Xianpei Han, Ben He, Shing-Chi Cheung, Le Sun

RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian (2024.lrec-main)

Challenge:	Existing code benchmarks focus on code generation, while those for code reasoning are insufficient.
Approach:	They propose a multi-lingual code reasoning benchmark that contains 19 programming languages and at least 600 subjects for each language.
Outcome:	The proposed model trains on Python and achieves 34.4% Pass@1 in other languages, revealing the cross-language generalization of LLMs.

Copied to clipboard

Adrian Cosma, Ioan-Bogdan Iordache, Paolo Rosso

Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RL (2026.findings-acl)

Challenge:	Large language models are capable of solving tasks in natural language, but most tests assume they are written in English.
Approach:	They propose to use a dataset to measure the generalization power of large language models in a language other than English to evaluate their code intelligence.
Outcome:	The proposed dataset provides a benchmark for evaluating the code intelligence of language models trained on Romanian / multilingual text and a fine-tuning set for pretrained Romanian models.

Copied to clipboard

Zhaofeng Wu, Shiqi Wang, Boya Peng, Anuj Kumar Goyal, Melanie Kambadur, Sebastian Ruder, Yoon Kim, Chloe Bi

Knowledge Tracing in Programming Education Integrating Students’ Questions (2025.acl-long)

Challenge:	Modern language models demonstrate impressive coding capabilities in common programming languages (PLs) but their performance in lower-resource PLs is often limited by training data availability.
Approach:	They propose a zero-shot cross-programming-language transfer task for code RL . they propose RL training in a source PL fails to improve performance on other target PLs .
Outcome:	The proposed approach improves transferability in Llama-3.1 code generation on parallel-stack model . it also improves performance on other target PLs, compared to single-PL SFT .

Copied to clipboard

Doyoun Kim, Suin Kim, Yohan Jo

MultiFileTest: A Multi-File-Level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms (2026.findings-acl)

Challenge:	Existing knowledge tracing models that ignore student questions are suboptimal for programming education because of the complexity of coding tasks and the diverse methods students use to solve problems.
Approach:	They propose a model that leverages students' questions and automatically extracted skill information to enhance the accuracy of predicting students' performance on subsequent problems.
Outcome:	The proposed model improves student performance across Python programming courses of differing difficulty levels.

Copied to clipboard

Yibo Wang, Congying Xia, Wenting Zhao, Jiangshu Du, Chunyu Miao, Zhongfen Deng, Philip S. Yu, Chen Xing

Challenge:	Existing evaluation benchmarks for LLM unit test generation focus on function-level code rather than on more practical, challenging multi-file codebases.
Approach:	They propose a multi-file-level benchmark for unit test generation covering Python, Java, and JavaScript.
Outcome:	The proposed benchmarks show that most LLMs exhibit moderate performance on MultiFileTest, highlighting the benchmark’s inherent difficulty.

STARQA: A Question Answering Dataset for Complex Analytical Reasoning over Structured Databases (2025.emnlp-main)

Copied to clipboard

Mounica Maddela, Lingjue Xie, Daniel Preotiuc-Pietro, null Mausam

POSTCONDBENCH: Benchmarking Correctness and Completeness in Formal Postcondition Inference (2026.findings-acl)

Challenge:	Existing Text2SQL systems do not support SQL analysts in their primary work of performing complex analytics on specialized databases.
Approach:	They propose to decompose STARQA questions using SQL and Python to perform reasoning on specialized relational databases.
Outcome:	The proposed approach decomposes the task through a combination of SQL and Python, and achieves better performance on the more difficult questions.

Copied to clipboard

Gehao Zhang, Juan Zhai

Challenge:	Existing benchmarks emphasize correctness under limited evaluation settings . evaluation of formal specifications is time-consuming, errorprone and requires substantial expertise.
Approach:	They propose a multilingual benchmark for evaluating method-level postcondition generation from real-world software.
Outcome:	The proposed benchmarks show that evaluation remains a key bottleneck . 420 Python and Java tasks are paired with a high-quality postcondition set .

OmniCode: A Benchmark for Evaluating Software Development Agents (2026.findings-acl)

Copied to clipboard

Atharv Sonwane, Eng-Shen Tu, Wei-Chung Lu, Claas Beger, Carter Larsen, Debjit Dhar, Simon Alford, Rachel Chen, Ronit Pattanayak, Tuan Anh Dang, Guohao Chen, Gloria Geng, Kevin Ellis, Saikat Dutta

GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities (2026.acl-long)

Challenge:	popular coding benchmarks focus on narrowly scoped tasks such as competition programming and patch generation.
Approach:	They propose a software engineering benchmark that aims to provide a broader set of tasks beyond code or patch generation.
Outcome:	The proposed framework performs well on bug fixing for Python, test generation, code review fixing, and style fixing with popular agent frameworks such as SWE-Agent.

Copied to clipboard

Diganta Misra, Nizar Islah, Victor May, Brice Rauby, Zihan Wang, Justine Gehring, Antonio Orvieto, Muawiz Sajjad Chaudhary, Eilif B. Muller, Irina Rish, Samira Ebrahimi Kahou, Massimo Caccia