Ranking LLM-Generated Loop Invariants for Program Verification (2023.findings-emnlp)
Copied to clipboard
Saikat Chakraborty, Shuvendu Lahiri, Sarah Fakhoury, Akash Lal, Madanlal Musuvathi, Aseem Rastogi, Aditya Senthilnathan, Rahul Sharma, Nikhil Swamy
| Challenge: | Large Language Models (LLMs) are capable of synthesizing inductive loop invariants for a class of programs in a 0-shot setting, yet require several samples to generate the correct invariant. |
| Approach: | They propose a re-ranking approach to generate inductive loop invariants using Large Language Models . they propose reranking rankers that can distinguish between correct and incorrect attempts . |
| Outcome: | The proposed method reduces the number of calls to a verifier by comparing the generated results with the original model. |
Similar Papers
AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Assertions have been the de facto collateral for hardware for over a decade. |
| Approach: | They propose a benchmark to evaluate LLMs’ effectiveness for assertion generation quantitatively. |
| Outcome: | The proposed benchmark compares state-of-the-art LLMs with existing benchmarks and shows that they generate higher fractions of functionally correct assertions. |
HALoGEN: Fantastic LLM Hallucinations and Where to Find Them (2025.acl-long)
Copied to clipboard
| Challenge: | generative large language models produce hallucinations that are not aligned with world knowledge or input context. |
| Approach: | They propose a hallucination benchmark framework that measures hallucinism in large language models . they evaluate 150,000 generations from 14 language models and find they are riddled with hallucinos . |
| Outcome: | The proposed framework evaluates 150,000 generations from 14 language models. |
CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used to judge code, but their reliability remains poorly understood. |
| Approach: | They propose a benchmark to evaluate Large Language Models as code judges . they find that small reasoning models outperform larger non-reasoning models . |
| Outcome: | The proposed benchmark evaluates LLM-as-a-Judge models across three coding tasks. |
RankPrompt: Step-by-Step Comparisons Make Language Models Better Reasoners (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing solutions to reasoning tasks require extensive human annotations or fail in scenarios with inconsistent responses. |
| Approach: | They propose a new method that enables LLMs to self-rank their responses without additional resources. |
| Outcome: | The proposed method improves reasoning performance of ChatGPT and GPT-4 with 13% improvement over existing methods. |
InductionBench: LLMs Fail in the Simplest Complexity Class (2025.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks focus on deductive reasoning, largely overlooking inductive reasoning. |
| Approach: | They propose a benchmark to evaluate the inductive reasoning ability of large language models. |
| Outcome: | The proposed benchmark demonstrates that even the most advanced modelw struggle to master the simplest complexity classes within the subregular hierarchy of functions. |
LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient (2026.acl-long)
Copied to clipboard
Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Using generic and efficient benchmark generators, human annotators are limited by inefficiency . current benchmark generator methods rely on seed signals, leading to long cycles and high costs . |
| Approach: | They propose a framework to evaluate LLMs as generic benchmark generators and integrate them as BenchMaker. |
| Outcome: | The proposed framework achieves comparable performance to human-annotated benchmarks on most metrics. |
How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a systematic and comprehensive empirical evaluation of state-of-the-art reranking methods is presented. |
| Approach: | They evaluate 22 reranking methods including 40 variants across established benchmarks . primary goal is to determine whether performance disparity exists between LLM-based reranters and lightweight counterparts based on novel queries . |
| Outcome: | The proposed methods perform better on familiar queries than lightweight models, the authors show . |
Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly used as rerankers, but their ranking behavior can be steered by small, natural-sounding prompts. |
| Approach: | They propose a token optimization method that crafts concise textual perturbations to consistently promote a target item in LLM-generated rankings. |
| Outcome: | The proposed method outperforms state-of-the-art base-lines and is hard to detect. |
GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing LLM leaderboards often reference scores reported in other papers without consistent settings and prompts, which may encourage cherry-picking favored settings and for better results. |
| Approach: | They propose an open-source and reproducible LLM evaluation suite built on top of OpenAI Evals that systematically evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
| Outcome: | The evaluation suite is built on top of OpenAI Evals and evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
Improving LLM Generations via Fine-Grained Self-Endorsement (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have demonstrated remarkable capabilities but can still fail frequently on knowledge-intensive tasks. |
| Approach: | They propose a self-endorsement framework that leverages fine-grained fact-level comparisons across multiple sampled responses. |
| Outcome: | The proposed framework can improve factuality of generations with simple prompts across scales of LLMs. |