Papers by Ivan Lopatin
SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks (2025.emnlp-demos)
Copied to clipboard
Adamenko Pavel, Ivanov Mikhail, Aidar Valeev, Rodion Levichev, Pavel Zadorozhny, Ivan Lopatin, Dmitrii Babaev, Alena Fenogenova, Valentin Malykh
| Challenge: | SWE-bench is a static benchmark that collects only once and never updates. |
| Approach: | They propose a dynamic, continuously updated benchmark to address data contamination issues by collecting real-world GitHub issues and rigorous quality validation. |
| Outcome: | The proposed benchmarks are based on a dataset of 2,294 GitHub issues and their corresponding pull requests (PRs) the static nature of the benchmarks makes it hard to distinguish meaningful progress. |