Papers by Nizar Islah
GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities (2026.acl-long)
Copied to clipboard
Diganta Misra, Nizar Islah, Victor May, Brice Rauby, Zihan Wang, Justine Gehring, Antonio Orvieto, Muawiz Sajjad Chaudhary, Eilif B. Muller, Irina Rish, Samira Ebrahimi Kahou, Massimo Caccia
| Challenge: | Existing code evolution benchmarks lack execution-based evaluation for generating code compliant with specific library versions. |
| Approach: | They propose a new Python code completion problem that evaluates the ability of large language models to perform version-conditioned code generation. |
| Outcome: | The proposed benchmarks show that state-of-the-art systems can perform version-conditioned code generation with high success rates. |