Papers with DS-1000
NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Queries (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) generate code for productive activities, but current benchmarks for code synthesis are oriented towards introductory tasks on algorithm and data science. |
| Approach: | They propose a code benchmark to mirror the complexity and variety of scenarios in real-world coding tasks. |
| Outcome: | The proposed benchmark improves on 39 large language models with close HumanEval scores and achieves an efficiency increase of more than 4 times. |