Papers by Ameet Talwalkar
When Benchmarks Talk: Re-Evaluating Code LLMs with Interactive Feedback (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing static benchmarks that measure task performance often rely on a simple input-output configuration. |
| Approach: | They propose an evaluation pipeline that evaluates code models with different feedback types in an interactive setting. |
| Outcome: | The proposed evaluation pipeline compares model-user collaboration with static benchmarks by obfuscating inputs to a simulated user. |
Scaling Collaborative Effort with Agents (2026.findings-acl)
Copied to clipboard
Shannon Zejiang Shen, Valerie Chen, Ken Gu, Alexis Ross, Zixian Ma, Jillian Ross, Alex Gu, Chenglei Si, Wayne Chi, Andi Peng, Jocelyn J Shen, Ameet Talwalkar, Tongshuang Wu, David Sontag
| Challenge: | Current evaluations of agents focus on producing high-quality, final outputs in one shot, failing to account for the inherently iterative nature of many real-world problems. |
| Approach: | They propose a framework that captures how an agent’s utility grows with increasing user involvement. |
| Outcome: | The proposed framework captures how an agent’s utility grows with increasing user involvement, revealing a missing ingredient in agent design: the ability to sustain engagement and scaffold user understanding. |