Papers by Atrey Desai
Test-Time Reasoners Are Strategic Multiple-Choice Test-Takers (2026.acl-short)
Copied to clipboard
| Challenge: | Large language models (LLMs) give reasoning before answering, excelling in multiple-choice question answering (MCQA) . but, some studies find that LLMs sans reasoning fail in MCQA without using the question, i.e., choices-only. |
| Approach: | They propose to use reasoning LLMs to separate problematic data from less problematic strategies by examining reasoning traces. |
| Outcome: | The proposed models perform well in multiple-choice question answering without the question, but they fail to use the question. |
Filling in the Mechanisms: How do LMs Learn Filler-Gap Dependencies under Developmental Constraints? (2026.findings-acl)
Copied to clipboard
| Challenge: | Language models lack language-specific biases, yet still posit some important syntactic generalizations. |
| Approach: | They applied Distributed Alignment Search to checkpoints of a language model from the BabyLM challenge to evaluate whether representations of filler-gap dependencies transfer between wh-questions and topicalization. |
| Outcome: | The results suggest shared, yet item-sensitive mechanisms may develop with limited training data. |
BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks (2026.acl-long)
Copied to clipboard
Nishant Balepur, Bhavya Rajasekaran, Hyunjin Jane Oh, Michael Xie, Atrey Desai, Vipul Gupta, Steven James Moore, Eunsol Choi, Rachel Rudinger, Jordan Lee Boyd-Graber
| Challenge: | Multiple-choice question answering (MCQ) is standard in NLP, but benchmarks lack rigorous quality control. |
| Approach: | They propose an education-inspired toolkit that uses LLM judges to flag flaws in MCQs . they validate the tool with annotations and run it to audit 12 benchmarks based on 19-rule education rubric . |
| Outcome: | The proposed toolkit flags three common MCQ flaws based on a 19-rule education rubric . contaminated MCqs tend to inflate accuracy, while writing errors lower it and change rankings . |