Papers by Nilesh Kumar
Language Model Probabilities are Not Calibrated in Numeric Contexts (2025.acl-long)
Copied to clipboard
Charles Lovering, Michael Krumdick, Viet Dac Lai, Varshini Reddy, Seth Ebner, Nilesh Kumar, Rik Koncel-Kedziorski, Chris Tanner
| Challenge: | Using language model outputs, we find that even in simple settings, the best LMs (1) are poorly calibrated and (2) have systematic biases. |
| Approach: | They argue that language model outputs should capture natural distributions over multiple options within their textual contexts. |
| Outcome: | The proposed model outputs are calibrated to the numeric content of their contexts. |