Papers by Jackson Pond
Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks (2024.findings-emnlp)
Copied to clipboard
Jack Gallifant, Shan Chen, Pedro Moreira, Nikolaj Munch, Mingye Gao, Jackson Pond, Leo Anthony Celi, Hugo Aerts, Thomas Hartvigsen, Danielle Bitterman
| Challenge: | Medical knowledge is context-dependent and requires consistent reasoning across various natural language expressions of semantically equivalent phrases. |
| Approach: | They create a robustness dataset to evaluate performance differences on medical benchmarks . they swap brand and generic drug names using physician expert annotations based on medical terminology . |
| Outcome: | The proposed model shows a consistent performance drop of 1-10% on medical benchmarks. |