Papers by Vitaly Shalumov
Measuring the Robustness of NLP Models to Domain Shifts (2024.findings-emnlp)
Copied to clipboard
Nitay Calderon, Naveh Porat, Eyal Ben-David, Alexander Chapanin, Zorik Gekhman, Nadav Oved, Vitaly Shalumov, Roi Reichart
| Challenge: | Existing research on domain robustness (DR) relies on the Source Drop (SD) but lacks a complementary metric, a new study finds . |
| Approach: | They propose to use the Target Drop (TD) to measure domain DR . they use a DR benchmark consisting of 7 diverse tasks to measure both metrics . |
| Outcome: | The proposed model types excel in-domain, but few-shot LLMs often surpass them cross-domain showing better robustness. |