Papers by Itay Itzhak
Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens (2022.naacl-main)
Copied to clipboard
| Challenge: | Standard pre-trained language models do not see the characters that compose each token's string representation. |
| Approach: | They probe the embedding layer of pretrained language models and show that models learn the internal character composition of whole word and subword tokens without seeing the characters coupled with the tokens. |
| Outcome: | The embedding layers of RoBERTa and GPT2 hold enough information to accurately spell up to a third of the vocabulary and reach high character ngram overlap across all token types. |
DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation (2025.findings-acl)
Copied to clipboard
Eliya Habba, Ofir Arviv, Itay Itzhak, Yotam Perlitz, Elron Bandel, Leshem Choshen, Michal Shmueli-Scheuer, Gabriel Stanovsky
| Challenge: | Recent work found that LLMs are sensitive to arbitrary prompt dimensions . this challenges traditional single-prompt evaluation practices . |
| Approach: | They present a large-scale dataset containing prompt perturbations of various evaluation benchmarks . they examine LLM sensitivity from an holistic perspective and assess the joint effects of perturbations along various dimensions . |
| Outcome: | The proposed dataset aims to democratize evaluation research and enable robust protocols . it includes more than 250M prompt perturbations and model outputs . |
Trust Me, I’m Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on large language model (LLM) hallucinations associated with model uncertainty or inaccurate knowledge. |
| Approach: | They define and investigate a type of hallucination where a model can answer a question correctly but a perturbation causes it to produce a hallucinous response with high certainty. |
| Outcome: | The proposed mitigations outperform existing methods on CHOKE hallucinations . the findings highlight the need to understand their origins and improve mitigation strategies . |