Papers by Natalie Shapira
How Well Do Large Language Models Perform on Faux Pas Tests? (2023.findings-acl)
Copied to clipboard
| Challenge: | Using large language models, we investigate the ability of large language model to generate correct responses to questions involving descriptions of faux pas situations. |
| Approach: | They investigate the ability of large language models to generate correct responses to questions involving descriptions of faux pas situations. |
| Outcome: | The proposed model fails to generate correct responses to presupposition questions and many of the correct responses are attributed to over-interpretation by the human reader. |
Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models (2024.eacl-long)
Copied to clipboard
Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, Vered Shwartz
| Challenge: | Recent work suggests that Large Language Models (LLMs) exhibit Neural Theory-of-Mind (N-ToM) however, prior work reached conflicting conclusions regarding those abilities. |
| Approach: | They examine the extent of Large Language Models’ N-ToM abilities through an extensive evaluation of 6 tasks and find that LLMs struggle with adversarial examples . |
| Outcome: | The proposed metrics show that LLMs exhibit certain N-ToM abilities, but this behavior is far from robust. |
Factual Retrieval in LLMs Is a Redundant, Distributed and Non-Contiguous Process (2026.acl-long)
Copied to clipboard
| Challenge: | Existing work posits that factual knowledge is stored at the last entity token position, but the precise mechanics of how facts are retrieved from model parameters remain unclear. |
| Approach: | They propose an iterative patching protocol to identify a minimal subset of layers necessary for attribute retrieval. |
| Outcome: | The proposed method shows that models possess multiple paths for the same entity and fact, highlighting a high degree of redundancy in attribute computation. |