Papers by Sagi Shaier
Comparing Template-based and Template-free Language Model Probing (2024.eacl-long)
Copied to clipboard
| Challenge: | Template-based and template-based approaches rank models differently except for the top domain-specific models. |
| Approach: | They evaluate 16 different cloze-task language model probing approaches on 10 probing English datasets to answer questions about model rankings and absolute scores. |
| Outcome: | The results show that the template-based and template-free approaches rank models differently except for the top domain-specific models. |
It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehension (2024.findings-acl)
Copied to clipboard
| Challenge: | Experimenting with 9 large language models across 3 datasets, emphasizing the context yields superior results compared to question emphasis. |
| Approach: | They ask: How does the order of inputs affect model performance? |
| Outcome: | Experiments with 9 large language models show that emphasizing the question and context improves model performance. |
MALAMUTE: A Multilingual, Highly-granular, Template-free, Education-based Probing Dataset (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing cloze-style benchmarks for language models lack specific, granular areas of knowledge and often rely on templates that can bias models. |
| Approach: | They propose a multilingual, template-free, and highly granular probing dataset comprising expert-written, peer-reviewed probes from 71 university-level textbooks across three languages. |
| Outcome: | The proposed dataset covers eight domains, each with up to 14 subdomains, further broken down into concepts and concept-based prompts. |
Desiderata For The Context Use Of Question Answering Systems (2024.eacl-long)
Copied to clipboard
| Challenge: | Prior work has uncovered a set of common problems in state-of-the-art context-based question answering systems, such as a lack of attention to the context when it conflicts with a model’s parametric knowledge and a loss of consistency with their answers. |
| Approach: | They propose to examine the desiderata for context-based question answering systems and then compare them to a set of prior work. |
| Outcome: | The proposed models are based on 15 datasets and evaluated on 5 datasets. |
Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citations (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on citation generation has focused on unambiguous settings with single answers, failing to address the complexity of real-world scenarios. |
| Approach: | They propose a task of QA with source citation in ambiguous settings where multiple valid answers exist, where multiple sources exist. |
| Outcome: | The proposed framework generates multiple answers and cites their sources, allowing users to verify the factuality of each answer and make informed decisions. |