Papers by Michal Shmueli-Scheuer
Efficient Benchmarking (of Language Models) (2024.naacl-long)
Copied to clipboard
Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, Leshem Choshen
| Challenge: | Efficient Benchmarking is a method for reducing computation costs of LM evaluation without compromising reliability. |
| Approach: | They propose to reduce the computation costs of LM evaluation without compromising reliability by using a new measure - Decision Impact on Reliability. |
| Outcome: | The proposed benchmarks reduce computation costs by x100 or more, while maintaining reliability. |
Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI (2024.naacl-demo)
Copied to clipboard
Elron Bandel, Yotam Perlitz, Elad Venezian, Roni Friedman, Ofir Arviv, Matan Orbach, Shachar Don-Yehiya, Dafna Sheinwald, Ariel Gera, Leshem Choshen, Michal Shmueli-Scheuer, Yoav Katz
| Challenge: | Textual data processing pipelines are tailored to specific datasets, task and model combinations. |
| Approach: | They propose a library for customizable textual data preparation and evaluation tailored to generative language models. |
| Outcome: | Unitxt is a library for customizable textual data preparation and evaluation tailored to generative language models. |
Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data (2026.findings-acl)
Copied to clipboard
Ofir Arviv, Kristjan Greenewald, Yotam Perlitz, Hadar Mulian, Michal Shmueli-Scheuer, Leshem Choshen
| Challenge: | Current evaluation practices, typically employing fixed-size benchmarks, are inherently wasteful, continuing to the predetermined sample size even when the CI reaches 2.5, saving 80% of the evaluation cost. |
| Approach: | They propose an adaptive evaluation framework that combines sequential testing with stopping criteria tailored to common evaluation needs such as diminishing returns detection and minimum detectable effect size. |
| Outcome: | The proposed framework reduces computational cost and reliability while maintaining statistical significance. |
TalkSumm: A Dataset and Scalable Annotation Method for Scientific Paper Summarization Based on Conference Talks (P19-1)
Copied to clipboard
| Challenge: | Currently, no large-scale training data is available for the task of scientific paper summarization. |
| Approach: | They propose a method that automatically generates scientific paper summaries by utilizing videos of scientific conferences. |
| Outcome: | The proposed model performs similar to models trained on a dataset of summaries created manually. |
DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation (2025.findings-acl)
Copied to clipboard
Eliya Habba, Ofir Arviv, Itay Itzhak, Yotam Perlitz, Elron Bandel, Leshem Choshen, Michal Shmueli-Scheuer, Gabriel Stanovsky
| Challenge: | Recent work found that LLMs are sensitive to arbitrary prompt dimensions . this challenges traditional single-prompt evaluation practices . |
| Approach: | They present a large-scale dataset containing prompt perturbations of various evaluation benchmarks . they examine LLM sensitivity from an holistic perspective and assess the joint effects of perturbations along various dimensions . |
| Outcome: | The proposed dataset aims to democratize evaluation research and enable robust protocols . it includes more than 250M prompt perturbations and model outputs . |
Active Learning for Natural Language Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to NLG are limited by the lack of annotated data. |
| Approach: | They propose to use active learning to reduce the cost of manual annotation to improve annotation efficiency by selecting the most informative examples to label. |
| Outcome: | The proposed approach surpasses baseline of random example selection in some cases but not in others. |
A Survey on Evaluation of LLM-based Agents (2026.findings-acl)
Copied to clipboard
Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, Michal Shmueli-Scheuer
| Challenge: | This paper provides the first comprehensive survey of evaluation methods for LLM-based agents . LLMs are static, having fixed knowledge, and confined to text-to-text interaction. |
| Approach: | They analyze the evaluation of LLM-based agents across five perspectives . they identify current trends and key gaps in evaluation methods . |
| Outcome: | The proposed evaluation frameworks and tools are based on five perspectives . the results highlight current trends and identify gaps in future research . |
Navigating the Modern Evaluation Landscape: Considerations in Benchmarks and Frameworks for Large Language Models (LLMs) (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | General-purpose Language Models have changed the world of Natural Language Processing, if not the world itself. |
| Approach: | This tutorial will lay the foundations and explain the basics of evaluation and compare traditional methods to newly developed methods. |
| Outcome: | The tutorial assumes little familiarity with metrics, datasets, prompts and benchmarks . it will compare traditional methods to newly developed methods . |
Detecting Egregious Conversations between Customers and Virtual Agents (N18-1)
Copied to clipboard
Tommy Sandbank, Michal Shmueli-Scheuer, Jonathan Herzig, David Konopnicki, John Richards, David Piorkowski
| Challenge: | 80% of businesses plan to use chatbots by 2020, according to recent studies . but some bad conversations can be difficult to detect and could lead to litigation . |
| Approach: | They propose a method to detect bad conversations using behavioral cues from the user and patterns in agent responses. |
| Outcome: | The proposed method improves the detection F1 score by 20% over textual features. |
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents (2026.acl-demo)
Copied to clipboard
| Challenge: | Agentic systems are becoming more capable of defining strategies, taking actions, and solving complex, multi-step tasks. |
| Approach: | They propose an automatic, dynamic, and easy-to-use evaluation framework that provides textual insights into agent behavior on three levels of granularity: system, trace, and node. |
| Outcome: | The proposed framework produces high-quality, data-driven, insightful feedback on system, trace, and node. |
A Summarization System for Scientific Documents (D19-3)
Copied to clipboard
Shai Erera, Michal Shmueli-Scheuer, Guy Feigenblat, Ora Peled Nakash, Odellia Boni, Haggai Roitman, Doron Cohen, Bar Weiner, Yosi Mass, Or Rivlin, Guy Lev, Achiya Jerbi, Jonathan Herzig, Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, Francesca Bonin, David Konopnicki
| Challenge: | a qualitative user study identified the most valuable scenarios for scientific content consumption. |
| Approach: | They propose a system that retrieves and summarizes scientific documents for a given information need. |
| Outcome: | The proposed system ingested 270,000 scientific papers and validated with human experts. |
Quality Controlled Paraphrase Generation (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that high quality paraphrases are difficult to generate because of their low flexibility and scalability. |
| Approach: | They propose a quality-guided controlled paraphrase generation model that allows directly controlling the quality dimensions of the generated paraphrase. |
| Outcome: | The proposed method generates paraphrases which maintain original meaning while achieving higher diversity than the uncontrolled baseline. |
Think Again! The Effect of Test-Time Compute on Preferences, Opinions, and Beliefs of Large Language Models (2025.acl-industry)
Copied to clipboard
| Challenge: | Large Language Models exhibit subjective preferences, opinions, and beliefs, which may shape their behavior, influence advice and recommendations, and potentially reinforce certain viewpoints. |
| Approach: | They developed a benchmark to assess LLMs’ subjective inclinations across societal, cultural, ethical, and personal domains. |
| Outcome: | The proposed benchmark assesses LLMs’ subjective inclinations across societal, cultural, ethical, and personal domains. |