Papers by Asaf Yehudai
FastFit: Fast and Effective Few-Shot Text Classification with a Multitude of Classes (2024.naacl-demo)
Copied to clipboard
| Challenge: | Few-shot prompting of large language models (LLMs) via API calls presents a unique challenge when dealing with a multitude of classes that share similar semantic meanings. |
| Approach: | They present a Python package that integrates batch contrastive learning and token-level similarity score to provide fast few-shot classification. |
| Outcome: | The proposed method significantly improves multi-class classification speed and accuracy across English and Multilingual datasets. |
Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach (2021.emnlp-main)
Copied to clipboard
| Challenge: | cuneiform clay tablets were written in 2500 BCE - 100 CE and are a target of extensive transcription and transliteration efforts due to their deterioration. |
| Approach: | They propose to use a masked language modelling task to complete missing text given cuneiform clay tablets written on cuniform signswedges (2500 BCE - 100 CE) they develop models which automatically complete these missing signs based on contextual cues and greedy decoding schemes. |
| Outcome: | The proposed models perform well on missing token prediction (89% hit@5) despite data scarcity (1M tokens), and human evaluations show that they are able to transcribe texts in extinct languages. |
Selective Self-to-Supervised Fine-Tuning for Generalization in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can be fine-tuned on task-specific data to improve performance on target tasks but can be overfitted resulting in a loss of generalization. |
| Approach: | They propose a method that uses the correct model responses from a training set to fine-tune the model using the correct response and the gold response for the remaining samples. |
| Outcome: | The proposed approach reduces model specialization during the fine-tuning stage while improving generalization. |
Will it Merge? On The Causes of Model Mergeability (2026.findings-acl)
Copied to clipboard
| Challenge: | Model merging has emerged as a promising technique for combining fine-tuned models into a single expert model without retraining. |
| Approach: | They propose a model merging technique that preserves weak model knowledge . they define mergeability as a property of model updates that captures how well they retain trained knowledge when merged with other model updates. |
| Outcome: | The proposed method preserves weak knowledge in the base model. |
JuStRank: Benchmarking LLM Judges for System Ranking (2025.acl-long)
Copied to clipboard
| Challenge: | Recent work has focused on instance-based evaluation of LLM judges, where a judge is evaluated over a set of responses, or response pairs, while being agnostic to their source systems. |
| Approach: | They propose to validate the quality of the LLM judge itself by comparing system scores to a human-based ranking. |
| Outcome: | The proposed model fails to validate the quality of the judge itself, ignoring critical factors affecting system-level ranking, such as a judge’s positive or negative bias towards certain systems. |
Mediocrity is the key for LLM as a Judge Anchor Selection (2026.acl-long)
Copied to clipboard
| Challenge: | a poor selection of an anchor can dramatically reduce correlation with human rankings . traditional reference-based metrics are often ill-suited for open-ended generation . |
| Approach: | They evaluate 22 different anchors on a Arena-Hard-v2.0 dataset and quantify the effect size of anchor selection. |
| Outcome: | The proposed model is better or worse than all other models, but it is rarely indicative of the relative ranking of the models. |
A Survey on Evaluation of LLM-based Agents (2026.findings-acl)
Copied to clipboard
Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, Michal Shmueli-Scheuer
| Challenge: | This paper provides the first comprehensive survey of evaluation methods for LLM-based agents . LLMs are static, having fixed knowledge, and confined to text-to-text interaction. |
| Approach: | They analyze the evaluation of LLM-based agents across five perspectives . they identify current trends and key gaps in evaluation methods . |
| Outcome: | The proposed evaluation frameworks and tools are based on five perspectives . the results highlight current trends and identify gaps in future research . |
More Bang for your Context: Virtual Documents for Question Answering over Long Documents (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models struggle to utilize long contexts efficiently, resulting in a question answering problem. |
| Approach: | They propose a method to generate a short document that contains the most relevant parts for a given context window. |
| Outcome: | The proposed method improves the QA task by providing a short and focused VDoc to the LLM while keeping the context window full. |
Applying Intrinsic Debiasing on Downstream Tasks: Challenges and Considerations for Machine Translation (2024.emnlp-main)
Copied to clipboard
| Challenge: | In this study, we examine three considerations for intrinsic debiasing in neural machine translation models. |
| Approach: | They propose to measure the extrinsic bias of neural machine translation models by embedding them in a neural embeddable space and using different tokens to debias them. |
| Outcome: | The proposed methods over-rely on gender stereotypes and over-represent them in their models. |
Reinforcement Learning with Large Action Spaces for Neural Machine Translation (2022.coling-1)
Copied to clipboard
| Challenge: | Recent work has argued that the gains produced by Reinforcement learning are mostly due to promoting tokens that have already received a fairly high probability in pre-training. |
| Approach: | They hypothesize that the large action space is a main obstacle to RL’s effectiveness in MT by reducing the size of the vocabulary without changing the vocabulary. |
| Outcome: | The proposed method improves by 1.5 BLEU points on average. |
Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents (2026.acl-demo)
Copied to clipboard
| Challenge: | Agentic systems are becoming more capable of defining strategies, taking actions, and solving complex, multi-step tasks. |
| Approach: | They propose an automatic, dynamic, and easy-to-use evaluation framework that provides textual insights into agent behavior on three levels of granularity: system, trace, and node. |
| Outcome: | The proposed framework produces high-quality, data-driven, insightful feedback on system, trace, and node. |
Evaluating and Improving the Coreference Capabilities of Machine Translation Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Currently, end-to-end models learn coreference resolution implicitly by observing aligned sentences in bilingual corpora. |
| Approach: | They develop a method that derives coreference clusters from MT output and evaluates them without requiring annotations in the target language. |
| Outcome: | The proposed model outperforms existing models on three challenging benchmarks. |
A Grounded Preference Model for LLM Alignment (2024.findings-acl)
Copied to clipboard
Tahira Naseem, Guangxuan Xu, Sarathkrishna Swaminathan, Asaf Yehudai, Subhajit Chaudhury, Radu Florian, Ramón Astudillo, Asim Munawar
| Challenge: | Large Language Models (LLMs) suffer from factual inconsistency and hallucination despite recent advances . training a preference model requires substantial human annotation, which is expensive and labor-intensive. |
| Approach: | They propose to generate synthetic grounded preference data and train a Grounded Preference Model to assess the overall quality of grounded responses. |
| Outcome: | The proposed model can generate much better grounded responses as judged by GPT4 and achieves the TRUE faithfulness Benchmark. |