Papers by Leshem Choshen

DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering (2023.acl-long)

Challenge:	Efficient Benchmarking is a method for reducing computation costs of LM evaluation without compromising reliability.
Approach:	They propose to reduce the computation costs of LM evaluation without compromising reliability by using a new measure - Decision Impact on Reliability.
Outcome:	The proposed benchmarks reduce computation costs by x100 or more, while maintaining reliability.

Copied to clipboard

Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, Omri Abend

Question Answering

Challenge:	Question answering models have access to two sources of knowledge during inference time: parametric knowledge and contextual knowledge.
Approach:	They propose a new paradigm in which QA models are trained to disentangle the two sources of knowledge.
Outcome:	The proposed model generates two answers for a given question based on parametric and contextual knowledge.

Cluster & Tune: Boost Cold Start Performance in Text Classification (2022.acl-long)

Copied to clipboard

Eyal Shnarch, Ariel Gera, Alon Halfon, Lena Dankin, Leshem Choshen, Ranit Aharonov, Noam Slonim

Information Extraction & Text Mining

Challenge:	Existing methods to fine-tune pre-trained models for text classification are poor in practice.
Approach:	They propose to add an intermediate unsupervised classification task between pre-training and fine-tuning phases to boost performance of pre-trained models.
Outcome:	The proposed method improves performance on topical classification tasks when labeled data is scarce.

Inherent Biases in Reference-based Evaluation for Grammatical Error Correction (P18-1)

Copied to clipboard

Leshem Choshen, Omri Abend

Label Sleuth: From Unlabeled Text to a Classifier in a Few Hours (2022.emnlp-demos)

Challenge:	Existing evaluation systems obtain comparable or superior performance compared to humans by making few but targeted changes to the input.
Approach:	They propose to re-scale M 2 by the inter-annotator agreement and increase the number of references in any feasible range to overcome low coverage bias in GEC evaluation.
Outcome:	The proposed measure overcomes low coverage bias in GEC evaluation by re-scaling or increasing the number of references in any feasible range.

Copied to clipboard

Eyal Shnarch, Alon Halfon, Ariel Gera, Marina Danilevsky, Yannis Katsis, Leshem Choshen, Martin Santillan Cooper, Dina Epelboim, Zheng Zhang, Dakuo Wang

Text Classification

Challenge:	Label Sleuth is an open source system for labeling and creating text classifiers which does not require coding skills nor machine learning knowledge.
Approach:	Label Sleuth is an open source system for labeling and creating text classifiers which does not require coding skills nor machine learning knowledge.
Outcome:	Label Sleuth is an open source system for labeling and creating text classifiers.

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data (2026.acl-long)

Copied to clipboard

Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett, Rafael Mosquera, Sara Hincapié Monsalve, Thom Vaughan, Damian Stewart, Malte Ostendorff, Idris Abdulmumin, Vukosi Marivate, Shamsuddeen Hassan Muhammad, Atnafu Lambebo Tonja, Hend Al-Khalifa, Nadia Ghezaiel Hammouda, Verrah Akinyi Otiende, Tack Hwa Wong, Jakhongir Saydaliev, Melika Nobakhtian, Muhammad Ravi Shulthan Habibi, Chalamalasetti Kranti, Carol Muchemi, Khang Nguyen, Faisal Muhammad Adam, Luis Frentzen Salim, Reem Alqifari, Cynthia Jayne Amol, Joseph Marvin Imperial, Ilker Kesen, Ahmad Mustafid, Pavel Stepachev, Leshem Choshen, David Anugraha, Hamada Nayel, Seid Muhie Yimam, Vallerie Alexandra Putra, My Chiffon Nguyen, Azmine Toushik Wasi, Gouthami Vadithya, Rob Van Der Goot, Lanwenn ar C’horr, Karan Dua, Andrew Yates, Mithil Bangera, Yeshil Bangera, Hitesh Laxmichand Patel, Shu Okabe, Fenal Ashokbhai Ilasariya, Dmitry Gaynullin, Genta Indra Winata, Yiyuan Li, Juan Pablo Martínez, Amit Agarwal, Ikhlasul Akmal Hanif, Raia Abu Ahmad, Esther Adenuga, Filbert Aurelian Tjiaranata, Weerayut Buaphet, Michael Anugraha, Sowmya Vajjala, Benjamin L Rice, Azril Hafizi Amirudin, Jesujoba Oluwadara Alabi, Srikant Panda, Yassine Toughrai, Bruhan Kyomuhendo, Daniel Ruffinelli, null Akshata, Manuel Goulão, Ej Zhou, Ingrid Gabriela Franco Ramirez, Cristina Aggazzotti, Konstantin Dobler, Jun Kevin, Quentin Pagès, Nicholas Andrews, Nuhu Ibrahim, Mattes Ruckdeschel, Amr Keleg, Mike Zhang, Casper Rufaro Muziri, Saron Samuel, Sotaro Takeshita, Kun Kerdthaisong, Luca Foppiano, Rasul Dent, Tommaso Green, Ahmad Mustapha Wali, Kamohelo Makaaka, Vicky Feliren, Inshirah Idris, Hande Celikkanat, Abdulhamid Abubakar, Jean Maillard, Benoît Sagot, Thibault Clérice, Kenton Murray, Sarah K. K. Luger

Text Classification

Challenge:	Language identification (LID) is a fundamental step in curating multilingual corpora.
Approach:	They introduce CommonLID, a community-driven, human-annotated LID benchmark for the web domain, covering 109 languages.
Outcome:	The proposed benchmark covers 109 languages and shows that existing evaluations overestimate accuracy for many languages in the web domain.

Reference-less Measure of Faithfulness for Grammatical Error Correction (N18-2)

Copied to clipboard

Leshem Choshen, Omri Abend

Knowledge is a Region in Weight Space for Fine-tuned Language Models (2023.findings-emnlp)

Challenge:	Existing reference-less measures (RLMs) for measuring grammaticality are expensive to collect and limited by the large number of valid outputs.
Approach:	They propose a semantic measure for Grammatical Error Correction that compares the semantic symbolic structure of the source and correction without relying on manually-curated references.
Outcome:	The proposed measure shows that it can be applied consistently to ungrammatical text, and that valid corrections obtain a high USim similarity score to the source, and invalid corrections get lower scores.

Copied to clipboard

Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, Leshem Choshen

The Language of Legal and Illegal Activity on the Darknet (P19-1)

Challenge:	Specifically, language models that have been finetuned on the same dataset form a tight cluster in the weight space, while models finetuning on different datasets from the same underlying task form s looser clusters.
Approach:	They analyze the weight space and the underlying loss landscape of different models to determine how they are interconnected and how they perform.
Outcome:	The proposed method improves accuracy on 11 out of 12 datasets by starting from the center of the region, if not more, than using the pretrained model.

Copied to clipboard

Leshem Choshen, Dan Eldad, Daniel Hershcovich, Elior Sulem, Omri Abend

The Grammar-Learning Trajectories of Neural Language Models (2022.acl-long)

Challenge:	a study of the characteristics of text in the Darknet shows that it has legal and illegal content.
Approach:	They compare texts for selling legal and illegal drugs to a control condition . they find several distinguishing features between legal and illicit texts .
Outcome:	The authors compare legal and illegal texts to a clear net website with similar content as a control condition.

Copied to clipboard

Leshem Choshen, Guy Hacohen, Daphna Weinshall, Omri Abend

Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021.emnlp-main)

Challenge:	In this paper, we show that neural language models with different initialization, architecture, and training data acquire linguistic phenomena in a similar order, despite their different end performance.
Approach:	They propose to use mutual inductive bias to study linguistic representations implicit in NLMs.
Outcome:	The proposed approach shows that NLMs with different initialization, architecture, and training data acquire linguistic phenomena in a similar order, despite their different end performance.

Copied to clipboard

Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, Omri Abend

Natural Language Interfaces

Challenge:	Existing evaluation methods for factual consistency in knowledge-grounded dialogues are unreliable and limit their applicability.
Approach:	They propose an automatic evaluation metric for factual consistency in knowledge-grounded dialogue using automatic question generation and question answering.
Outcome:	The proposed evaluation metric consistently shows higher correlation with human judgements.

Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI (2024.naacl-demo)

Copied to clipboard

Elron Bandel, Yotam Perlitz, Elad Venezian, Roni Friedman, Ofir Arviv, Matan Orbach, Shachar Don-Yehiya, Dafna Sheinwald, Ariel Gera, Leshem Choshen, Michal Shmueli-Scheuer, Yoav Katz

Mediators in Determining what Processing BERT Performs First (2021.naacl-main)

Challenge:	Textual data processing pipelines are tailored to specific datasets, task and model combinations.
Approach:	They propose a library for customizable textual data preparation and evaluation tailored to generative language models.
Outcome:	Unitxt is a library for customizable textual data preparation and evaluation tailored to generative language models.

Copied to clipboard

Aviv Slobodkin, Leshem Choshen, Omri Abend

Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data (2026.findings-acl)

Challenge:	Probing neural models for the ability to perform downstream tasks using their activation patterns is often used to localize what parts of the network specialize in performing which tasks.
Approach:	They propose to consider the prediction’s context length as a potential mediating factor and consider the length of the span whose processing is minimally required to perform the prediction.
Outcome:	The proposed model can get 196 different rankings when probing with seven tasks, the authors show .

Copied to clipboard

Ofir Arviv, Kristjan Greenewald, Yotam Perlitz, Hadar Mulian, Michal Shmueli-Scheuer, Leshem Choshen

Green & Sustainable NLP

Challenge:	Current evaluation practices, typically employing fixed-size benchmarks, are inherently wasteful, continuing to the predetermined sample size even when the CI reaches 2.5, saving 80% of the evaluation cost.
Approach:	They propose an adaptive evaluation framework that combines sequential testing with stopping criteria tailored to common evaluation needs such as diminishing returns detection and minimum detectable effect size.
Outcome:	The proposed framework reduces computational cost and reliability while maintaining statistical significance.

GrASP: A Library for Extracting and Exploring Human-Interpretable Textual Patterns (2022.lrec-1)

Copied to clipboard

Piyawat Lertvittayakumjorn, Leshem Choshen, Eyal Shnarch, Francesca Toni

Semantics-aware Attention Improves Neural Machine Translation (2022.starsem-1)

Challenge:	a Python library is available for extracting patterns from textual data.
Approach:	They propose a Python library for extracting patterns from textual data . it integrates a public implementation of the existing GrASP algorithm .
Outcome:	The proposed library integrates a public implementation of the existing GrASP algorithm.

Copied to clipboard

Aviv Slobodkin, Leshem Choshen, Omri Abend

Machine Translation

Challenge:	Existing attempts to integrate semantic structures into NMT Transformers have failed .
Approach:	They propose two parameter-free methods for injecting semantic information into Transformers, using a Scene-Aware Self-Attention (SASA) head and a Scenario-Award Cross-Action (SACrA) head.
Outcome:	The proposed methods improve on the vanilla Transformer and syntax-aware models for four language pairs and show an additional gain when using both semantic and syntactic structures in some language pairs.

Will it Merge? On The Causes of Model Mergeability (2026.findings-acl)

Copied to clipboard

Adir Rahamim, Asaf Yehudai, Boaz Carmeli, Leshem Choshen, Yosi Mass, Yonatan Belinkov

DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation (2025.findings-acl)

Challenge:	Model merging has emerged as a promising technique for combining fine-tuned models into a single expert model without retraining.
Approach:	They propose a model merging technique that preserves weak model knowledge . they define mergeability as a property of model updates that captures how well they retain trained knowledge when merged with other model updates.
Outcome:	The proposed method preserves weak knowledge in the base model.

Copied to clipboard

Eliya Habba, Ofir Arviv, Itay Itzhak, Yotam Perlitz, Elron Bandel, Leshem Choshen, Michal Shmueli-Scheuer, Gabriel Stanovsky

ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning (2023.acl-long)

Challenge:	Recent work found that LLMs are sensitive to arbitrary prompt dimensions . this challenges traditional single-prompt evaluation practices .
Approach:	They present a large-scale dataset containing prompt perturbations of various evaluation benchmarks . they examine LLM sensitivity from an holistic perspective and assess the joint effects of perturbations along various dimensions .
Outcome:	The proposed dataset aims to democratize evaluation research and enable robust protocols . it includes more than 250M prompt perturbations and model outputs .

Copied to clipboard

Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Leshem Choshen

Mediocrity is the key for LLM as a Judge Anchor Selection (2026.acl-long)

Challenge:	Pretraining has been shown to scale well with compute, data size and data diversity.
Approach:	They propose a method that provides benefits of multitask learning but leverages distributed computation . they propose 'coldfusion' can create synergistic loop where finetuned models can be "recycled"
Outcome:	The proposed method outperforms RoBERTa and previous multitask models on 35 datasets.

Copied to clipboard

Shachar Don-Yehiya, Asaf Yehudai, Leshem Choshen, Omri Abend

Challenge:	a poor selection of an anchor can dramatically reduce correlation with human rankings . traditional reference-based metrics are often ill-suited for open-ended generation .
Approach:	They evaluate 22 different anchors on a Arena-Hard-v2.0 dataset and quantify the effect size of anchor selection.
Outcome:	The proposed model is better or worse than all other models, but it is rarely indicative of the relative ranking of the models.

Will it Blend? Blending Weak and Strong Labeled Data in a Neural Network for Argumentation Mining (P18-2)

Copied to clipboard

Eyal Shnarch, Carlos Alzate, Lena Dankin, Martin Gleize, Yufang Hou, Leshem Choshen, Ranit Aharonov, Noam Slonim

Reasoning

Challenge:	Obtaining high quality labeled data for natural language understanding tasks is slow, error-prone, complicated and expensive.
Approach:	They propose a method to blend weak and strong labeled data during the training of neural networks using a topic-dependent evidence detection dataset.
Outcome:	The proposed method improves the training of neural networks when a small amount of labeled data is available.

Navigating the Modern Evaluation Landscape: Considerations in Benchmarks and Frameworks for Large Language Models (LLMs) (2024.lrec-tutorials)

Copied to clipboard

Leshem Choshen, Ariel Gera, Yotam Perlitz, Michal Shmueli-Scheuer, Gabriel Stanovsky

Label-Efficient Model Selection for Text Generation (2024.acl-long)

Challenge:	General-purpose Language Models have changed the world of Natural Language Processing, if not the world itself.
Approach:	This tutorial will lay the foundations and explain the basics of evaluation and compare traditional methods to newly developed methods.
Outcome:	The tutorial assumes little familiarity with metrics, datasets, prompts and benchmarks . it will compare traditional methods to newly developed methods .

Copied to clipboard

Shir Ashury Tahan, Ariel Gera, Benjamin Sznajder, Leshem Choshen, Liat Ein-Dor, Eyal Shnarch

Human Learning by Model Feedback: The Dynamics of Iterative Prompting with Midjourney (2023.emnlp-main)

Challenge:	Model selection for a given task can entail extensive annotation of the quality of outputs of different models.
Approach:	They propose a model-agnostic method to make an informed decision between candidate text generation models based on preference annotations.
Outcome:	The proposed method reduces the required number of annotations by up to 75% while maintaining high evaluation reliability.

Copied to clipboard

Shachar Don-Yehiya, Leshem Choshen, Omri Abend

Low-Resource NLP

Challenge:	Generating images with Text-to-Image models often requires multiple trials, where human users iteratively update their prompt based on feedback, namely the output image.
Approach:	They compile a dataset of iterative interactions of human users with Midjourney and analyze the dynamics of the user prompts along these iterations.
Outcome:	The proposed model produces better images for a specific language style than other models.

Jump to Conclusions: Short-Cutting Transformers with Linear Transformations (2024.lrec-main)

Copied to clipboard

Alexander Yom Din, Taelin Karidi, Leshem Choshen, Mor Geva

Where to start? Analyzing the potential value of intermediate models (2023.emnlp-main)

Challenge:	Transformer-based language models create hidden representations of inputs at every layer, but only use final-layer representations for prediction.
Approach:	They propose a method for casting hidden representations as final representations, bypassing transformer computation in-between.
Outcome:	The proposed method produces more accurate predictions from hidden layers across various model scales, architectures, and data distributions.

Copied to clipboard

Leshem Choshen, Elad Venezian, Shachar Don-Yehiya, Noam Slonim, Yoav Katz

Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion (2024.emnlp-main)

Challenge:	a finetuned model may be better base models than the vanilla pretrained model . this scheme, often referred to as intertraining, is the focus of the present work .
Approach:	They propose a scheme to analyze the potential intertraining gain independently for the target dataset and for a base model being considered as a starting point.
Outcome:	The proposed model is strong even if training data was not aligned with target dataset.

Copied to clipboard

Kerem Zaman, Leshem Choshen, Shashank Srivastava

Reinforcement Learning with Large Action Spaces for Neural Machine Translation (2022.coling-1)

Challenge:	a recent study examined the effects of model fusion on learning of shortcuts and social biases in fine-tuned language models.
Approach:	They investigate whether model fusion can be used to reduce unwanted knowledge . they examine classification tasks with artificially fusioned models .
Outcome:	The proposed model fusion can reduce unshared knowledge, the authors show . their results show that the model merged models can improve performance and generalize .

Copied to clipboard

Asaf Yehudai, Leshem Choshen, Lior Fox, Omri Abend

Machine Translation

Challenge:	Recent work has argued that the gains produced by Reinforcement learning are mostly due to promoting tokens that have already received a fairly high probability in pre-training.
Approach:	They hypothesize that the large action space is a main obstacle to RL’s effectiveness in MT by reducing the size of the vocabulary without changing the vocabulary.
Outcome:	The proposed method improves by 1.5 BLEU points on average.

Automatic Metric Validation for Grammatical Error Correction (P18-1)

Copied to clipboard

Leshem Choshen, Omri Abend

Are You Convinced? Choosing the More Convincing Evidence with a Siamese Network (P19-1)

Challenge:	Existing methods for metric validation in GEC suffer from low inter-rater agreement.
Approach:	They propose an automatic method for GEC metric validation that overcomes many of the difficulties in the existing method.
Outcome:	The proposed method sheds new light on metric quality and shows valid edits are penalized by existing metrics.

Copied to clipboard

Martin Gleize, Eyal Shnarch, Leshem Choshen, Lena Dankin, Guy Moshkowich, Ranit Aharonov, Noam Slonim

Reasoning

Challenge:	Recent advances in argument detection have made it easier to identify the more convincing arguments.
Approach:	They propose a new data set of pairs of evidence labeled for convincingness that is more challenging than existing alternatives.
Outcome:	The proposed method outperforms baselines on convincingness data and its own.

Pretraining Language Models for Diachronic Linguistic Change Discovery (2026.findings-eacl)

Copied to clipboard

Elisabeth Fittschen, Sabrina Xin Li, Tom Lippincott, Leshem Choshen, Craig Messner

Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability (2024.findings-acl)

Challenge:	Large language models are increasingly used as knowledge discovery tools . historical linguistics and literary studies often construct arguments on the basis of distinctions between phenomena like time-period or genre.
Approach:	They propose to use LLMs to train large language models over modest historical corpora without allowing contamination from anachronistic data.
Outcome:	The proposed model better respects historical divisions and is more computationally efficient compared to the standard approach of fine-tuning an existing LLM.

Copied to clipboard

Afra Feyza Akyürek, Ekin Akyürek, Leshem Choshen, Derry Wijaya, Jacob Andreas

Active Learning for BERT: An Empirical Study (2020.emnlp-main)

Challenge:	Existing language models (LMs) generate factually correct text and estimate truth values of individual claims, but they do not reflect a coherent, manipulable model of the world.
Approach:	They propose a method that uses language models to identify implications of (and contradictions within) the text they generate.
Outcome:	The proposed method improves LM factuality by 3-26% across the CREAK, MQuAKE, and Reversal Curse datasets.

Copied to clipboard

Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, Noam Slonim

Unsupervised Expressive Rules Provide Explainability and Assist Human Experts Grasping New Domains (2020.findings-emnlp)

Challenge:	Existing approaches to deal with data scarcity are active learning (AL) and pre-trained models are not being considered.
Approach:	They propose to use active learning techniques to cope with data scarcity in binary text classification scenarios where the annotation budget is very small and the data is often skewed.
Outcome:	The proposed methods improve BERT performance in binary text classification scenarios where the annotation budget is very small and the data is often skewed.

Copied to clipboard

Eyal Shnarch, Leshem Choshen, Guy Moshkowich, Ranit Aharonov, Noam Slonim

Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation (2025.acl-long)

Challenge:	Understanding unexplored data is a slow process, and there is no labeled data at hand.
Approach:	They propose to use unsupervised methods to reveal rules which cluster unexplored corpus by its prominent categories to help domain experts understand their texts.
Outcome:	The proposed rules can be bootstrapped to identify target categories and deepen understanding of the data.

Copied to clipboard

Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, Sara Hooker

Multilinguality

Challenge:	Reliable multilingual evaluation is difficult and culturally appropriate evaluation is even harder to achieve.
Approach:	They propose a multilingual evaluation framework that aims to mitigate these biases by improving translations and annotation practices.
Outcome:	The proposed framework improves translation quality and cultural coverage and is culturally sensitive and culturally agnostic.

The ShareLM Collection and Plugin: Contributing Human-Model Chats for the Benefit of the Community (2025.acl-demo)

Copied to clipboard

Shachar Don-Yehiya, Leshem Choshen, Omri Abend