Papers by Yoav Goldberg

Adversarial Concept Erasure in Kernel Space (2022.emnlp-main)

Challenge:	Using large language models, we investigate the ability of large language model to generate correct responses to questions involving descriptions of faux pas situations.
Approach:	They investigate the ability of large language models to generate correct responses to questions involving descriptions of faux pas situations.
Outcome:	The proposed model fails to generate correct responses to presupposition questions and many of the correct responses are attributed to over-interpretation by the human reader.

Copied to clipboard

Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, Ryan Cotterell

CIKQA: Learning Commonsense Inference with a Unified Knowledge-in-the-loop QA Paradigm (2023.findings-eacl)

Challenge:	Large neural networks in NLP produce real-valued representations that encode the bit of human language that they were trained on.
Approach:	They propose a kernelization of the recently-proposed linear concept-removal objective and propose to remove linear subspaces from the representation space.
Outcome:	The proposed kernelization protects against the ability of nonlinear adversaries to recover the concept.

Copied to clipboard

Hongming Zhang, Yintong Huo, Yanai Elazar, Yangqiu Song, Yoav Goldberg, Dan Roth

Reasoning

Challenge:	Existing commonsense reasoning datasets target different knowledge types, modalities, and formats, but how to help machines acquire and infer over commonsensical knowledge is still unclear.
Approach:	They propose a commonsense reasoning benchmark to motivate commonsensing progress from two perspectives: (1) Evaluating whether models can distinguish knowledge quality by predicting if the knowledge is enough to answer the question or not.
Outcome:	The proposed model outperforms existing models in evaluating their generalization capabilities across tasks while demonstrating that distinguishing knowledge quality remains challenging for current models.

Asking It All: Generating Contextualized Questions for any Semantic Role (2021.emnlp-main)

Copied to clipboard

Valentina Pyatkin, Paul Roit, Julian Michael, Yoav Goldberg, Reut Tsarfaty, Ido Dagan

Question Generation

Challenge:	Existing approaches to question generation require conditioning on existing answers in text . previous work required human-curated templates, limiting coverage and question fluency .
Approach:	They propose a task of role question generation that produces a prototype and revises it to be contextually appropriate for the passage.
Outcome:	The proposed model generates diverse and well-formed questions for a large, broad-coverage ontology of predicates and roles.

Hierarchy Builder: Organizing Textual Spans into a Hierarchy to Facilitate Navigation (2023.acl-demo)

Copied to clipboard

Itay Yair, Hillel Taub-Tabib, Yoav Goldberg

MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents (2026.tacl-1)

Challenge:	Information extraction systems produce hundreds to thousands of strings on a specific topic.
Approach:	They propose a method that allows users to consume a large collection of related textual strings in an exploratory mode.
Outcome:	The proposed method allows users to consume a large collection of related textual strings in an exploratory mode.

Copied to clipboard

Tomer Wolfson, Harsh Trivedi, Mor Geva, Yoav Goldberg, Dan Roth, Tushar Khot, Ashish Sabharwal, Reut Tsarfaty

Reasoning

Challenge:	Automated agents powered by large language models are becoming more ingrained into how people seek information . but evaluation benchmarks for LLMs rarely feature natural questions that are time-consuming . a new benchmark, MoNaCo, aims to address this gap by eliciting and manually answering time-wasting questions .
Approach:	They propose a benchmark of 1,315 natural and time-consuming questions that require dozens of intermediate steps to solve.
Outcome:	MoNaCo benchmarks achieve at least 61.2% F1 in real-world time-consuming questions hampered by low recall and hallucinations . Frontier LLMs evaluated on MoN achieving at least 61% F1, harmed by low memory and halluzinations.

A Formal Hierarchy of RNN Architectures (2020.acl-main)

Copied to clipboard

William Merrill, Gail Weiss, Yoav Goldberg, Roy Schwartz, Noah A. Smith, Eran Yahav

Challenge:	Existing theories of expressive power of RNNs are limited.
Approach:	They propose a formal hierarchy of the expressive capacity of RNN architectures based on two formal properties: space complexity and rational recurrence.
Outcome:	The proposed model is based on the theory of “saturated” RNNs and shows that it obeys a similar hierarchy to unsaturated RNN models.

Unsupervised Mapping of Arguments of Deverbal Nouns to Their Corresponding Verbal Labels (2023.findings-acl)

Copied to clipboard

Aviv Weinstein, Yoav Goldberg

Low-Resource NLP

Challenge:	Deverbal nouns are nominal forms of verbs used in English texts to describe events or actions . many NLP systems neglect to handle nominalized constructions, resulting in limited applications .
Approach:	They propose to map arguments of deverbal nouns to universal-dependency relations of verbal constructions . they propose to use the same labels as verbal cases to map the arguments .
Outcome:	The proposed approach maps arguments of nominalized nouns to the corresponding verbal constructions.

McPhraSy: Multi-Context Phrase Similarity and Clustering (2022.findings-emnlp)

Copied to clipboard

Amir Cohen, Hila Gonen, Ori Shapira, Ran Levy, Yoav Goldberg

Text Clustering

Challenge:	Existing methods for estimating phrase similarity use the phrase context only during training, instead relying on the phrase itself.
Approach:	They propose a novel algorithm that leverages multiple contexts during inference to estimate the similarity of phrases based on multiple context.
Outcome:	The proposed method outperforms existing models on two phrase similarity datasets by 13.3% and a new task that relies on phrase similarities in the product reviews domain.

Does BERT Pretrained on Clinical Notes Reveal Sensitive Data? (2021.naacl-main)

Copied to clipboard

Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, Byron Wallace

Simple and Effective Multi-Token Completion from Masked Language Models (2023.findings-eacl)

Challenge:	Pretraining large (masked) language models over EHR data has yielded consistent performance gains across tasks.
Approach:	They propose to use large Transformers to release pretraining models over EHRs . they propose to recover patient names and conditions associated with them .
Outcome:	The proposed models recover patient names and conditions associated with patients . the proposed models share the model parameters for use by other researchers .

Copied to clipboard

Oren Kalinsky, Guy Kushilevitz, Alexander Libov, Yoav Goldberg

LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models (2022.emnlp-demos)

Challenge:	Pre-trained neural masked language models are limited to predicting a single token . recent pre-tried LMs like T5 do allow predicting multi-token completions, but are more expensive to train and run.
Approach:	They propose two ways to adapt pre-trained masked language models to produce multi-token completions.
Outcome:	The proposed method surpasses current state-of-the-art models while being more parameter efficient.

Copied to clipboard

Mor Geva, Avi Caciularu, Guy Dar, Paul Roit, Shoval Sadde, Micah Shlain, Bar Tamir, Yoav Goldberg

Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets (D19-1)

Challenge:	Transformer-based language models (LMs) are opaque and unexplained, causing problems for endusers and developers who wish to debug or fix their behaviour.
Approach:	They propose an interactive debugger tool for transformer-based LMs that provides a fine-grained interpretation of the model's internal prediction process and a powerful framework for intervening in LM behavior.
Outcome:	The proposed tool provides a fine-grained interpretation of the model's internal prediction construction process, and a powerful framework for intervening in LM behavior.

Copied to clipboard

Mor Geva, Yoav Goldberg, Jonathan Berant

Facts2Story: Controlling Text Generation by Key Facts (2020.coling-main)

Challenge:	Having only a few workers generate the majority of dataset examples raises concerns about data diversity .
Approach:	They perform a series of experiments to investigate annotator biases in recent NLU datasets . they find that models are able to recognize the most productive annotators .
Outcome:	The results show that models can recognize the most productive annotators and do not generalize well to examples from annotator that did not contribute to the training set.

Copied to clipboard

Eyal Orbach, Yoav Goldberg

Text Generation

Challenge:	Existing methods for story generation struggle with staying coherent for long periods of time.
Approach:	They propose a controlled generation task which expands a sequence of facts into a longer narrative.
Outcome:	The proposed model produces competitive fluency while adhering to the requested facts.

oLMpics-On What Language Model Pre-training Captures (2020.tacl-1)

Copied to clipboard

Alon Talmor, Yanai Elazar, Yoav Goldberg, Jonathan Berant

HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark (2023.findings-emnlp)

Challenge:	Recent success of pre-trained language models has spurred widespread interest in their capabilities.
Approach:	They propose an evaluation protocol that includes zero-shot evaluation and no fine-tuning . they propose to compare the learning curve of a fine- tuned LM to the learning of multiple controls .
Outcome:	The proposed evaluation protocol compares the learning curve of a fine-tuned LM to the learning of multiple controls.

Copied to clipboard

Amir Cohen, Hilla Merhav-Fine, Yoav Goldberg, Reut Tsarfaty

Guiding LLM to Fool Itself: Automatically Manipulating Machine Reading Comprehension Shortcut Triggers (2023.findings-emnlp)

Challenge:	Current benchmarks for Hebrew Natural Language Processing (NLP) focus mainly on morpho-syntactic tasks, neglecting the semantic dimension of language understanding.
Approach:	They propose to use Hebrew machine reading comprehension (MRC) as extractive Question Answering to address this problem.
Outcome:	The proposed benchmark features 30,147 question-answer pairs derived from both Hebrew Wikipedia articles and Israeli tech news.

Copied to clipboard

Mosh Levy, Shauli Ravfogel, Yoav Goldberg

Nakdan: Professional Hebrew Diacritizer (2020.acl-demos)

Challenge:	Recent applications of LLMs in Machine Reading Comprehension systems have shown impressive results, but the use of shortcuts has emerged as a potential threat to their reliability.
Approach:	They propose a framework that guides an editor to add potential shortcuts-triggers to samples.
Outcome:	The proposed framework can edit trigger shortcuts in samples that fool LLMs . it also shows that GPT4 can be deceived by its own edits (15% drop in F1).

Copied to clipboard

Avi Shmidman, Shaltiel Shmidman, Moshe Koppel, Yoav Goldberg

Structured Data in NLP

Challenge:	a system for automatic diacritization of Hebrew Text is available for both casual and expert users.
Approach:	They propose a system for automatic diacritization of Hebrew Text . the system combines declarative linguistic knowledge with machine learning models .
Outcome:	The proposed system is available for both casual and expert users.

Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages (N19-1)

Copied to clipboard

Shauli Ravfogel, Yoav Goldberg, Tal Linzen

Scalable Evaluation and Improvement of Document Set Expansion via Neural Positive-Unlabeled Learning (2021.eacl-main)

Challenge:	Recent studies have identified both strengths and limitations of recurrent neural networks (RNNs) in applied natural language processing tasks.
Approach:	They propose a paradigm that addresses typological differences between languages . they create synthetic versions of English and train them to predict agreement features .
Outcome:	The proposed model improves on predicting agreement with subject and object, suggesting that RNNs have a recency bias.

Copied to clipboard

Alon Jacovi, Gang Niu, Yoav Goldberg, Masashi Sugiyama

Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent (2021.emnlp-main)

Challenge:	Current methods for document set expansion for large collections are based on word-frequency or bag-of-words document similarity metrics.
Approach:	They propose to extend the IR approach by treating the problem as an instance of positive-unlabeled (PU) learning . they propose solutions for each challenge and empirically validate them with ablation tests .
Outcome:	The proposed method improves on a PubMed abstract retrieval task . it is compared with existing methods and empirically validated with ablation tests .

Copied to clipboard

William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, Noah A. Smith

Large Scale Substitution-based Word Sense Induction (2022.acl-long)

Challenge:	Evidence is emerging that neural networks learn due to inductive bias in the training routine, typically a variant of gradient descent (GD).
Approach:	They propose to characterize GD as an inductive bias in transformer training . they document norm growth in transformer language models and show they are saturated .
Outcome:	Empirically, we document norm growth in the training of transformer language models . the results suggest saturation is a new characterization of an inductive bias implicit in GD .

Copied to clipboard

Matan Eyal, Shoval Sadde, Hillel Taub-Tabib, Yoav Goldberg

Syntactic Search by Example (2020.acl-demos)

Challenge:	Word forms are ambiguous, and derive meaning from the context in which they appear . word sense induction can be performed over a corpus-derived sense inventory .
Approach:	They propose a word-sense induction method based on pre-trained masked language models . they train a static word embeddings algorithm on the sense-tagged corpus .
Outcome:	The proposed method outperforms existing senseful embeddings methods on Wikipedia and on an outlier detection dataset.

Copied to clipboard

Micah Shlain, Hillel Taub-Tabib, Shoval Sadde, Yoav Goldberg

Multimodality

Challenge:	a new system allows a user to search a large linguistically annotated corpus using syntactic patterns over dependency graphs.
Approach:	They propose a query language that allows a user to search a large linguistically annotated corpus using syntactic patterns over dependency graphs.
Outcome:	The proposed system searches the English wikipedia and English pubmed abstracts at a rapid speed.

Break It Down: A Question Understanding Benchmark (2020.tacl-1)

Copied to clipboard

Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, Jonathan Berant

Question Answering

Challenge:	Understanding natural language questions entails the ability to break down a question into the requisite steps for computing its answer.
Approach:	They introduce a Question Decomposition Meaning Representation (QDMR) for questions . they demonstrate that QDMRs can be annotated at scale using a hotpotQA dataset .
Outcome:	The proposed model outperforms several natural baselines in the open-domain question answering hotpotQA dataset and can be deterministically converted to a pseudo-SQL formal language.

A Dataset for N-ary Relation Extraction of Drug Combinations (2022.naacl-main)

Copied to clipboard

Aryeh Tiktinsky, Vijay Viswanathan, Danna Niezni, Dana Meron Azagury, Yosi Shamay, Hillel Taub-Tabib, Tom Hope, Yoav Goldberg

NERetrieve: Dataset for Next Generation Named Entity Recognition and Retrieval (2023.findings-emnlp)

Challenge:	Combination therapies are becoming standard of care for diseases such as cancer, tuberculosis, malaria and HIV.
Approach:	They construct an expert-annotated dataset for extracting drug combinations from the scientific literature.
Outcome:	The proposed dataset is the first relation extraction dataset consisting of variable-length relations.

Copied to clipboard

Uri Katz, Matan Vetzler, Amir Cohen, Yoav Goldberg

Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemes (2021.tacl-1)

Challenge:	Named Entity Recognition (NER) is a widely adopted NLP task . authors present three variants of NER task, with dataset to support them .
Approach:	They propose three variants of the NER task, together with a dataset to support them . they propose a move towards more fine-grained entities and zero-shot recognition .
Outcome:	The proposed model matches or surpasses existing models in NER tasks . the proposed model is based on a large, silver-annotated corpus of 4 million paragraphs .

Copied to clipboard

Ofer Sabo, Yanai Elazar, Yoav Goldberg, Ido Dagan

On the Practical Computational Power of Finite Precision RNNs for Language Recognition (P18-2)

Challenge:	a recent study has focused on few-shot learning (FSL) for relation classification, but it requires large amounts of training data.
Approach:	They propose a method for deriving more realistic few-shot test data from available datasets for supervised RC.
Outcome:	The proposed method yields a challenging benchmark for FSL-RC on which state of the art models show poor performance.

Copied to clipboard

Gail Weiss, Yoav Goldberg, Eran Yahav

Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data (2020.emnlp-main)

Challenge:	Recurrent Neural Networks (RNNs) are famously known to be Turing complete, but this relies on infinite precision in the states and unbounded computation time.
Approach:	They propose to use LSTM and Elman-RNN with ReLU activation to study RNNs . they show that LS and ReLU-RNns can easily implement counting behavior .
Outcome:	The LSTM and the Elman-RNN with ReLU activation are stronger than the RNN with squashing activation and the GRU.

Copied to clipboard

Shachar Rosenman, Alon Jacovi, Yoav Goldberg

Neighboring Words Affect Human Interpretation of Saliency Explanations (2023.findings-acl)

Challenge:	Existing methods for relation extraction (RE) use shallow heuristics that do not generalize to challenge-set data.
Approach:	They propose to annotate a dataset to test whether relation extraction models are generalized to the challenge-set data.
Outcome:	The proposed model performs better on the challenge-set compared with the SOTA models on the same dataset.

Copied to clipboard

Alon Jacovi, Hendrik Schuff, Heike Adel, Ngoc Thang Vu, Yoav Goldberg

Explainability & Interpretability in NLP

Challenge:	Recent studies found that superficial factors such as word length can distort human interpretation of the communicated saliency scores.
Approach:	They conduct a user study to examine how the marking of a word’s neighboring words affect the explainee’s perception of the word’ s importance in the context of . a saliency explanation.
Outcome:	The findings question whether text-based saliency explanations should continue to be communicated at word level and inform future research on alternative methods.

Data-driven Coreference-based Ontology Building (2024.findings-emnlp)

Copied to clipboard

Shir Ashury Tahan, Amir Cohen, Nadav Cohen, Yoram Louzoun, Yoav Goldberg

NER Retriever: Zero-Shot Named Entity Retrieval with Type-Aware Embeddings (2025.findings-emnlp)

Challenge:	a new ontology is based on coreference resolution, but it is not comprehensive . a recent study found that ontologies categorize concepts into groups and arrange them in a hierarchy .
Approach:	They derive coreference chains from a corpus of 30 million biomedical abstracts and construct a graph based on the string phrases within these chains.
Outcome:	The proposed ontology overlaps significantly with human-authored ontologies.

Copied to clipboard

Or Shachar, Uri Katz, Yoav Goldberg, Oren Glickman

Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them (N19-1)

Challenge:	NER Retriever uses a user-defined type description to retrieve documents mentioning entities of that type.
Approach:	They propose a zero-shot retrieval framework for ad-hoc Named Entity Recognition . a user-defined type description is used to retrieve documents mentioning entities of that type .
Outcome:	The proposed framework outperforms lexical and dense retrieval baselines on three benchmarks.

Copied to clipboard

Hila Gonen, Yoav Goldberg

Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? (2020.acl-main)

Challenge:	Existing methods to remove gender bias from word embeddings are insufficient, we argue . existing methods for gender-neutral modeling are ineffective, we conclude .
Approach:	They propose methods to reduce gender bias in word embeddings by debiasing them using text corpora.
Outcome:	The proposed methods show that they can reduce gender bias in word embeddings . the proposed methods are insufficient and should not be trusted, the authors argue .

Copied to clipboard

Alon Jacovi, Yoav Goldberg

Neural Extractive Search (2021.acl-demo)

Challenge:	Current approaches to interpretability evaluation focus on faithfulness criteria . current approaches focus on readability, plausibility and faithfulness .
Approach:	They argue that current binary definition of faithfulness sets unrealistic standards . they argue that a more graded definition would be of greater practical utility .
Outcome:	The proposed approach is based on three assumptions and lacks a graded definition of faithfulness.

Copied to clipboard

Shauli Ravfogel, Hillel Taub-Tabib, Yoav Goldberg

Measuring and Improving Consistency in Pretrained Language Models (2021.tacl-1)

Challenge:	a domain expert often needs to extract structured information from large corpora.
Approach:	They propose a search paradigm called "extractive search" that extends search queries with capture-slots to allow for rapid extraction.
Outcome:	The proposed search paradigm can be extended with neural similarity techniques.

Copied to clipboard

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, Yoav Goldberg

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2022.acl-short)

Challenge:	In this paper, we examine whether pretrained language models are consistent with factual knowledge.
Approach:	They propose a method to improve consistency of pretrained language models . consistency is a desirable property of a good language understanding model, they argue .
Outcome:	The proposed model improves consistency and shows that it is effective.

Copied to clipboard

Elad Ben Zaken, Yoav Goldberg, Shauli Ravfogel

Language Modeling for Code-Switching: Evaluation, Integration of Monolingual Data, and Discriminative Training (D19-1)

Challenge:	Large pre-trained models are expensive to train and deploy . large data sets make finetuning expensive to deploy - a new paradigm .
Approach:	They propose a sparse-finetuning method where only bias-terms are modified . they show that applying BitFit on pre-trained BERT models is competitive .
Outcome:	The proposed method is competitive with sparse-finetuning methods on small-to-medium training data.

Copied to clipboard

Hila Gonen, Yoav Goldberg

Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models (2024.eacl-long)

Challenge:	Code-switching (CS) is a linguistic phenomenon defined as "the alternation of two languages within a single discourse, sentence or constituent."
Approach:	They propose an ASR-motivated evaluation setup which is decoupled from an ASL system and the choice of vocabulary . they propose a discriminative training approach which works better than generative language modeling .
Outcome:	The proposed evaluation setup is better than generative language modeling, the authors show . the proposed setup is decoupled from an ASR system and the choice of vocabulary .

Copied to clipboard

Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, Vered Shwartz

Aligning Vector-spaces with Noisy Supervised Lexicon (N19-1)

Challenge:	Recent work suggests that Large Language Models (LLMs) exhibit Neural Theory-of-Mind (N-ToM) however, prior work reached conflicting conclusions regarding those abilities.
Approach:	They examine the extent of Large Language Models’ N-ToM abilities through an extensive evaluation of 6 tasks and find that LLMs struggle with adversarial examples .
Outcome:	The proposed metrics show that LLMs exhibit certain N-ToM abilities, but this behavior is far from robust.

Copied to clipboard

Noa Yehezkel Lubin, Jacob Goldberger, Yoav Goldberg

Representation Learning

Challenge:	Current approaches to learning to translate between two vector spaces assume that the lexicon defines alignment pairs is noise-free.
Approach:	They propose a model that accounts for noisy pairs and propose supervised learning problems for this problem.
Outcome:	The proposed model significantly improves translation accuracy on bilingual word embedding translation and mapping between diachronic embeddable spaces.

Word Sense Induction with Neural biLM and Symmetric Patterns (D18-1)

Copied to clipboard

Asaf Amrami, Yoav Goldberg

Erratum: Measuring and Improving Consistency in Pretrained Language Models (2021.tacl-1)

Challenge:	Existing methods for word sense induction use a language model to predict probable substitutes for target words.
Approach:	They propose to use a language model to predict probable substitutes for target words . they replace the ngram-based language model with a recurrent model to generate strong substitute vectors .
Outcome:	The proposed method surpasses the current state-of-the-art on the SemEval 2013 task by a large margin.

Copied to clipboard

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, Yoav Goldberg

Ab Antiquo: Neural Proto-language Reconstruction (2021.naacl-main)

Challenge:	During production of this paper, an error was introduced to the formula on the bottom of the right column of page 1020.
Approach:	the formula was changed in the last two terms of the paper .
Outcome:	the correct formula is now available on the web.

Copied to clipboard

Carlo Meloni, Shauli Ravfogel, Yoav Goldberg

Split and Rephrase: Better Evaluation and Stronger Baselines (P18-2)

Challenge:	Historical linguists have identified regularities in the process of historic sound change.
Approach:	They propose a method to reconstruct proto-words based on cognates in daughter languages . they use a dataset of 8,000 comparative entries to analyze phonological changes .
Outcome:	The proposed method outperforms conventional methods in a proto-word reconstruction task.

Copied to clipboard

Roee Aharoni, Yoav Goldberg

Challenge:	a dataset mapping a complex sentence to a sequence of sentences conveying the same meaning is challenging in NLP.
Approach:	They propose a neural split and a copy-mechanism to break a complex sentence into several shorter sentences that convey the same meaning.
Outcome:	The proposed model outperforms the baseline model by 8.68 BLEU and further improves on the task.

Log-linear Guardedness and its Implications (2023.acl-long)

Copied to clipboard

Shauli Ravfogel, Yoav Goldberg, Ryan Cotterell

Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals (2021.tacl-1)

Challenge:	Existing methods for erasing human-interpretable concepts from neural representations that assume linearity are not fully understood.
Approach:	They define linear guardedness as the inability of an adversary to predict the concept directly from the representation . they show that a log-linear model can be constructed that indirectly recovers the concept .
Outcome:	The proposed model can be constructed that indirectly recovers the erased concept in some cases.

Copied to clipboard

Yanai Elazar, Shauli Ravfogel, Alon Jacovi, Yoav Goldberg

Explainability & Interpretability in NLP

Challenge:	Amnesic probing is a method that focuses on how information is being used, rather than on what information is encoded.
Approach:	They propose a method that focuses on how the information is being used rather than on what information is encoded.
Outcome:	The proposed method is based on a BERT dataset to ask questions that were not possible before . it shows that probing performance is not correlated to task importance .

Understanding Transformer Memorization Recall Through Idioms (2023.eacl-main)

Copied to clipboard

Adi Haviv, Ido Cohen, Jacob Gidron, Roei Schuster, Yoav Goldberg, Mor Geva

Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection (2020.acl-main)

Challenge:	Existing methods for analyzing memorization use definitions that are based on model performance, which changes between models and often also between training runs.
Approach:	They propose idioms as inputs that typically trigger memory recall and propose a set of English idiomas to test their methodological framework for probing and characterizing recall of memorized sequences in transformer LMs.
Outcome:	The proposed framework compares model behavior on memorized vs. non-memorized inputs across different model sizes and architectures.

Copied to clipboard

Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, Yoav Goldberg

Text Classification

Challenge:	Word embeddings, pre-trained language models, and deep learning methods are becoming effective for text classification.
Approach:	They propose a method for removing information from neural representations using null-space projection.
Outcome:	The proposed method mitigates bias in word embeddings and increases fairness in multi-class classification.

Breaking NLI Systems with Sentences that Require Simple Lexical Inferences (P18-2)

Copied to clipboard

Max Glockner, Vered Shwartz, Yoav Goldberg

Challenge:	a new test set shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge.
Approach:	They create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge.
Outcome:	The new examples are simpler than the SNLI test set, but the state-of-the-art systems perform poorly on it.

Conformal Nucleus Sampling (2023.findings-acl)

Copied to clipboard

Shauli Ravfogel, Yoav Goldberg, Jacob Goldberger

Challenge:	Modern language generation methods employ one of a handful of standard decoding strategies . greedy search and other methods generate dull text or degenerate text .
Approach:	They employ a calibration procedure to calibrate the parameter p as a function of the entropy of the next word distribution.
Outcome:	The proposed method overconfidently selects the word with the highest probability . the greedy method and its beam search variations tend to return dull text or degenerate text .

Adversarial Removal of Demographic Attributes from Text Data (D18-1)

Copied to clipboard

Yanai Elazar, Yoav Goldberg

Robustness in NLP

Challenge:	Recent advances in Representation Learning and Adversarial Training remove unwanted features from the learned representation.
Approach:	They show that demographic information of authors is encoded in the intermediate representations learned by text-based neural classifiers.
Outcome:	The proposed approach achieves higher accuracies on the same dataset, the authors show . they show that the proposed approach is effective in removing unwanted features from the learned representations.

Aligning Faithful Interpretations with their Social Attribution (2021.tacl-1)

Copied to clipboard

Alon Jacovi, Yoav Goldberg

Transfer Learning Between Related Tasks Using Expected Label Proportions (D19-1)

Challenge:	a common pattern of faithfulness is decomposition of a model into steps and inspecting the intermediate steps.
Approach:	They propose to formalize faithfulness as an accurate attribution of causality to the model . they propose an alternative causal chain to remedy the issues .
Outcome:	The proposed model interpretations fail because they are inconsistent and vague . the proposed model is based on a social attribution model and a highlight method .

Copied to clipboard

Matan Ben Noach, Yoav Goldberg

Sentiment Analysis

Challenge:	Existing methods of data supervision are limited by labeled training data.
Approach:	They propose a method where models are trained based on expected label proportions.
Outcome:	The proposed method improves on a sentence-level sentiment predictor and is cumulative with LM-based pretraining.

Text-based NP Enrichment (2022.tacl-1)

Copied to clipboard

Yanai Elazar, Victoria Basmov, Yoav Goldberg, Reut Tsarfaty

Challenge:	Existing NLP tasks and benchmarks do not cover all NP-mediated relations . we aim to enrich each NP in a text with all the preposition-mediated relationships that hold between it and other NPs in the text.
Approach:	They propose a task to enrich NPs with preposition-mediated relations that hold between them . they build a large-scale dataset and analyze the data to test the task .
Outcome:	The proposed task is based on a large-scale dataset and fine-tuned language models.

Lexical Generalization Improves with Larger Models and Longer Training (2022.findings-emnlp)

Copied to clipboard

Elron Bandel, Yoav Goldberg, Yanai Elazar

Challenge:	Excessive utilization of lexical overlap heuristics can lead to failure on challenging inputs.
Approach:	They analyze the use of lexical overlap heuristics in natural language inference, paraphrase detection, and reading comprehension using a contrastive dataset.
Outcome:	The proposed model can be used to improve performance on a wide range of tasks, but it is often wrong.

Conjunct Resolution in the Face of Verbal Omissions (2023.acl-long)

Copied to clipboard

Royi Rassin, Yoav Goldberg, Reut Tsarfaty

Challenge:	Verbal omissions occur when verbs and arguments are omitted from subsequent clauses . state-of-the-art models struggle with this task, but have limited results .
Approach:	They propose a conjunct resolution task that uses a split-and-rephrase paradigm to recover verbal omissions . they propose omitted words in bold and omitted words in red .
Outcome:	The proposed method performs decently, but leaves ample room for improvement.

Including Signed Languages in Natural Language Processing (2021.acl-long)

Copied to clipboard

Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, Malihe Alikhani

Challenge:	Existing research in Sign Language Processing (SLP) rarely explores signed languages . authors urge adoption of an efficient tokenization method and the collection of real-world signed language data .
Approach:	They propose to include signed languages as a research area with high social and scientific impact . they review the limitations of current SLP models and identify the open challenges .
Outcome:	The proposed model should include signed languages as a research area with high social and scientific impact.

Knowledge Navigator: LLM-guided Browsing Framework for Exploratory Search in Scientific Literature (2024.findings-emnlp)

Copied to clipboard

Uri Katz, Mosh Levy, Yoav Goldberg

A Practical Method for Generating String Counterfactuals (2025.findings-naacl)

Challenge:	Knowledge Navigator organizes retrieved documents into a navigable, two-level hierarchy of named and descriptive topics and subtopics.
Approach:	They propose to organize retrieved scientific documents into a navigable, two-level hierarchy of named and descriptive topics and subtopics.
Outcome:	The proposed system provides an overall view of the research themes in a domain while also enabling iterative search and deeper knowledge discovery within specific subtopics.

Copied to clipboard

Matan Avitan, Ryan Cotterell, Yoav Goldberg, Shauli Ravfogel

Challenge:	Interventions targeting the representation space of language models (LMs) have emerged as an effective means to influence model behavior.
Approach:	They propose a method to convert representation counterfactuals into string counterf actuals and analyze the linguistic alterations corresponding to the intervention.
Outcome:	The proposed method analyzes linguistic alterations corresponding to a given representation space intervention and interprets features utilized to encode a specific concept.

pyBART: Evidence-based Syntactic Transformations for IE (2020.acl-demos)

Copied to clipboard

Aryeh Tiktinsky, Yoav Goldberg, Reut Tsarfaty

LingMess: Linguistically Informed Multi Expert Scorers for Coreference Resolution (2023.eacl-main)

Challenge:	Syntactic dependencies are designed to accurately reflect syntactical relations, but they do not make semantic relations explicit.
Approach:	They propose a Python library for converting English Enhanced UD trees to Enhanced or Enhanced representations.
Outcome:	The proposed representations are linguistically sound and make lexical relations explicit . the proposed representation scores higher than Enhanced UD graphs, while requiring fewer patterns.

Copied to clipboard

Shon Otmazgin, Arie Cattan, Yoav Goldberg

Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema (2021.emnlp-main)

Challenge:	Current coreference systems use a single pairwise scoring component to assign mentions a score . different kinds of mentions require different information sources to assess their score - a problem that requires many decisions .
Approach:	They propose a linguistically motivated categorization of mention-pairs into 6 types of coreference decisions and learn a dedicated scoring function for each category.
Outcome:	The proposed model significantly improves the pairwise scorer and overall performance on the English Ontonotes coreference corpus and 5 additional datasets.

Copied to clipboard

Yanai Elazar, Hongming Zhang, Yoav Goldberg, Dan Roth

Reasoning

Challenge:	Pre-trained language models have boosted performance on some WS benchmarks, but the source of improvement is not clear.
Approach:	They propose a method that uses twin sentences for evaluation and two new baselines that account for artifacts in WS benchmarks.
Outcome:	The proposed evaluation method is suboptimal for the Winograd Schema . it uses twin sentences to account for commonsense reasoning abilities .

F-coref: Fast, Accurate and Easy to Use Coreference Resolution (2022.aacl-demo)

Copied to clipboard

Shon Otmazgin, Arie Cattan, Yoav Goldberg

Evaluating D-MERIT of Partial-annotation on Information Retrieval (2024.emnlp-main)

Challenge:	Existing models for coreference resolution are difficult to implement, consume a lot of GPU memory and take long to process each document.
Approach:	They propose a python package for fast, accurate, and easy-to-use English coreference resolution.
Outcome:	The proposed model can process 2.8K OntoNotes documents in 25 seconds on a V100 GPU, compared to 6 minutes for the LingMess model and 12 minutes of the popular AllenNLP coreference model.

Copied to clipboard

Royi Rassin, Yaron Fairstein, Oren Kalinsky, Guy Kushilevitz, Nachshon Cohen, Alexander Libov, Yoav Goldberg

Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora (2020.acl-main)

Challenge:	Using partially-annotated datasets for evaluation can lead to false conclusions . a dataset containing only a subset of relevant passages might result in misleading rankings .
Approach:	They propose to use a Wikipedia passage retrieval evaluation set to contain all relevant passages for each query.
Outcome:	The proposed dataset can be downloaded from https://d-merit.github.io.

Copied to clipboard

Hila Gonen, Ganesh Jawahar, Djamé Seddah, Yoav Goldberg

Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation (N19-1)

Challenge:	comparing two corpus texts and searching for words that differ in their usage between them is a common problem in digital humanities and computational social science.
Approach:	They propose an alternative approach that does not use vector space alignment, and instead considers the neighbors of each word.
Outcome:	The proposed method is interpretable and stable in 9 different setups and is highly reliable.

Copied to clipboard

Amit Moryossef, Yoav Goldberg, Ido Dagan

Text Generation

Challenge:	Modern neural generation systems conflate these two steps into a single end-to-end differentiable system.
Approach:	They propose to split the generation process into a symbolic text-planning stage that is faithful to the input, followed by a neural generation stage that focuses only on realization.
Outcome:	The proposed method improves reliability and adequacy while maintaining fluent output.

Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand? (2021.tacl-1)

Copied to clipboard

William Merrill, Yoav Goldberg, Roy Schwartz, Noah A. Smith

A Two-Stage Masked LM Method for Term Set Expansion (2020.acl-main)

Challenge:	Language models trained on billions of tokens have recently led to unprecedented results on many NLP tasks.
Approach:	They examine whether assertions enable a system to emulate representations preserving semantic relations like equivalence.
Outcome:	The proposed model can emulate representations preserving semantic relations like equivalence, but it can become uncomputable for classes of languages where expressions can take different values in different contexts.

Copied to clipboard

Guy Kushilevitz, Shaul Markovitch, Yoav Goldberg

SetExpander: End-to-end Term Set Expansion Based on Multi-Context Term Embeddings (C18-2)

Challenge:	Existing methods for Term Set Expansion are either distributional or pattern-based . Term set expansion is a task of expanding a small seed set of example terms into a larger set of terms that belong to the same semantic category.
Approach:	They propose a method which uses neural masked language models to expand a small seed set of terms into a larger set of semantic terms.
Outcome:	The proposed method outperforms state-of-the-art methods due to the small seed set size . it uses neural masked language models to query large, pre-trained mlms .

Copied to clipboard

Jonathan Mamou, Oren Pereg, Moshe Wasserblat, Ido Dagan, Yoav Goldberg, Alon Eirew, Yael Green, Shira Guskin, Peter Izsak, Daniel Korat

Factual Retrieval in LLMs Is a Redundant, Distributed and Non-Contiguous Process (2026.acl-long)

Challenge:	SetExpander is a corpus-based system for expanding a seed set of terms into a more complete set of words belonging to the same semantic class.
Approach:	They propose to use a corpus-based system for expanding a seed set of terms into a more complete set of words that belong to the same semantic class.
Outcome:	The proposed system can expand a seed set of terms, validate it, re-expand the expanded set and store it, thus simplifying the extraction of domain-specific fine-grained semantic classes.

Copied to clipboard

Hail Hochman, Natalie Shapira, Yoav Goldberg

Bootstrapping Relation Extractors using Syntactic Search by Examples (2021.eacl-main)

Challenge:	Existing work posits that factual knowledge is stored at the last entity token position, but the precise mechanics of how facts are retrieved from model parameters remain unclear.
Approach:	They propose an iterative patching protocol to identify a minimal subset of layers necessary for attribute retrieval.
Outcome:	The proposed method shows that models possess multiple paths for the same entity and fact, highlighting a high degree of redundancy in attribute computation.

Copied to clipboard

Matan Eyal, Asaf Amrami, Hillel Taub-Tabib, Yoav Goldberg

Text Generation

Challenge:	Existing methods for supervised relation extraction still require a large quantity of training data.
Approach:	They propose a process for bootstrapping training datasets which can be performed quickly by non-NLP-experts.
Outcome:	The proposed method outperforms models trained on manual and distant data augmentation techniques and the search-based approach with the NLG method.

Linguistically Motivated Sign Language Segmentation (2023.findings-emnlp)

Copied to clipboard

Amit Moryossef, Zifan Jiang, Mathias Müller, Sarah Ebling, Yoav Goldberg

Compressing Pre-trained Language Models by Matrix Decomposition (2020.aacl-main)

Challenge:	Sign language segmentation is a crucial task in sign language processing systems.
Approach:	They propose to combine two kinds of segmentation: segmentation into individual signs and segmentation to segment into phrases, larger units comprising several signs.
Outcome:	The proposed model is based on linguistic cues observed in sign language corpora and replaces the predominant IO tagging scheme with BIO taging to account for continuous signing.

Copied to clipboard

Matan Ben Noach, Yoav Goldberg

Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space (2022.emnlp-main)

Challenge:	Large pre-trained language models reach state-of-the-art results when fine-tuned individually; They also come with a significant memory and computational requirements, calling for methods to reduce model sizes (green AI).
Approach:	They propose a two-stage model-compression method to reduce a model’s inference time cost by decompressing the model into smaller matrices and performing feature distillation on the internal representation.
Outcome:	The proposed method reduces the number of parameters by 0.4x and increases inference speed by 1.45x while preserving the information contained within the model.

Copied to clipboard

Mor Geva, Avi Caciularu, Kevin Wang, Yoav Goldberg