Papers by Gabriel Stanovsky
Leveraging Collection-Wide Similarities for Unsupervised Document Structure Extraction (2024.findings-acl)
Copied to clipboard
| Challenge: | Document collections of various domains share some underlying collection-wide structure . structure can be useful in various use cases across different domains, such as legal, medical, or financial . |
| Approach: | They propose to identify the typical structure of document within a collection by using header paraphrases to ground topics to respective document locations. |
| Outcome: | The proposed method extracts meaningful collection-wide structure from documents in three domains in English and Hebrew. |
Crowdsourcing Question-Answer Meaning Representations (N18-2)
Copied to clipboard
| Challenge: | Existing datasets for predicate-argument relationships are lacking highly skilled and trained annotators. |
| Approach: | They propose a crowdsourcing scheme to generate question-answer pairs that represent predicate-argument relationships in sentences as a set of question-announcer pairs. |
| Outcome: | The proposed model covers the vast majority of predicate-argument relationships in existing datasets along with many previously under-resourced ones, including implicit arguments and relations. |
ScheMatiQ: From Research Question to Structured Data through Interactive Schema Discovery (2026.acl-demo)
Copied to clipboard
| Challenge: | a new approach to natural-language research questions requires manual effort to generate an annotation schema and label the corpus. |
| Approach: | They propose a natural-language search tool that takes a question and a corpus to produce a schema and db with a web interface that lets steer and revise the extraction. |
| Outcome: | The proposed model yields outputs that support real-world analysis in law and computational biology. |
A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank (2022.naacl-main)
Copied to clipboard
| Challenge: | Pretraining languages improve cross-lingual transfer for BERT-based models . Interestingly, PLMs exhibit zero-shot cross-linguistic abilities on downstream examples in languages seen only during pretraining. |
| Approach: | They develop a quadratic time complexity method to estimate pretraining languages' relations between linguistic features and two downstream tasks. |
| Outcome: | The proposed method is effective on a diverse set of languages spanning different linguistic features and two downstream tasks. |
Evaluating Question Answering Evaluation (D19-58)
Copied to clipboard
| Challenge: | Existing n-gram based QA metrics have a number of drawbacks and are not suitable for all extractive tasks. |
| Approach: | They propose to use BERTScore to evaluate translation for question answering (QA) they also explore whether existing n-gram based metrics are suitable for generative QA . |
| Outcome: | The proposed BERTScore metric fails to provide stronger correlation with human judgements . |
Can LLMs Help Uncover Insights about LLMs? A Large-Scale, Evolving Literature Analysis of Frontier LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Recent surveys of literature highlight the overwhelming growth of Large Language Models (LLMs). |
| Approach: | They propose a semi-automated literature analysis approach that automates literature analysis using LLMs. |
| Outcome: | The proposed approach reduces paper surveying and data extraction by 93% compared to manual methods. |
The Right Tool for the Job: Matching Model and Instance Complexities (2020.acl-main)
Copied to clipboard
| Challenge: | a large increase in the size of NLP models can increase production costs and reduce adoption on real-time devices. |
| Approach: | They propose a modification to contextual representation fine-tuning which allows for an early exit from neural network calculations for simple instances and late exit for hard instances. |
| Outcome: | The proposed method produces models which are up to five times faster than the state of the art while preserving their accuracy. |
Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach (2021.emnlp-main)
Copied to clipboard
| Challenge: | cuneiform clay tablets were written in 2500 BCE - 100 CE and are a target of extensive transcription and transliteration efforts due to their deterioration. |
| Approach: | They propose to use a masked language modelling task to complete missing text given cuneiform clay tablets written on cuniform signswedges (2500 BCE - 100 CE) they develop models which automatically complete these missing signs based on contextual cues and greedy decoding schemes. |
| Outcome: | The proposed models perform well on missing token prediction (89% hit@5) despite data scarcity (1M tokens), and human evaluations show that they are able to transcribe texts in extinct languages. |
ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Moments (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LLMs are highly sensitive to prompt phrasing, yet standard benchmarks typically report performance using a single prompt, raising concerns about the reliability of such evaluations. |
| Approach: | They propose a stochastic method of moments evaluation over the space of meaning-preserving prompt perturbations and propose resamplings to estimate the number of prompt re-sampleds needed to obtain meaningful results. |
| Outcome: | The proposed method is model-, task-, and metric-agnostic, offering a recipe for meaningful and robust evaluation. |
Time to Talk: LLM Agents for Asynchronous Group Communication in Mafia Games (2025.findings-emnlp)
Copied to clipboard
| Challenge: | LLMs are used in synchronous communication, where a human user and a model communicate in alternating turns. |
| Approach: | They develop an adaptive asynchronous LLM agent consisting of two modules that decide what to say and a scheduler that decides when to say it. |
| Outcome: | The proposed agent performs on par with human players in online Mafia games and in its ability to blend in with the other human players. |
Evaluating Gender Bias in Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Using morphological analysis, we find that MT models exhibit gender-biased translation errors when training data encode stereotypes not relevant for the task. |
| Approach: | They propose an automatic gender bias evaluation method for eight target languages with grammatical gender based on morphological analysis. |
| Outcome: | The proposed method is based on two recent coreference resolution datasets composed of English sentences cast participants into non-stereotypical gender roles. |
Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have found evidence of gender bias in machine translation and coreference resolution models using mostly synthetic diagnostic datasets. |
| Approach: | They propose a semi-automatic method to vastly extend synthetic, small diagnostic datasets to include grammatical patterns indicating stereotypical and non-stereotypical gender-role assignments. |
| Outcome: | The proposed method extends the existing dataset to 108K diverse English sentences. |
Process-Level Representation of Scientific Protocols with Interactive Annotation (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing efforts to automate wet lab workflows are focusing on graph-prediction models that capture both concrete, exact quantities ("30 minutes") and vague instructions ("swirl") |
| Approach: | They manually annotate PEGs in a corpus of complex lab protocols with a novel interactive textual simulator that keeps track of entity traits and semantic constraints during annotation. |
| Outcome: | The proposed graph-prediction models are good at entity identification and local relation extraction while addressing challenges such as cross-sentence relations and long-range coreference. |
Supervised Open Information Extraction (N18-1)
Copied to clipboard
| Challenge: | Existing methods for Open Information Extraction (Open IE) use semisupervised approaches or rule-based algorithms. |
| Approach: | They propose a supervised approach to Open Information Extraction (Open IE) they build on recent deep Semantic Role Labeling models to extract Open IE tuples . |
| Outcome: | The proposed model outperforms state-of-the-art Open IE systems on benchmark datasets. |
DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation (2025.findings-acl)
Copied to clipboard
Eliya Habba, Ofir Arviv, Itay Itzhak, Yotam Perlitz, Elron Bandel, Leshem Choshen, Michal Shmueli-Scheuer, Gabriel Stanovsky
| Challenge: | Recent work found that LLMs are sensitive to arbitrary prompt dimensions . this challenges traditional single-prompt evaluation practices . |
| Approach: | They present a large-scale dataset containing prompt perturbations of various evaluation benchmarks . they examine LLM sensitivity from an holistic perspective and assess the joint effects of perturbations along various dimensions . |
| Outcome: | The proposed dataset aims to democratize evaluation research and enable robust protocols . it includes more than 250M prompt perturbations and model outputs . |
Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that supervised models exploit data artifacts to achieve good test scores while their performance severely degrades on samples outside their training distribution. |
| Approach: | They propose a method which automatically generates contrast sets for the visual question answering task by using a semantic input representation. |
| Outcome: | The proposed method computes the answer of perturbed questions, thus reducing annotation cost and enabling thorough evaluation of models’ performance on various semantic aspects. |
Schema-Driven Information Extraction from Heterogeneous Tables (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on information extraction from tables has focused on developing custom pipelines for each table collection. |
| Approach: | They propose a task that transforms tabular data into structured records following a human-authored schema. |
| Outcome: | The proposed task achieves F1 scores ranging from 74.2 to 96.1 while maintaining cost efficiency. |
🧑🍳 Cooking Up Creativity: Enhancing LLM Creativity through Structured Recombination (2026.tacl-1)
Copied to clipboard
| Challenge: | Large Language Models excel at many tasks, yet struggle to generate truly creative ideas. |
| Approach: | They propose a novel approach that enhances Large Language Models' creativity by manipulating structured representations of existing ideas. |
| Outcome: | The proposed model outperforms GPT-4o in novelty and diversity and outperformed GPT-0 in creative generation. |
MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing reading comprehension metrics rely on token overlap and are agnostic to the nuances of reading comprehension. |
| Approach: | They propose a benchmark for training and evaluating generative reading comprehension metrics: MOdeling Correctness with Human Annotations. |
| Outcome: | The proposed benchmark outperforms baseline metrics by 10 to 36 absolute Pearson points on held-out annotations. |
Are Layout-Infused Language Models Robust to Layout Distribution Shifts? A Case Study with Scientific Documents (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work has shown that infusing layout features into language models improves processing of visually-rich documents such as scientific papers. |
| Approach: | They propose a method to evaluate layout-infused language models that incorporate layout features into their models to emulate layout distribution shifts. |
| Outcome: | The proposed model performs better under layout distribution shifts than in-distribution conditions. |
A Computational Acquisition Model for Multimodal Word Categorization (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent advances in self-supervised modeling of text and images open new opportunities for computational models of child language acquisition. |
| Approach: | They propose a multimodal language acquisition model trained from image-caption pairs on naturalistic data using cross-modal self-supervision. |
| Outcome: | The proposed model learns word categories and object recognition abilities, the authors show . their model is trained from image-caption pairs on naturalistic data using cross-modal self-supervision . |
Navigating the Modern Evaluation Landscape: Considerations in Benchmarks and Frameworks for Large Language Models (LLMs) (2024.lrec-tutorials)
Copied to clipboard
| Challenge: | General-purpose Language Models have changed the world of Natural Language Processing, if not the world itself. |
| Approach: | This tutorial will lay the foundations and explain the basics of evaluation and compare traditional methods to newly developed methods. |
| Outcome: | The tutorial assumes little familiarity with metrics, datasets, prompts and benchmarks . it will compare traditional methods to newly developed methods . |
Cross-document Coreference Resolution over Predicted Mentions (2021.findings-acl)
Copied to clipboard
| Challenge: | Cross-document coreference resolution has been under-explored in recent years . however, the challenge of cross-document resolution remains relatively under-studied . |
| Approach: | They propose a model for cross-document coreference resolution from raw text that extends a prominent withindocument corefer model to the CD setting. |
| Outcome: | The proposed model achieves competitive results for event and entity coreference resolution on gold mentions. |
Controlled Crowdsourcing for High-Quality QA-SRL Annotation (2020.acl-main)
Copied to clipboard
Paul Roit, Ayal Klein, Daniela Stepanov, Jonathan Mamou, Julian Michael, Gabriel Stanovsky, Luke Zettlemoyer, Ido Dagan
| Challenge: | Question-answer driven Semantic Role Labeling (QA-SRL) is an open and natural flavour of SRL, potentially attainable from laymen. |
| Approach: | They propose a question-answer driven semantic role labeling approach that uses question-announced questions to label predicate-argument relationships. |
| Outcome: | The proposed method yields high-quality annotation with dramatically higher coverage, enabling future replicable research of natural semantic annotations. |
On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent work shows that deep learning models are sensitive to low-level correlations between simple features and specific output labels, leading to over-fitting and lack of generalization. |
| Approach: | They propose to eliminate single-word correlations altogether to mitigate this problem . they highlight several alternatives to dataset balancing to enhance contexts . |
| Outcome: | The proposed approach to balancing datasets is insufficient, the authors argue . they suggest enhancing datasets with richer contexts and abstaining from interaction . |
PromptSuite: A Task-Agnostic Framework for Multi-Prompt Generation (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Recent studies have demonstrated that LLMs are highly sensitive to small, meaning-preserving variations in task formulation. |
| Approach: | They propose a framework that enables the automatic generation of various prompts. |
| Outcome: | The proposed framework provides meaningful variations to support strong evaluation practices. |
A Large-Scale Multilingual Study of Visual Constraints on Linguistic Selection of Descriptions (2023.findings-eacl)
Copied to clipboard
| Challenge: | a multilingual study examines how vision constrains linguistic choice . we use existing annotations to investigate the effect of different visual conditions on numeral expressions in captions . |
| Approach: | They propose a method that leverages existing corpora of images with captions written by native speakers to constrain linguistic choice. |
| Outcome: | The proposed method covers four languages and five linguistic properties, including verb transitivity and use of numerals. |
Leveraging Digitized Newspapers to Collect Summarization Data in Low-Resource Languages (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies suggest that summarization in English may be solved, or even "dead" However, there are no accessible, high-quality summarizing datasets in under-represented languages. |
| Approach: | They propose a method for collecting naturally occurring summaries via front-page teasers, where editors summarize full length articles. |
| Outcome: | The proposed method is suited to varying linguistic resources and is available in seven languages. |
Applying Intrinsic Debiasing on Downstream Tasks: Challenges and Considerations for Machine Translation (2024.emnlp-main)
Copied to clipboard
| Challenge: | In this study, we examine three considerations for intrinsic debiasing in neural machine translation models. |
| Approach: | They propose to measure the extrinsic bias of neural machine translation models by embedding them in a neural embeddable space and using different tokens to debias them. |
| Outcome: | The proposed methods over-rely on gender stereotypes and over-represent them in their models. |
GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation (2022.emnlp-main)
Copied to clipboard
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, Daniel Weld
| Challenge: | Effective human evaluation of text generation tasks remains an important, open area for research. |
| Approach: | They propose a system for running standardized human evaluations across different generation tasks. |
| Outcome: | The proposed system produces standardized human evaluations across tasks . it crowdsources predictions and ranks systems on leaderboards . the proposed system is not reproducible over time and different annotator populations . |
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (N19-1)
Copied to clipboard
| Challenge: | a large body of work has highlighted the brittleness of reading comprehension systems . a crowdsourced, adversarially-created, 55k-question benchmark requires a more comprehensive understanding of paragraphs . |
| Approach: | They propose a reading comprehension benchmark that requires Discrete Reasoning over the content of paragraphs. |
| Outcome: | The proposed benchmarks show that the best systems only achieve 38.4% F1 on the generalized accuracy metric, while human performance is 96%. |
“Covid vaccine is against Covid but Oxford vaccine is made at Oxford!” Semantic Interpretation of Proper Noun Compounds (2022.emnlp-main)
Copied to clipboard
| Challenge: | Proper noun compounds are used in short-form domains but are largely ignored in information-seeking applications. |
| Approach: | They propose to annotate a manually annotated dataset of 22.5K proper noun compounds . they use supervised learning to generate interpretations from the compounds based on target knowledge . |
| Outcome: | The proposed dataset is 60 times larger than prior noun compound datasets and includes non-compositional examples. |
Active Learning for Coreference Resolution using Discrete Annotation (2020.acl-main)
Copied to clipboard
| Challenge: | Exhaustively annotating coreference is expensive as it requires tracking coreference chains across long passages of text. |
| Approach: | They propose a pairwise annotation technique which asks annotators to identify mention antecedents if a presented mention pair is not coreferent. |
| Outcome: | The proposed method is much more efficient when combined with a mention clustering algorithm for selecting which examples to label . future work can use the proposed protocol to develop coreference models for new domains. |
Evaluating and Improving the Coreference Capabilities of Machine Translation Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Currently, end-to-end models learn coreference resolution implicitly by observing aligned sentences in bilingual corpora. |
| Approach: | They develop a method that derives coreference clusters from MT output and evaluates them without requiring annotations in the target language. |
| Outcome: | The proposed model outperforms existing models on three challenging benchmarks. |
Semantics as a Foreign Language (D18-1)
Copied to clipboard
| Challenge: | (2017): Syntactic grammars capture propositions, but graph-based representations aim to capture a wider notion of propositions. |
| Approach: | They propose a neural sequence-to-sequence framework which can recover syntactic linearizations by a sequence-based approach. |
| Outcome: | The proposed framework performs almost on-par with previous state-of-the-art approaches while requiring less parallel training annotations. |
Trust Me, I’m Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on large language model (LLM) hallucinations associated with model uncertainty or inaccurate knowledge. |
| Approach: | They define and investigate a type of hallucination where a model can answer a question correctly but a perturbation causes it to produce a hallucinous response with high certainty. |
| Outcome: | The proposed mitigations outperform existing methods on CHOKE hallucinations . the findings highlight the need to understand their origins and improve mitigation strategies . |
Y’all should read this! Identifying Plurality in Second-Person Personal Pronouns in English Texts (D19-55)
Copied to clipboard
| Challenge: | Various languages, such as Spanish, Hebrew, or French, have different words to distinguish between singular "you" and plural "you". |
| Approach: | They train a model to distinguish between the single/plural ‘you’ in English using in-domain training. |
| Outcome: | The proposed model achieves reasonable accuracy, but there is room for improvement in the domain-transfer scenario. |
Spot the Odd Man Out: Exploring the Associative Power of Lexical Resources (D18-1)
Copied to clipboard
| Challenge: | Existing word embeddings assign only one vector to each word, resulting in word disambiguation on smaller scales. |
| Approach: | They propose a task which aims to test different properties of word representations. |
| Outcome: | The proposed task is intuitive enough to annotate on a large scale while teasing out properties of popular lexical resources. |
Do Zombies Understand? A Choose-Your-Own-Adventure Exploration of Machine Cognition (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in LLMs have sparked a debate on whether they understand text. |
| Approach: | They propose two working definitions for understanding which explicitly acknowledge the question of consciousness and draw connections with a rich literature in philosophy, psychology and neuroscience. |
| Outcome: | The proposed models achieve impressive results on various benchmarks, seeming to generalize to unseen tasks and domains. |
More Documents, Same Length: Isolating the Challenge of Multiple Documents in RAG (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Retrieval-Augmented Generation (RAG) enhances the accuracy of Large Language Models by leveraging relevant external documents during generation. |
| Approach: | They evaluate various language models on custom datasets derived from QA tasks . they keep context length and position of relevant information constant while varying the number of documents . |
| Outcome: | The proposed method improves the accuracy of large language models by leveraging external documents . increasing document count reduces performance by up to 20%, the authors find . |
The State and Fate of Summarization Datasets: A Survey (2025.naacl-long)
Copied to clipboard
| Challenge: | Summarization is the task of shortening a text while preserving the most important information it contains. |
| Approach: | They propose a novel ontology covering sample properties, collection methods and distribution covering sample characteristics, collection method and distribution. |
| Outcome: | The proposed ontology covers sample properties, collection methods and distribution, and can be used to streamline future research into a more coherent body of work. |
Realistic Evaluation Principles for Cross-document Coreference Resolution (2021.starsem-1)
Copied to clipboard
| Challenge: | Using permissive evaluation protocols, cross-document coreference resolution models produce inflated results. |
| Approach: | They propose to decouple evaluation of mention detection from coreference linking . they argue that models should not exploit the synthetic topic structure of the standard ECB+ dataset . |
| Outcome: | The proposed evaluation principles yield lower results than previous lenient evaluation methods. |
Data Efficient Masked Language Modeling for Vision and Language (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is one of the key sub-tasks in vision-language pretraining. |
| Approach: | They propose a masking strategy that masks tokens with a 15% probability for text-only data. |
| Outcome: | The proposed masking strategy outperforms the baseline model on a prompt-based probing task designed to elicit image objects. |