Papers by Jonathan Berant

Value-aware Approximate Attention (2021.emnlp-main)

Challenge:	Generative Adversarial Networks (GANs) do not suffer from the problem of exposure bias.
Approach:	They propose to approximate the distribution of text generated by a GAN and compare it to traditional probability-based LM metrics.
Outcome:	The proposed method performs significantly worse than state-of-the-art LMs on several GAN-based models and can accelerate progress in GAN text generation.

Copied to clipboard

Ankit Gupta, Jonathan Berant

Challenge:	Existing approximations of dot-product attention ignore the value vectors . a value-aware objective outperforms an optimal approximate that ignores values .
Approach:	They propose an approximation of a value-aware objective that substantially outperforms an optimal approximate that ignores values.
Outcome:	The proposed value-aware objective outperforms an optimal approximation that ignores values in the context of language modeling.

Few-Shot Question Answering by Pretraining Span Selection (2021.acl-long)

Copied to clipboard

Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, Omer Levy

Achieving Model Robustness through Discrete Adversarial Training (2021.emnlp-main)

Challenge:	Pretraining models with recurring span selection are effective, but perform poorly in a few-shot setting.
Approach:	They propose recurring span selection scheme that asks model to select correct span in passage with multiple sets of recurring recurrings.
Outcome:	The proposed model achieves 72.7 F1 on multiple benchmarks while maintaining competitive performance in the high-resource setting.

Copied to clipboard

Maor Ivgi, Jonathan Berant

Robustness in NLP

Challenge:	Discrete adversarial attacks are symbolic perturbations to a language input that preserve the output label but lead to predicting error.
Approach:	They propose a discrete adversarial attack based on best-first search and random sampling attacks that are not based upon expensive search procedures.
Outcome:	The proposed attack outperforms offline augmentation and speedups on three datasets.

Large Language Models for Psycholinguistic Plausibility Pretesting (2024.findings-eacl)

Copied to clipboard

Samuel Amouyal, Aya Meltzer-Asscher, Jonathan Berant

Text Segmentation as a Supervised Learning Task (N18-2)

Challenge:	Psycholinguists typically use language models to create controlled materials . plausibility judgments are often based on coarse-grained judgements, but fine-grounded ones do not .
Approach:	They investigate whether Language Models can be used to generate plausibility judgments . they find that plausible judgements from LMs are highly related to human judgements - whereas other LM models are not .
Outcome:	The proposed language models can generate plausibility judgments from human evaluators . the proposed models do not provide satisfactory discriminative power .

Copied to clipboard

Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, Jonathan Berant

Syntactic Text Processing

Challenge:	Existing datasets for text segmentation are small in size and do not represent the natural distribution of text in documents.
Approach:	They propose a large dataset for text segmentation that is automatically extracted and labeled from Wikipedia and develop a model based on this dataset.
Outcome:	The proposed model generalizes well to unseen natural text.

Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets (D19-1)

Copied to clipboard

Mor Geva, Yoav Goldberg, Jonathan Berant

Transformer Feed-Forward Layers Are Key-Value Memories (2021.emnlp-main)

Challenge:	Having only a few workers generate the majority of dataset examples raises concerns about data diversity .
Approach:	They perform a series of experiments to investigate annotator biases in recent NLU datasets . they find that models are able to recognize the most productive annotators .
Outcome:	The results show that models can recognize the most productive annotators and do not generalize well to examples from annotator that did not contribute to the training set.

Copied to clipboard

Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy

When the LM misunderstood the human chuckled: Analyzing garden path effects in humans and language models (2025.acl-long)

Challenge:	Using feed-forward layers, we show that the learned patterns are human-interpretable, and that lower layers tend to capture shallow patterns, while upper layers learn more semantic ones.
Approach:	They propose that feed-forward layers in transformer-based language models operate as key-value memories where each key correlates with textual patterns in the training examples and each value induces a distribution over the output vocabulary.
Outcome:	The proposed model is based on key-value memories with a key-level correlation with the training examples and a distribution over the output vocabulary.

Copied to clipboard

Samuel Joseph Amouyal, Aya Meltzer-Asscher, Jonathan Berant

oLMpics-On What Language Model Pre-training Captures (2020.tacl-1)

Challenge:	Modern Large Language Models (LLMs) have shown human-like abilities in many language tasks, sparking interest in comparing LLMs’ and humans’ language processing.
Approach:	They propose to answer two questions: 1. What makes garden-path sentences hard for humans? 2. Do the same reasons make garden- path sentences hard?
Outcome:	The proposed models show that humans struggle with specific syntactic complexities, with some models showing high correlation with human comprehension.

Copied to clipboard

Alon Talmor, Yanai Elazar, Yoav Goldberg, Jonathan Berant

Don’t paraphrase, detect! Rapid and Effective Data Collection for Semantic Parsing (D19-1)

Challenge:	Recent success of pre-trained language models has spurred widespread interest in their capabilities.
Approach:	They propose an evaluation protocol that includes zero-shot evaluation and no fine-tuning . they propose to compare the learning curve of a fine- tuned LM to the learning of multiple controls .
Outcome:	The proposed evaluation protocol compares the learning curve of a fine-tuned LM to the learning of multiple controls.

Copied to clipboard

Jonathan Herzig, Jonathan Berant

Decoupling Structure and Lexicon for Zero-Shot Semantic Parsing (D18-1)

Challenge:	a major hurdle on the road to conversational interfaces is the difficulty in collecting data that maps language utterances to logical forms . crowdsourcing and crowdsourcing have been used to generate pseudo-language paired with logical form . however, this data collection method often leads to low performance on real data .
Approach:	They propose a method that uses crowdsourcing to map language utterances to logical forms . they quantify the effects of mismatches between the true and induced distributions .
Outcome:	The proposed method leads to 70.6 accuracy on the true distribution, compared to 51.3 in paraphrase-based data collection.

Copied to clipboard

Jonathan Herzig, Jonathan Berant

Crawling The Internal Knowledge-Base of Language Models (2023.findings-eacl)

Challenge:	Existing methods for training semantic parsers in new domains require expensive supervision and lack the ability to generalize to new domain.
Approach:	They propose a zero-shot approach to parsing utterances in unseen domains . they map an utterant to an abstract, domain independent, logical form and replace slots with KB constants based on lexical alignment scores and global inference .
Outcome:	The proposed model achieves 53.4% accuracy on 7 domains in the OVERNIGHT dataset, significantly better than other zero-shot baselines and performs as good as a parser trained on over 30% of the target domain examples.

Copied to clipboard

Roi Cohen, Mor Geva, Jonathan Berant, Amir Globerson

What’s in Your Head? Emergent Behaviour in Multi-Task Transformer Models (2021.emnlp-main)

Challenge:	Existing methods for representing factual knowledge in a language model are insufficient.
Approach:	They propose a procedure for “crawling” the internal knowledge-base of a language model by expanding a knowledge-graph around it.
Outcome:	The proposed method yields high precision graphs (82-92%) while emitting a reasonable number of facts per entity.

Copied to clipboard

Mor Geva, Uri Katz, Aviv Ben-Arie, Jonathan Berant

Dolomites: Domain-Specific Long-Form Methodical Tasks (2025.tacl-1)

Challenge:	Existing paradigms for multi-task training involve a shared pre-trained language model and a small, thin network (head) given an input, a target head is the head that is selected for outputting the final prediction.
Approach:	They examine the behaviour of non-target heads when given input that belongs to a different task than the one they were trained for.
Outcome:	The non-target heads exhibit emergent behaviour, which may explain the target task, or generalize beyond their original task.

Copied to clipboard

Chaitanya Malaviya, Priyanka Agrawal, Kuzman Ganchev, Pranesh Srinivasan, Fantine Huot, Jonathan Berant, Mark Yatskar, Dipanjan Das, Mirella Lapata, Chris Alberti

Challenge:	Experts in various fields perform methodical writing tasks to plan, organize, and report their work.
Approach:	They propose a benchmark with specifications for 519 methodical writing tasks . they use expert revisions of up to 10 model-generated examples to evaluate contemporary language models.
Outcome:	The proposed benchmark includes specifications for 519 methodical writing tasks . it includes examples with input and output examples, and is available at https://dolomites-benchmark.github.io/ .

Break It Down: A Question Understanding Benchmark (2020.tacl-1)

Copied to clipboard

Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, Jonathan Berant

Injecting Numerical Reasoning Skills into Language Models (2020.acl-main)

Challenge:	Understanding natural language questions entails the ability to break down a question into the requisite steps for computing its answer.
Approach:	They introduce a Question Decomposition Meaning Representation (QDMR) for questions . they demonstrate that QDMRs can be annotated at scale using a hotpotQA dataset .
Outcome:	The proposed model outperforms several natural baselines in the open-domain question answering hotpotQA dataset and can be deterministically converted to a pseudo-SQL formal language.

Copied to clipboard

Mor Geva, Ankit Gupta, Jonathan Berant

Value-based Search in Execution Space for Mapping Instructions to Programs (N19-1)

Challenge:	Existing models for numerical reasoning are limited by their flexibility and require specialized architectures to capture high-level skills.
Approach:	They propose to inject numerical reasoning skills into pre-trained LMs by generating large amounts of data and training in a multi-task setup.
Outcome:	The proposed model performs better on DROP than other models of comparable size while maintaining high performance on standard RC tasks.

Copied to clipboard

Dor Muhlgay, Jonathan Herzig, Jonathan Berant

Programming Languages in NLP

Challenge:	Existing methods to map instructions to programs require searching for good programs at training time.
Approach:	They propose a search algorithm that uses the target world state to train a critic network that predicts the expected reward of every search state.
Outcome:	The proposed algorithm significantly improves on all three domains compared to baselines on the SCONE dataset.

What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary (2023.acl-long)

Copied to clipboard

Ori Ram, Liat Bezalel, Adi Zicher, Yonatan Belinkov, Jonathan Berant, Amir Globerson

Answering Questions by Meta-Reasoning over Multiple Chains of Thought (2023.emnlp-main)

Challenge:	Dense retrieval models based on text representations have proven very effective, but when applied off-the-shelf they often experience a severe drop in performance.
Approach:	They propose to interpret the vector representations produced by dual encoders by projecting them into the model’s vocabulary space.
Outcome:	The proposed model significantly improves on the BEIR benchmark and in zero-shot settings.

Copied to clipboard

Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, Jonathan Berant

Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments (2022.findings-emnlp)

Challenge:	Modern systems for multi-hop question answering (QA) break questions into a sequence of reasoning steps, termed chain-of-thought (CoT) Often, multiple chains are sampled and aggregated, but the intermediate steps themselves are discarded.
Approach:	They propose a method which prompts large language models to meta-reason over multiple chains of thought rather than aggregate their answers.
Outcome:	The proposed approach outperforms baselines on 7 multi-hop QA datasets.

Copied to clipboard

Maor Ivgi, Yair Carmon, Jonathan Berant

Learning to Search in Long Documents Using Document Structure (C18-1)

Challenge:	Neural scaling laws define a predictable relationship between a model’s parameter count and its performance after training in the form of a power law.
Approach:	They perform an empirical investigation of language understanding tasks and evaluate their results to determine whether scaling laws can be used to accelerate model development.
Outcome:	The proposed scaling laws can be exploited for debugging convergence when training large models, and can predict the performance of larger models.

Copied to clipboard

Mor Geva, Jonathan Berant

White-to-Black: Efficient Distillation of Black-Box Adversarial Attacks (N19-1)

Challenge:	Reading comprehension models are dominated by recurrent neural networks (RNNs) as documents become longer and questions become complex, sequential reading becomes a significant bottleneck.
Approach:	They propose a reading comprehension framework that uses document trees to model an agent that interleaves quick navigation with more expensive answer extraction.
Outcome:	The proposed model improves question answering performance compared to existing models and has a strong information-retrieval baseline.

Copied to clipboard

Yotam Gil, Yoav Chai, Or Gorodissky, Jonathan Berant

Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing (P19-1)

Challenge:	Recent work in natural language processing generates adversarial examples using white-box access . a neural network can learn to emulate the behavior of a white- box attack and generalize well to new examples.
Approach:	They propose an adversarial training approach that assumes white-box access to an attacker's model and optimizes the input directly against it.
Outcome:	The proposed approach reduces example generation time by 19x-39x and exposes the Google Perspective API vulnerability.

Copied to clipboard

Ben Bogin, Jonathan Berant, Matt Gardner

Multimodality

Challenge:	Semantic parsing to SQL has largely ignored the structure of the database schema . a recent study used a simple DB that was observed at both training and test time.
Approach:	They propose a semantic parser where the schema structure is encoded with a graph neural network and used at both encoding and decoding time.
Outcome:	The proposed parser improves from 33.8% to 39.4%, dramatically above the current state of the art, which is at 19.7%.

Weakly Supervised Semantic Parsing with Abstract Examples (P18-1)

Copied to clipboard

Omer Goldman, Veronica Latcinnik, Ehud Nave, Amir Globerson, Jonathan Berant

Efficient Long-Text Understanding with Short-Text Models (2023.tacl-1)

Challenge:	training semantic parsers from weak supervision complicates training in two ways . spurious programs that accidentally lead to a correct denotation add noise to training .
Approach:	They propose to use tokens in both language utterance and program to map denotations to executable programs.
Outcome:	The proposed method improves performance and reaches 82.5% accuracy compared to the best reported accuracy so far.

Copied to clipboard

Maor Ivgi, Uri Shaham, Jonathan Berant

AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks? (2024.emnlp-main)

Challenge:	Existing transformer-based pretrained language models cannot be applied to long sequences due to their quadratic complexity.
Approach:	They propose a simple approach to long sequences that re-uses battle-tested short-text pretrained LMs.
Outcome:	The proposed approach is competitive with specialized models that are up to 50x larger and require a dedicated and expensive pretraining step.

Copied to clipboard

Ori Yoran, Samuel Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, Jonathan Berant

Unobserved Local Structures Make Compositional Generalization Hard (2022.emnlp-main)

Challenge:	Current language models and retrieval-augmented LMs are limited in their ability to perform tasks on the web.
Approach:	They propose a benchmark to evaluate language agents built on top of language models . they propose 'AssistantBench' which includes 214 tasks that can be automatically evaluated .
Outcome:	The proposed agent outperforms existing agents in a new benchmark for language agents on the web.

Copied to clipboard

Ben Bogin, Shivanshu Gupta, Jonathan Berant

Inferring Implicit Relations in Complex Questions with Language Models (2022.findings-emnlp)

Challenge:	Recent studies show sequence-to-sequence models struggle to generalize to new compositions . little is known on what makes generalization hard on a particular test instance .
Approach:	They propose a criterion for the difficulty of an example that is hard if it contains a local structure that was not observed at training time.
Outcome:	The proposed rule predicts instance-level generalization well across 5 different datasets.

Copied to clipboard

Uri Katz, Mor Geva, Jonathan Berant

SmBoP: Semi-autoregressive Bottom-up Semantic Parsing (2021.naacl-main)

Challenge:	A prominent challenge for language understanding systems is the ability to answer implicit reasoning questions where the evidence for answering the question is not mentioned explicitly.
Approach:	They propose to decouple inference of reasoning steps from execution by evaluating models of implicit relation inference.
Outcome:	The proposed model fails on the implicit reasoning QA task, but infers implicit relations . the proposed model is compared with other models that fail on the same task .

Copied to clipboard

Ohad Rubin, Jonathan Berant

A Simple and Effective Model for Answering Multi-span Questions (2020.emnlp-main)

Challenge:	Existing semantic parsers decode syntax using a top-down depth-first traversal.
Approach:	They propose a semi-autoregressive bottom-up parser that constructs at decoding step t the top-K sub-trees of height t.
Outcome:	The proposed method achieves 2.2x speed-up in decoding time and 5x speed up in training time on a zero-shot semantic parsing benchmark.

Copied to clipboard

Elad Segal, Avia Efrat, Mor Shoham, Amir Globerson, Jonathan Berant

Natural Language Interfaces

Challenge:	Existing models for reading comprehension restrict output space to a set of single contiguous spans . multi-span questions are problematic because they require multiple inputs - a task that requires a sequence tagging problem .
Approach:	They propose a simple architecture for answering multi-span questions by casting the task as a sequence tagging problem.
Outcome:	The proposed model significantly improves performance on span extraction questions from DROP and Quoref by 9.9 and 5.5 EM points respectively.

DiscoFuse: A Large-Scale Dataset for Discourse-Based Sentence Fusion (N19-1)

Copied to clipboard

Mor Geva, Eric Malmi, Idan Szpektor, Jonathan Berant

Challenge:	Existing datasets for sentence fusion are small and insufficient for training modern neural models.
Approach:	They propose a method for automatically-generating fusion examples from raw text . they apply their method to Wikipedia and Sports articles to generate fusion models .
Outcome:	The proposed method improves performance on WebSplit when viewed as a sentence fusion task.

In-Context Learning with Long-Context Models: An In-Depth Exploration (2025.naacl-long)

Copied to clipboard

Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R. Gormley, Graham Neubig

Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2022.acl-long)

Challenge:	In-context learning is limited by context length, but it can be used for many tasks.
Approach:	They study the behavior of in-context learning at an extreme context length . example retrieval shows excellent performance at low context lengths but has diminished gains .
Outcome:	The proposed model can perform many tasks with reasonable accuracy when a few examples are provided in-context.

Copied to clipboard

Ori Yoran, Alon Talmor, Jonathan Berant

Multimodality

Challenge:	Large pre-trained language models struggle in tasks that require reasoning . recent work shows that they struggle in performing symbolic reasoning operations without substantial amounts of additional data.
Approach:	They propose to leverage semi-structured tables and generate at scale question-paragraph pairs where answering the question requires reasoning over multiple facts in the paragraph.
Outcome:	The proposed model outperforms T5, a popular pre-trained encoder-decoder model, on reasoning-focused reading comprehension datasets.

SCROLLS: Standardized CompaRison Over Long Language Sequences (2022.emnlp-main)

Copied to clipboard

Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, Omer Levy

Challenge:	Standard NLP benchmarks focus on short texts, but long texts are produced in the context of longer discourses.
Approach:	They propose a new benchmark that places models in context of long texts that require reasoning over long texts.
Outcome:	The proposed task sets are based on a set of long-text datasets and host a live leaderboard to facilitate research on model architecture and pretraining methods.

Global Reasoning over Database Structures for Text-to-SQL Parsing (D19-1)

Copied to clipboard

Ben Bogin, Matt Gardner, Jonathan Berant

Low-Resource NLP

Challenge:	Existing semantic parsers only select a set of database constants at training time . current models only consider local information, not global ones .
Approach:	They propose a semantic parser that globally reasons about the structure of the query to make a more contextually-informed selection of database constants.
Outcome:	The proposed model increases accuracy from 39.4% to 47.4% on a zero-shot semantic parsing dataset with complex databases.

Improving Compositional Generalization in Semantic Parsing (2020.findings-emnlp)

Copied to clipboard

Inbar Oren, Jonathan Herzig, Nitish Gupta, Matt Gardner, Jonathan Berant

Polyglot Semantic Parsing in APIs (N18-1)

Challenge:	Generalization of models to out-of-distribution data has sparked substantial interest . compositional generalization is the ability to systematically generalize to test examples composed of components seen during training .
Approach:	They propose to extend compositional generalization in semantic parsing by using contextual representations and training attention to agree with pre-computed token alignments.
Outcome:	The proposed extensions improve compositional generalization on OOD compositions.

Copied to clipboard

Kyle Richardson, Jonathan Berant, Jonas Kuhn

Multilinguality

Challenge:	Existing approaches to semantic parsing work by training individual models for each available parallel dataset of text-meaning pairs.
Approach:	They propose a polyglot semantic translation approach that trains on multiple datasets and natural languages to learn parsing models.
Outcome:	The proposed model can be used for parsing a wide variety of natural languages and output languages, and achieves state-of-the-art performance on the above datasets.

MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (P19-1)

Copied to clipboard

Alon Talmor, Jonathan Berant

ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding (2023.findings-emnlp)

Challenge:	A large number of reading comprehension (RC) datasets have been created, but little research has been done on whether they generalize to one another and the extent to which existing datasets can be leveraged for improving performance on new ones.
Approach:	They propose a BERT-based reading comprehension model that can be trained on multiple RC datasets.
Outcome:	The proposed model can be trained on multiple RC datasets and improve performance on five RC data.

Copied to clipboard

Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, Omer Levy

Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020.findings-emnlp)

Challenge:	Existing benchmarks for long text understanding focus on short sequences, such as BigBench and HELM.
Approach:	They propose a zero-shot benchmark for natural language understanding over long texts . they adapt six tasks from the SCROLLS benchmark and add four new datasets .
Outcome:	The proposed benchmark outperforms ChatGPT and GPT-4 in a number of open tasks.

Copied to clipboard

Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou

Challenge:	Standard test sets for supervised learning evaluate in-distribution generalization but are misleading when a dataset has systematic gaps.
Approach:	They propose a more rigorous annotation paradigm for NLP that helps to close systematic gaps in the test data.
Outcome:	The proposed model performs significantly lower on contrast sets than on the original test sets—up to 25% in some cases.

COVR: A Test-Bed for Visually Grounded Compositional Generalization with Real Images (2021.emnlp-main)

Copied to clipboard

Ben Bogin, Shivanshu Gupta, Matt Gardner, Jonathan Berant

Visual Data in NLP

Challenge:	Existing benchmarks for visual-grounded models have focused on synthetic images . et al., 2018: compositional generalization is crucial for building models that generalize to new settings.
Approach:	They propose a test-bed for visually-grounded compositional generalization with real images.
Outcome:	The proposed test-bed enables compositional splits where models need to generalize to new concepts and compositions in a zero- or few-shot setting.

SEMQA: Semi-Extractive Multi-Source Question Answering (2024.naacl-long)

Copied to clipboard

Tal Schuster, Adam Lelkes, Haitian Sun, Jai Gupta, Jonathan Berant, William Cohen, Donald Metzler

Natural Language Interfaces

Challenge:	Recent proposed long-form question answering systems have shown promising capabilities, but attributing and verifying their generated abstractive answers can be difficult.
Approach:	They propose a task that summarises multiple sources in a semi-extractive fashion . they create a dataset with human-written semi-extractive answers to natural and generated questions .
Outcome:	The proposed task summarizes multiple sources in a semi-extractive fashion and produces fine in-line attributions by-design that are easy to verify, interpret, and evaluate.

Learning to Retrieve Passages without Supervision (2022.naacl-main)

Copied to clipboard

Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, Amir Globerson

Information Retrieval

Challenge:	Dense retrievers for open domain question answering have been shown to achieve impressive performance by training on large datasets of question-passage pairs.
Approach:	They propose to use recurring spans to create pseudo examples for contrastive learning.
Outcome:	The proposed model outperforms all pretrained baselines on a wide range of ODQA datasets and is competitive with BM25, a strong sparse baseline.

On Making Reading Comprehension More Comprehensive (D19-58)

Copied to clipboard

Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, Sewon Min

Comparing human and language models sentence processing difficulties on complex structures (2026.acl-long)

Challenge:	Getting machines to "understand" text is a vast and long-standing problem, made more challenging by the fact that it is not even clear what it means to understand text.
Approach:	They propose a question-based approach to machine reading comprehension that uses a natural language question to test a system's comprehension of a passage of text.
Outcome:	The proposed questions have surface cues or other biases that allow a model to shortcut the intended reasoning process.

Copied to clipboard

Samuel Joseph Amouyal, Aya Meltzer-Asscher, Jonathan Berant

Span-based Semantic Parsing for Compositional Generalization (2021.acl-long)

Challenge:	Large language models (LLMs) that converse with humans are a reality, but do LLMs experience human-like processing difficulties?
Approach:	They systematically compare human and LLM sentence comprehension across seven challenging linguistic structures.
Outcome:	The proposed model achieves near perfect accuracy on non-GP structures, but struggles on GP structures.

Copied to clipboard

Jonathan Herzig, Jonathan Berant

Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition (2022.tacl-1)

Challenge:	despite success of sequence-to-sequence models, they fail in compositional generalization . a span-based parser that predicts a utterance over spans improves performance .
Approach:	They propose a span-based parser that predicts a utterance over a given span tree . they propose to use CKY to encode how partial programs compose over spans .
Outcome:	The proposed model performs better on random splits than baselines that require compositional generalization.

Copied to clipboard

Mor Geva, Tomer Wolfson, Jonathan Berant

Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering (2021.tacl-1)

Challenge:	Recent efforts to create challenge benchmarks that test the abilities of natural language understanding models have largely depended on human annotations.
Approach:	They propose a framework for automatic reasoning-oriented perturbation of question-answer pairs that decomposes a question into reasoning steps required to answer it and generates new question-anchor pairs.
Outcome:	The proposed framework generates evaluation sets for reading comprehension benchmarks and generates examples without human intervention.

Copied to clipboard

Ben Bogin, Sanjay Subramanian, Matt Gardner, Jonathan Berant

Contextualized Word Representations for Reading Comprehension (N18-2)

Challenge:	Neural networks fail to generalize to out-of-distribution examples that contain new compositions.
Approach:	They propose a model that computes a representation and denotation for all question spans in a bottom-up, compositional manner using a CKY-style parser.
Outcome:	The proposed model achieves 96.1% accuracy on a challenging dataset compared to baseline models on . previous models failed to generalize to out-of-distribution examples .

Copied to clipboard

Shimi Salant, Jonathan Berant

Representation Learning

Challenge:	Reading comprehension (RC) is a high-level task in natural language understanding that requires reading a document and answering questions about its content.
Approach:	They propose to provide a standard neural network for reading a document and answering a question about its content.
Outcome:	The proposed model improves on the competitive SQuAD dataset by providing rich contextualized word representations and allowing it to choose between context-dependent and context-independent representations.

The Web as a Knowledge-Base for Answering Complex Questions (N18-1)

Copied to clipboard

Alon Talmor, Jonathan Berant

Obtaining Faithful Interpretations from Compositional Neural Networks (2020.acl-main)

Challenge:	Recent work on reading comprehension made headway in answering simple questions, but tackling complex questions is still an ongoing research challenge.
Approach:	They propose to decompose complex questions into a sequence of simple questions and compute the final answer from the sequence of answers.
Outcome:	The proposed framework improves performance from 20.8 precision@1 to 27.5 precision@1.

Copied to clipboard

Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, Matt Gardner

Analyzing Transformers in Embedding Space (2023.acl-long)

Challenge:	Neural module networks (NMNs) are a popular approach for modeling compositionality but prior work implicitly assumed that the structure of the network modules provides a faithful explanation of the model’s reasoning.
Approach:	They propose to use auxiliary supervision to train a model with a structured model that can understand the reasoning process and make better choices for module architecture.
Outcome:	The proposed models on two datasets show that the proposed models do not provide a faithful explanation of model behaviour.

Copied to clipboard

Guy Dar, Mor Geva, Ankit Gupta, Jonathan Berant

Evaluating the Evaluation of Diversity in Natural Language Generation (2021.eacl-main)

Challenge:	Understanding Transformer-based models has attracted significant attention . a zero-pass approach is feasible for some parameters, and for two-layer attention networks .
Approach:	They propose a theoretical framework where parameters of a trained Transformer are interpreted by projecting them into the embedding space.
Outcome:	The proposed framework shows that pre-trained and fine-tuned models can be interpreted in embedding space.

Copied to clipboard

Guy Tevet, Jonathan Berant

Text Generation

Challenge:	Existing methods for controlling diversity by tuning a “decoding parameter” affect form but not meaning.
Approach:	They propose a framework that measures correlation between a diversity metric and a parameter that controls some aspect of diversity in generated text.
Outcome:	The proposed framework outperforms existing methods in estimating diversity . it shows that humans outperformed existing methods but affect form but not meaning .

Learning To Retrieve Prompts for In-Context Learning (2022.naacl-main)

Copied to clipboard

Ohad Rubin, Jonathan Herzig, Jonathan Berant

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (N19-1)

Challenge:	In-context learning is a new paradigm in natural language understanding . large pre-trained language models can be expensive to update .
Approach:	They propose an efficient method for retrieving training examples as prompts from annotated data and an LM.
Outcome:	The proposed method outperforms prior work and multiple baselines on three sequence-to-sequence tasks.

Copied to clipboard

Alon Talmor, Jonathan Herzig, Nicholas Lourie, Jonathan Berant

Diverse Demonstrations Improve In-context Compositional Generalization (2023.acl-long)

Challenge:	Recent work on question answering relies on factoid questions with little general knowledge.
Approach:	They propose a dataset to capture commonsense question answering with prior knowledge . they extract multiple-choice questions that discriminate between the source and target concepts .
Outcome:	The proposed dataset captures commonsense reasoning beyond associations . it obtains 56% accuracy, well below human performance, which is 89% .

Copied to clipboard

Itay Levy, Ben Bogin, Jonathan Berant

BERTese: Learning to Speak to BERT (2021.eacl-main)

Challenge:	In-context learning has shown great success in i.i.d semantic parsing splits . however, in compositional generalization, selecting similar demonstrations is insufficient .
Approach:	They propose a method to select diverse demonstrations that collectively cover all the structures required in the output program and encourage the model to generalize to new structures from these demonstrations.
Outcome:	The proposed method improves performance across three compositional generalization datasets and finetuning.

Copied to clipboard

Adi Haviv, Jonathan Berant, Amir Globerson