Papers by Hannaneh Hajishirzi
Multi-hop Reading Comprehension through Question Decomposition and Rescoring (P19-1)
Copied to clipboard
| Challenge: | Existing systems for multi-hop reading comprehension decompose compositional questions into simpler sub-questions . authors propose a system that learns to break compositional multi- hop questions into simple singlehop sub-question . |
| Approach: | They propose a system that decomposes a compositional question into simpler sub-questions . they propose recast subquestion generation as a span prediction problem . |
| Outcome: | The proposed system generates as effective as human-authored sub-questions using 400 examples . it also provides explainable evidence for its decision making in the form of sub-questions . |
An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to condition models on a concise rationale are less accurate than models that can use the entire context. |
| Approach: | They propose a method to optimize a bound on the Information Bottleneck objective to extract concise rationales from a binary mask and an end-task predictor that uses only the residual sentences. |
| Outcome: | The proposed model outperforms existing norm-minimization techniques in task performance and agreement with human rationales in the ERASER benchmark. |
FaVIQ: FAct Verification from Information-seeking Questions (2022.acl-long)
Copied to clipboard
| Challenge: | Existing fact verification datasets with crowdsourced claims introduce subtle biases that are difficult to control for. |
| Approach: | They construct a large-scale fact verification dataset with ambiguous questions . they use a corpus of 188k claims to construct false and true claims . |
| Outcome: | The proposed dataset outperforms models trained on the dataset FEVER or in-domain data by up to 17% absolute. |
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models struggle with tasks requiring rich world knowledge, implying the difficulty of encoding a wealth of world knowledge in their parameters. |
| Approach: | They propose a retrieval-augmentation method that improves performance and reduces inference costs by only retrieving non-parametric memories when necessary. |
| Outcome: | The proposed method improves performance and reduces inference costs by only retrieving non-parametric memories when necessary. |
X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has adapted vision-and-language models to generative tasks like image captioning. |
| Approach: | They propose an extension to LXMERT with training refinements to generate images from text. |
| Outcome: | The proposed model can generate images from pieces of text while still being comparable to existing models. |
InSCIt: Information-Seeking Conversations with Mixed-Initiative Interactions (2023.tacl-1)
Copied to clipboard
Zeqiu Wu, Ryu Parish, Hao Cheng, Sewon Min, Prithviraj Ammanabrolu, Mari Ostendorf, Hannaneh Hajishirzi
| Challenge: | In information-seeking conversations, a user may ask questions that are under-specified or unanswerable. |
| Approach: | They present a dataset for information-seeking conversations with mixed-initiative interactions . they use Wikipedia to search for answers and provide relevant information . |
| Outcome: | The proposed system significantly underperforms humans in two of the most recent studies. |
CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning (2022.emnlp-main)
Copied to clipboard
Zeqiu Wu, Yi Luan, Hannah Rashkin, David Reitter, Hannaneh Hajishirzi, Mari Ostendorf, Gaurav Singh Tomar
| Challenge: | Existing models for conversational question answering require specific retrievers to understand user questions. |
| Approach: | They develop a query rewriting model CONQRR that rewrites a conversational question into a standalone question. |
| Outcome: | The proposed model achieves state-of-the-art on an open-domain conversational question answering dataset and is effective for two different off-the shelf retrievers. |
UNIFIEDQA: Crossing Format Boundaries with a Single QA System (2020.findings-emnlp)
Copied to clipboard
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, Hannaneh Hajishirzi
| Challenge: | Question answering (QA) tasks have been posed using a variety of formats . a new study aims to develop specialized QA models that can be used to train QA systems . |
| Approach: | They build a pre-trained question answering model that performs well across 19 QA datasets . they argue that format-specialized models can limit the ability to teach reasoning . |
| Outcome: | a new model that trains on QA datasets performs on par with 8 models trained on individual datasets . a single model that trained on UNIFIEDQA performs well on 19 QA data . |
MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms (N19-1)
Copied to clipboard
| Challenge: | Existing datasets in this domain do not offer precise operational annotations over diverse problem types due to noise and lack of formal operation-based representations. |
| Approach: | They propose a representation language to map problems to their operation programs . they also introduce an interpretable neural math problem solver . |
| Outcome: | The proposed model outperforms baseline models and the AQUA-RAT dataset on the AQuA-rat dataset. |
Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index (P19-1)
Copied to clipboard
| Challenge: | Existing open-domain question answering models require multiple documents on-demand for every input query. |
| Approach: | They propose query-agnostic indexable representations of document phrases that can drastically speed up open-domain question answering. |
| Outcome: | The proposed model can be trained and deployed even in a single 4-GPU server. |
Data-Efficient Finetuning Using Cross-Task Nearest Neighbors (2023.findings-acl)
Copied to clipboard
| Challenge: | Prior work shows training models on multitask data augmented with task descriptions transfers knowledge to new tasks. |
| Approach: | They propose to use unlabeled target-task data to train models on task descriptions . they use only 2% of the data from the P3 pool without labeled target task data . |
| Outcome: | The proposed model outperforms baseline models on 12 out of 14 datasets . it also provides better initialization than single model on target-task data . |
IIRC: A Dataset of Incomplete Information Reading Comprehension Questions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing reading comprehension tasks focus on questions for which the contexts provide all the information required to answer them, thus not evaluating a system’s performance at identifying a potential lack of sufficient information and locating sources for that information. |
| Approach: | They propose to use a dataset with 13K questions over paragraphs from English Wikipedia that provide only partial information to answer them, with the missing information occurring in one or more linked documents. |
| Outcome: | The proposed model achieves 31.1% F1 on the reading comprehension task, while estimated human performance is 88.4%. |
CREPE: Open-Domain Question Answering with False Presuppositions (2023.acl-long)
Copied to clipboard
| Challenge: | Existing question answering datasets assume all questions have well defined answers. |
| Approach: | They propose a QA dataset containing a distribution of false presuppositions . they find that 25% of questions contain false presumptions . |
| Outcome: | The proposed model finds that 25% of questions contain false presuppositions . the model can find presuffpositions moderately well, but struggle when predicting correctness . |
Phrase-Indexed Question Answering: A New Challenge for Scalable Document Comprehension (D18-1)
Copied to clipboard
| Challenge: | Existing QA models rely on learning interaction between document and question . current models require explicit attention to the document before or as it reads it . |
| Approach: | They propose a modular question answering task that enforces complete independence of the document encoder from the question encoder. |
| Outcome: | The proposed model achieves reasonable accuracy but significantly underperforms unconstrained QA models. |
Elaboration-Generating Commonsense Question Answering at Scale (2023.acl-long)
Copied to clipboard
| Challenge: | elaborations are generated using language models that generate background knowledge that helps improve performance . human evaluations show that the quality of the generated ellaborations is high . |
| Approach: | They propose to finetune smaller language models to generate useful intermediate context . they compare a language model with an answer predictor and generate elaborations . human evaluations show that the quality of the generated ellaborations is high . |
| Outcome: | The proposed framework outperforms other models on commonsense questions on four commons sense benchmarks. |
Probing Contextual Language Models for Common Ground with Visual Representations (2021.naacl-main)
Copied to clipboard
| Challenge: | Contextual language models have attracted great interest in probing what is encoded in their representations. |
| Approach: | They propose a probing model that evaluates how effective are text-only representations in distinguishing between matching and non-matching visual representations. |
| Outcome: | The proposed model outperforms text-only language models in instance retrieval, but underperform humans. |
PuMer: Pruning and Merging Tokens for Efficient Vision Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Large-scale vision language models use Transformers to perform cross-modal interactions . state-of-the-art models are memory intensive and expensive due to quadratic complexity . |
| Approach: | They propose a token reduction framework that uses text-informed Pruning and modality-aware Merging strategies to progressively reduce the tokens of input image and text. |
| Outcome: | The proposed framework improves inference speed and memory footprint on four vision language tasks. |
Probing Across Time: What Does RoBERTa Know and When? (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Current approaches to natural language processing rely on fixed artifacts such as language models . current studies have focused on how these models acquire and demonstrate knowledge . |
| Approach: | They apply probing techniques to examine how language models acquire knowledge . they aim to inform future work on more efficient pretraining and understanding dependencies . |
| Outcome: | The proposed model learns linguistic abstractions, factual and commonsense knowledge, and reasoning abilities fast, stably, and robustly across domains. |
HINT: Hypernetwork Instruction Tuning for Efficient Zero- and Few-Shot Generalisation (2023.acl-long)
Copied to clipboard
| Challenge: | Recent NLP models have shown the remarkable ability to generalise ‘zero-shot’ to new tasks using only natural language instructions as guidance. |
| Approach: | They introduce Hypernetworks for INstruction Tuning (HINT) which converts task instructions and examples into parameter-efficient modules inserted into an underlying model using a pretrained text encoder. |
| Outcome: | The proposed models outperform strong state-of-the-art models by over 10% when controlling for compute. |
A general framework for information extraction using dynamic span graphs (N19-1)
Copied to clipboard
| Challenge: | Existing frameworks for information extraction use a pipeline approach to identify entities and then use the detected entity spans for relation extraction and coreference resolution. |
| Approach: | They propose a framework for several information extraction tasks that share span representations using dynamically constructed span graphs. |
| Outcome: | The proposed framework significantly outperforms state-of-the-art on multiple information extraction tasks across multiple datasets reflecting different domains. |
OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens (2025.acl-demo)
Copied to clipboard
Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, Yen-Sung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Yu-Yen Cheng, Karen Farley, Taira Anderson, David Albright, Carissa Schoenick, Luca Soldaini, Dirk Groeneveld, Rock Yuren Pang, Pang Wei Koh, Noah A. Smith, Sophie Lebrecht, Yejin Choi, Hannaneh Hajishirzi, Ali Farhadi, Jesse Dodge
| Challenge: | tracing language models' outputs back to training data is a problem because they are trained on text corpora with trillions of tokens . existing methods for tracers have not been scaled to work within this multi-trillion-token setting . |
| Approach: | They propose a system that traces language models' outputs verbatim back to training data . OLMOTRACE retrieves documents from the model's training data that contain exact matches . |
| Outcome: | The proposed system can find verbatim matches between LM output and training data . it can be used to explore fact checking, hallucination, and creativity of language models . |
Nonparametric Masked Language Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. |
| Approach: | They introduce a nonparametric masked language model that replaces a softmax with a distribution over every phrase in a reference corpus and uses an in-batch approximation to train it. |
| Outcome: | The proposed model outperforms larger parametric models on 16 tasks including classification, fact probing and question answering. |
Text Generation from Knowledge Graphs with Graph Transformers (N19-1)
Copied to clipboard
| Challenge: | Existing methods for generating text with structured inputs are expensive and require manual annotation. |
| Approach: | They propose a graph transforming encoder which leverages relational structure of knowledge graphs without imposing linearization or hierarchical constraints. |
| Outcome: | The proposed system produces more informative texts than competing methods. |
Prompting Contrastive Explanations for Commonsense Reasoning Tasks (2021.findings-acl)
Copied to clipboard
| Challenge: | Large pretrained language models (PLMs) can achieve near-human performance on commonsense reasoning tasks, but provide little human-interpretable evidence of the underlying reasoning they use. |
| Approach: | They propose to use large pretrained language models to generate evidence for commonsense reasoning NLP tasks . they use models to contrast alternative explanations based on key attribute(s) required to justify the correct answer . |
| Outcome: | The proposed model improves performance on two commonsense reasoning benchmarks compared to previous non-contrastive alternatives. |
Aligning to Social Norms and Values in Interactive Narratives (2022.naacl-main)
Copied to clipboard
| Challenge: | Social value alignment is the ability to create agents that act in alignment with socially beneficial norms and values in interactive narratives or text-based games. |
| Approach: | They introduce a game-value ALignment agent that uses social commonsense to restrict its action space to actions that are aligned with socially beneficial values. |
| Outcome: | The proposed agent improves state-of-the-art task performance by 4% while reducing the frequency of socially harmful behaviors by 25% compared to strong contemporary value alignment approaches. |
MetaICL: Learning to Learn In Context (2022.naacl-main)
Copied to clipboard
| Challenge: | Large language models can do in-context learning by conditioning on a few training examples with no parameter updates or task-specific templates. |
| Approach: | They propose a meta-training framework where a pretrained language model is tuned to do in-context learning on a large set of training tasks. |
| Outcome: | The proposed framework outperforms baseline models on 142 NLP datasets and a range of target tasks with domain shifts. |
Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for zero-shot learning are based on in-context training, but performance drops when no demonstrations are available. |
| Approach: | They propose a new method that constructs pseudo-demonstrations for a given test input using a raw text corpus and applies techniques to reduce copying. |
| Outcome: | The proposed method outperforms previous zero-shot methods on nine classification datasets and is on par with in-context learning with labeled training data in the few-shot setting. |
Standardized Tests as benchmarks for Artificial Intelligence (D18-3)
Copied to clipboard
| Challenge: | Standardized tests have been proposed as replacements to the Turing test as a driver for progress in AI. |
| Approach: | et al. propose standardized tests as replacements to the Turing test as a driver for progress in AI. |
| Outcome: | a series of standardized tests have been proposed as replacements to the Turing test . the tutorial categorizes open domain and closed domain tests into two categories . open domain tests require the system to have significant domain knowledge and reasoning capabilities. |
Steering off Course: Reliability Challenges in Steering Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Prior studies have evaluated a few steering methods for language models, leaving gaps in understanding their robustness. |
| Approach: | They examine three steering methods for language models to examine their reliability . they use function vectors, task vectors and DoLa to steer models toward desirable outputs . |
| Outcome: | The proposed methods show that they are not robust enough to handle large models with large parameters. |
SciREX: A Challenge Dataset for Document-Level Information Extraction (2020.acl-main)
Copied to clipboard
| Challenge: | Conventional datasets and methods for information extraction focus on within-sentence relations from general Newswire text. |
| Approach: | They propose a document-level IE dataset that integrates automatic and human annotations to annotate entities and document- level N-ary relation identification from scientific articles. |
| Outcome: | The proposed dataset extends state-of-the-art IE models to document-level IE. |
Exploring The Landscape of Distributional Robustness for Question Answering Models (2022.findings-emnlp)
Copied to clipboard
Anas Awadalla, Mitchell Wortsman, Gabriel Ilharco, Sewon Min, Ian Magnusson, Hannaneh Hajishirzi, Ludwig Schmidt
| Challenge: | Existing methods for predicting distributional robustness fail to generalize reliably in a variety of test conditions. |
| Approach: | They conduct a large empirical evaluation to investigate the landscape of distributional robustness in question answering. |
| Outcome: | The proposed methods are more robust to distribution shifts than fully fine-tuned models, and few-shot prompt models exhibit better robustness than few- shot prompt models. |
Iconary: A Pictionary-Based Game for Testing Multimodal Communication with Drawings and Text (2021.emnlp-main)
Copied to clipboard
Christopher Clark, Jordi Salvador, Dustin Schwenk, Derrick Bonafilia, Mark Yatskar, Eric Kolve, Alvaro Herrasti, Jonghyun Choi, Sachin Mehta, Sam Skjonsberg, Carissa Schoenick, Aaron Sarnat, Hannaneh Hajishirzi, Aniruddha Kembhavi, Oren Etzioni, Ali Farhadi
| Challenge: | Communicating with humans is challenging for AIs because of its complexity and multimodality. |
| Approach: | They propose to use a game of drawing and guessing based on Pictionary to test AIs' understanding of the world and multi-modal gestures. |
| Outcome: | The proposed game is a test for mixing language and visual/symbolic communication in AI. |
Self-Instruct: Aligning Language Models with Self-Generated Instructions (2023.acl-long)
Copied to clipboard
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi
| Challenge: | Large “instruction-tuned” language models depend heavily on human-written instruction data . this limited quantity, diversity, and creativity hinders the generality of the tuned model . |
| Approach: | They propose a framework for improving instruction-following capabilities of pretrained language models by bootstrapping off their own generations. |
| Outcome: | The proposed framework outperforms existing public instruction datasets by 5% . it generates instructions, input, and output samples, then filters invalid or similar ones . |
Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern language models are trained on text data downsampled from massive text corpora like Common Crawl. |
| Approach: | They propose an efficient and scalable system that can make petabyte-level text corpora searchable by using the FM-index data structure. |
| Outcome: | The proposed system indexes 83TB of Internet text in 99 days with a single 128-core CPU node (or 19 hours if using 137 such nodes). |
DIALKI: Knowledge Identification in Conversational Systems through Dialogue-Document Contextualization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge grounding models focus on locating knowledge in document contexts that are relevant to the conversation. |
| Approach: | They propose a knowledge identification model that leverages document structure to provide dialogue-contextualized passage encodings and better locate knowledge relevant to the conversation. |
| Outcome: | The proposed model can be applied to document-grounded conversational datasets and shows generalization to unseen documents and long dialogue contexts. |
BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer (2024.naacl-long)
Copied to clipboard
Akari Asai, Sneha Kudugunta, Xinyan Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, Hannaneh Hajishirzi
| Challenge: | Recent advances in few-shot generalization in natural language processing focus on English. |
| Approach: | They propose a benchmark that unifies 15 diverse tasks across 54 languages in a sequence-to-sequence format and provides a fixed set of few-shot examples and instructions. |
| Outcome: | The proposed framework unifies 15 diverse tasks across 54 languages in a sequence-to-sequence format and provides a fixed set of few-shot examples and instructions. |
MedICaT: A Dataset of Medical Images, Captions, and Textual References (2020.findings-emnlp)
Copied to clipboard
Sanjay Subramanian, Lucy Lu Wang, Ben Bogin, Sachin Mehta, Madeleine van Zuylen, Sravanthi Parasa, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi
| Challenge: | Existing largescale datasets explicitly exclude compound figures . existing systems lack this ability to identify relevant subfigures . |
| Approach: | They propose a dataset of medical images in context that allows figure-to-text alignment . they use captions, inline references and manually annotated subfigures for compound figures . |
| Outcome: | The proposed dataset demonstrates the utility of inline references in image-text matching. |
OLMES: A Standard for Language Model Evaluations (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing models claim to perform better on tasks measuring model capabilities, but there is no standard setup for reproducible evaluations. |
| Approach: | They propose a document that is documented and practical for reproducible LLM evaluations and includes recommendations from existing literature and new experiments. |
| Outcome: | The proposed standard identifies and reviews the varying factors in evaluation practices adopted by the community, such as prompt formatting, choice of in-context examples, probability normalizations, and task formulation. |
Mixture Content Selection for Diverse Sequence Generation (D19-1)
Copied to clipboard
| Challenge: | Generating diverse sequences exhibit semantically one-to-many relationships between source and target sequences. |
| Approach: | They propose to separate diversification from generation using a general plug-and-play module that wraps around and guides an existing encoder-decoder model. |
| Outcome: | The proposed method shows that diversification and generation are separate steps in the same model and that the model is robust. |
TaskWeb: Selecting Better Source Tasks for Multi-task NLP (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work in NLP has shown that knowing task relationships via pairwise task transfer improves choosing one or more source tasks that help to learn a new target task. |
| Approach: | They propose a method to quantify task relationships via pairwise task transfer and build smaller training sets that improve zero-shot performances across 11 different target tasks. |
| Outcome: | The proposed method improves overall rankings and top-k precision of source tasks by 10% and 38%, respectively. |
Compositional Questions Do Not Necessitate Multi-hop Reasoning (P19-1)
Copied to clipboard
| Challenge: | a single-hop reasoning model can solve much more of the dataset than previously thought. |
| Approach: | They propose a single-hop BERT-based RC model that achieves 67 F1 . they propose an evaluation setting where humans are not shown all paragraphs . |
| Outcome: | The proposed model achieves 67 F1—comparable to state-of-the-art multi-hop models. |
Efficient Passage Retrieval with Hashing for Open-domain Question Answering (2021.acl-short)
Copied to clipboard
| Challenge: | Open-domain question answering systems often require large memory to run because of the massive size of their passage index. |
| Approach: | They propose a memory-efficient neural retrieval model that integrates a learning-to-hash technique into the state-of-the-art Dense Passage Retriever to represent the passage index using compact binary codes. |
| Outcome: | The proposed model significantly reduces memory cost from 65GB to 2GB without loss of accuracy on two open-domain question answering benchmarks. |
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts (2022.naacl-main)
Copied to clipboard
Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, Yejin Choi
| Challenge: | Recent work shows the surprising power of continuous prompts to language models for controlled generation and solving a wide range of tasks. |
| Approach: | They propose to extract a discrete (textual) interpretation of continuous prompts faithful to the problem they solve. |
| Outcome: | The proposed model can find prompts that solve a task while being projected to an arbitrary text with a smaller drop in accuracy. |
FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Large pre-trained models are capable of few-shot in-context learning (ICL) however, concatenated demonstrations are often excessively long and require additional computation. |
| Approach: | They propose to apply fusion-in-decoder (FiD) models to perform few-shot in-context learning (ICL) they propose to use concatenation-based, early-fusion, intermediate- and late-fusion methods to improve efficiency . |
| Outcome: | The proposed methods outperform concatenation-based models on 11 held-out tasks. |
Beyond Paragraphs: NLP for Long Sequences (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we will introduce document-level representation learning techniques . document-based learning is challenging due to the limited sequence length of many models . |
| Approach: | They will provide an overview of established long sequence NLP techniques and discuss memory-saving methods that are key to processing long sequences. |
| Outcome: | The tutorial will introduce the latest and ongoing techniques for document-level representation learning. |
CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on counterfactual data augmentation only considered restricted classes of perturbations, limiting their effectiveness. |
| Approach: | They propose a retrieval-augmented framework for creating diverse counterfactual perturbations for CDA. |
| Outcome: | Experiments on natural language inference and sentiment analysis show that the proposed framework can be used to encourage diversity in manually authored perturbations. |
s1: Simple test-time scaling (2025.emnlp-main)
Copied to clipboard
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candes, Tatsunori Hashimoto
| Challenge: | OpenAI’s o1 model showed this capability but did not publicly share its methodology, leading to many replication efforts. |
| Approach: | They curate a small dataset s1K with 1,000 reasoning questions based on three criteria we validate through ablations: difficulty, diversity, and quality. |
| Outcome: | The proposed model exceeds o1-preview on competition math questions by up to 27% (MATH and AIME24). |
A Discrete Hard EM Approach for Weakly Supervised Question Answering (D19-1)
Copied to clipboard
| Challenge: | Existing work on question answering tasks only provide weak supervision for how the answer should be computed . weak supervision is attractive because it is relatively easy to gather, allowing for large datasets . but weak supervision complicates learning because there are many different spurious ways to derive the correct answer. |
| Approach: | They propose a method to convert question answering tasks into discrete latent variable learning problems with a precomputed set of possible solutions that contains one correct option. |
| Outcome: | The proposed approach outperforms previous methods on six QA tasks and achieves state-of-the-art on five of them. |
Set the Clock: Temporal Alignment of Pretrained Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Language models (LMs) are trained on web text originating from many points in time and, in general, without any explicit temporal grounding. |
| Approach: | They construct a time-sensitive question dataset and use it to examine temporal alignment methods to align their internal knowledge to a target time. |
| Outcome: | The proposed methods improve LLaMa2's performance by 62% if they are fine tuned to the year 2022 . |
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023.emnlp-main)
Copied to clipboard
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, Hannaneh Hajishirzi
| Challenge: | Evaluating the factuality of long-form text generated by large language models (LMs) is non-trivial because (1) generations often contain a mixture of supported and unsupported pieces of information, making binary judgments of quality inadequate and (2) human evaluation is time-consuming and costly. |
| Approach: | They introduce a new evaluation that breaks a generation into a series of atomic facts and computes the percentage of atom facts supported by a reliable knowledge source. |
| Outcome: | The proposed model breaks a generation into atomic facts and computes the percentage of atomic fact supported by a reliable knowledge source. |
Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to augment retrieval-augmented generation models with retrievers often rely on spurious cues or generate hallucinations during inference. |
| Approach: | They propose a method to incorporate evidentiality of passages into training a retrieval-augmented generation model. |
| Outcome: | The proposed method outperforms its direct counterpart on all knowledge-intensive tasks. |
ComPO: Community Preferences for Language Model Personalization (2025.naacl-long)
Copied to clipboard
| Challenge: | Current methods for training language models with human feedback rely on subjective preferences that are assumed to account for an "average" user . however, annotating preferences is inherently subjective and results in generic models that generate outputs not preferred by many user groups. |
| Approach: | They propose a method to personalize preference optimization in LMs by contextualizing the probability distribution of model outputs with the preference provider. |
| Outcome: | The proposed method improves performance by focusing on group-level preferences rather than individual feedback. |
Generated Knowledge Prompting for Commonsense Reasoning (2022.acl-long)
Copied to clipboard
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, Hannaneh Hajishirzi
| Challenge: | Existing methods for commonsense reasoning rely on high-quality knowledge, but they are often dominated by large-scale pretrained models that are fine-tuned on a target benchmark. |
| Approach: | They develop generated knowledge prompting which generates knowledge from a language model and provides it as additional input when answering a question. |
| Outcome: | The proposed method improves state-of-the-art models on four commonsense reasoning tasks. |
SHARCS: Efficient Transformers Through Routing with Dynamic Width Sub-networks (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Several methods have been proposed to improve the inference efficiency of transformer-based models. |
| Approach: | They propose a new adaptive inference method that takes into account the hardness of input samples. |
| Outcome: | The proposed model outperforms or complements existing per-sample adaptive inference methods in terms of accuracy vs. FLOPs and can be applied to compressed and efficient transformer encoders to further improve their efficiency. |
Entity, Relation, and Event Extraction with Contextualized Span Representations (D19-1)
Copied to clipboard
| Challenge: | Existing frameworks for named entity recognition, relation extraction, and event extraction can be easily adapted for new tasks or datasets. |
| Approach: | They propose a framework that enumerates, refins, and scores text spans to capture local (within-sentence) and global (cross-sentent) context. |
| Outcome: | The proposed framework achieves state-of-the-art results on four datasets from a variety of domains. |
CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation (2024.emnlp-main)
Copied to clipboard
Tong Chen, Akari Asai, Niloofar Mireshghallah, Sewon Min, James Grimmelmann, Yejin Choi, Hannaneh Hajishirzi, Luke Zettlemoyer, Pang Wei Koh
| Challenge: | Existing studies focus on literal copying, but current methods reduce literal copy but not non-literal copying. |
| Approach: | They propose a benchmark to measure literal and non-literal copying in LMs . they use copyrighted fiction books as text sources to assess literal copying . |
| Outcome: | The proposed model measures literal and non-literal copying in copyrighted texts . large models show significantly more copying, with literal copying rates increasing . |
ZeroShotCeres: Zero-Shot Relation Extraction from Semi-Structured Webpages (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on information extraction from semi-structured websites has relied on manual data annotation and learning a model specific to a given template. |
| Approach: | They propose a solution for “zero-shot” open-domain relation extraction from webpages with previously unseen templates using a graph neural network-based approach. |
| Outcome: | The proposed model provides a 31% gain over baseline for zero-shot extraction in a new subject vertical. |
Task-aware Retrieval with Instructions (2023.findings-acl)
Copied to clipboard
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, Wen-tau Yih
| Challenge: | Existing models that learn intents from labeled data are complicated and require a vast number of annotated examples to train a model. |
| Approach: | They propose a general-purpose task-aware retrieval system with instructions that can adapt to a new task without any parameter updates. |
| Outcome: | The proposed system outperforms two benchmarks on a set of domains and tasks on X2-Retrieval. |
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2022.acl-long)
Copied to clipboard
| Challenge: | Despite the success of supervised learning, models often struggle with generalization across tasks. |
| Approach: | They propose to use crowdsourcing instructions to build a model that learns a new task by understanding the human-readable instructions that define it. |
| Outcome: | The proposed model can learn from seen tasks and generalize to unseen tasks given its natural crowdsourcing instructions. |
Machine Reading Comprehension using Case-based Reasoning (2023.findings-emnlp)
Copied to clipboard
Dung Thai, Dhruv Agarwal, Mudit Chaudhary, Wenlong Zhao, Rajarshi Das, Jay-Yoon Lee, Hannaneh Hajishirzi, Manzil Zaheer, Andrew McCallum
| Challenge: | Current state-of-the-art machine readers do not support case-based reasoning . |
| Approach: | They propose a method that extracts a set of similar cases from a nonparametric memory and then predicts an answer by selecting the span in the test context that is most similar to the contextualized representations of answers. |
| Outcome: | The proposed method outperforms baselines on NaturalQuestions and NewsQA by 11.5 and 8.4 EM. |
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020.findings-emnlp)
Copied to clipboard
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou
| Challenge: | Standard test sets for supervised learning evaluate in-distribution generalization but are misleading when a dataset has systematic gaps. |
| Approach: | They propose a more rigorous annotation paradigm for NLP that helps to close systematic gaps in the test data. |
| Outcome: | The proposed model performs significantly lower on contrast sets than on the original test sets—up to 25% in some cases. |
Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction (D18-1)
Copied to clipboard
| Challenge: | Existing relation extraction systems are designed for within-sentence relations, but extracting information from scientific articles requires relations across sentences. |
| Approach: | They propose a multi-task setup for identifying entities, relations, and coreference clusters in scientific articles . they develop a unified framework called SciIE with shared span representations to solve this problem . |
| Outcome: | The proposed model outperforms existing models without domain-specific features in scientific information extraction. |
Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback (2025.acl-long)
Copied to clipboard
Lester James Validad Miranda, Yizhong Wang, Yanai Elazar, Sachin Kumar, Valentina Pyatkin, Faeze Brahman, Noah A. Smith, Hannaneh Hajishirzi, Pradeep Dasigi
| Challenge: | Learning from human feedback has enabled the alignment of language models (LMs) with human preferences. |
| Approach: | They propose a Hybrid Preference routER that defers an annotation to either humans or LMs, achieving better annotation quality while reducing the cost of human-only annotation. |
| Outcome: | The proposed model achieves better annotation quality while reducing the cost of human-only annotation. |
On Making Reading Comprehension More Comprehensive (D19-58)
Copied to clipboard
| Challenge: | Getting machines to "understand" text is a vast and long-standing problem, made more challenging by the fact that it is not even clear what it means to understand text. |
| Approach: | They propose a question-based approach to machine reading comprehension that uses a natural language question to test a system's comprehension of a passage of text. |
| Outcome: | The proposed questions have surface cues or other biases that allow a model to shortcut the intended reasoning process. |
Reframing Instructional Prompts to GPTk’s Language (2022.findings-acl)
Copied to clipboard
| Challenge: | Using reframing techniques, we find that instructional prompts are easier to follow for Language Models (LMs) |
| Approach: | They propose reframing techniques for manual reformulation of prompts into more effective ones . they compare performance of LMs prompted with reframed instructions on 12 NLP tasks . |
| Outcome: | The reframing techniques used for prompt reformulation improve performance on 12 tasks . the techniques boost performance on LMs with different sizes compared with original prompts . |
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020.emnlp-main)
Copied to clipboard
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, Yejin Choi
| Challenge: | Large datasets have become commonplace in NLP research, but the emphasis on quantity has made it challenging to assess the quality of data. |
| Approach: | They propose a model-based tool to characterize and diagnose large datasets . they leverage the behavior of the model on individual instances during training . |
| Outcome: | Experiments on four datasets show that the tool can characterize and diagnose datasets with a model-based tool. |
XOR QA: Cross-lingual Open-Retrieval Question Answering (2021.naacl-main)
Copied to clipboard
| Challenge: | a dataset of 40k information-seeking questions across seven languages is used to answer multilingual question answering tasks. |
| Approach: | They propose a task framework that allows questions from one language to be answered via answer content from another language. |
| Outcome: | The proposed framework can be used to answer questions from one language to another . the dataset was built on 40K questions across 7 languages, but could not find same-language answers . |
Crystal: Introspective Reasoners Reinforced with Self-Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge-augmented reasoning methods fail to capture the *introspective* nature of knowledge required in commonsense reasoning. |
| Approach: | They propose a method to develop an introspective commonsense reasoner that introspects for knowledge statements related to the given question and makes an informed prediction. |
| Outcome: | The proposed method outperforms standard supervised finetuning and chain-of-thought distilled methods and enhances the transparency of the commonsense reasoning process. |
Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Adapting general-purpose language models to new skills is currently expensive . Adaptation to new skill sets requires repeated training or models forget older skills . |
| Approach: | They propose a parallel-train-then-merge procedure that adds new skills to preexisting models in isolation and later merges with the general model. |
| Outcome: | The proposed method is cheaper than retraining models on updated datasets . it improves model compliance with safe prompts while preserving model's ability to refuse dangerous or harmful prompts. |
RewardBench: Evaluating Reward Models for Language Modeling (2025.findings-naacl)
Copied to clipboard
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi
| Challenge: | Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models. |
| Approach: | They present a benchmark dataset and code-base for evaluation of reward models . they use prompt-chosen-rejected trios to benchmark how they perform on queries . |
| Outcome: | The proposed dataset compares RMs with other models on a set of questions. |
Extracting a Knowledge Base of Mechanisms from COVID-19 Papers (2021.naacl-main)
Copied to clipboard
Tom Hope, Aida Amini, David Wadden, Madeleine van Zuylen, Sravanthi Parasa, Eric Horvitz, Daniel Weld, Roy Schwartz, Hannaneh Hajishirzi
| Challenge: | COVID-19 has spawned a diverse body of scientific literature that is challenging to navigate . researchers are using automated tools to help find useful knowledge . |
| Approach: | They develop a schema to extract mechanism relations from scientific papers . their search engine, dataset and code are publicly available . |
| Outcome: | The proposed schema outperforms PubMed search in clinical trials. |
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling (2022.emnlp-main)
Copied to clipboard
| Challenge: | Abstractive summarization models often generate inconsistent summaries containing factual errors or fabricated content. |
| Approach: | They propose to generate representative examples of non-factual summaries through infilling language models and train a robust fact-correction model to post-edit them to improve factual consistency. |
| Outcome: | The proposed model outperforms previous methods in correcting factual errors on two popular summarization datasets. |
Noisy Channel Language Model Prompting for Few-Shot Text Classification (2022.acl-long)
Copied to clipboard
| Challenge: | Prior work has suggested methods for finding better prompt or scoring of the output from the model. |
| Approach: | They propose a noisy channel approach for language model prompting in few-shot text classification by in-context demonstration or prompt tuning. |
| Outcome: | The proposed model outperforms direct models in both demonstration and prompt tuning. |
Fact or Fiction: Verifying Scientific Claims (2020.emnlp-main)
Copied to clipboard
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, Hannaneh Hajishirzi
| Challenge: | SciFact is a dataset of 1.4K expert-written scientific claims paired with evidence-containing abstracts annotated with labels and rationales. |
| Approach: | They construct a dataset of 1.4K scientific claims paired with evidence-containing abstracts annotated with labels and rationales to test their system. |
| Outcome: | The proposed system can verify claims related to COVID-19 by identifying evidence from the CORD-19 corpus. |
AmbigQA: Answering Ambiguous Open-domain Questions (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing open-domain question answering systems assume questions have a single welldefined answer. |
| Approach: | They propose an open-domain question answering task which involves finding every plausible answer and rewriting the question for each one to resolve the ambiguity. |
| Outcome: | The proposed task is based on a dataset covering 14,042 open-domain questions . it shows that strong models benefit from weakly supervised learning . |
Contextualized Sparse Representations for Real-Time Open-Domain Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | Existing phrase retrieval models suffer from low accuracy due to limited scalability and speed . 'Open-domain question answering' is a task of answering generic factoid questions by looking up a large knowledge source, typically unstructured text corpora such as Wikipedia. |
| Approach: | They aim to augment existing phrase retrieval models with contextualized sparse representations to improve the quality of each phrase embedding. |
| Outcome: | The proposed model improves CuratedTREC and SQuAD-Open by 4% and 45x faster inference speeds over the existing model. |
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent research shows that relevant knowledge can provide useful context for commonsense tasks. |
| Approach: | They propose a method that learns to generate contextually relevant knowledge in response to given questions. |
| Outcome: | The proposed method shows consistent gains over 9 commonsense benchmarks. |
GooAQ: Open Question Answering with Diverse Answer Types (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on niche answer types, mainly short responses and, in a few cases, long responses, has failed to adequately address the answer diversity of questions. |
| Approach: | They propose to use Google's autocomplete feature to collect questions from a large-scale dataset with a variety of answer types to facilitate further research on improving QA with diverse response types. |
| Outcome: | The proposed model produces naturalistic questions that are short and expressed using simple language. |
A Systematic Examination of Preference Learning through the Lens of Instruction-Following (2025.naacl-long)
Copied to clipboard
Joongwon Kim, Anirudh Goyal, Aston Zhang, Bo Xiong, Rui Hou, Melanie Kambadur, Dhruv Mahajan, Hannaneh Hajishirzi, Liang Tan
| Challenge: | a recent study has found that preference learning is a key tool for enhancing LLM training and alignment. |
| Approach: | They use a synthetic data generation pipeline to generate 48,000 unique instruction-following prompts with 23 verifiable constraints to obtain preference pairs. |
| Outcome: | The proposed pipeline generates 48,000 unique instruction-following prompts with 23 verifiable constraints that enable fine-grained and automated quality assessments of model responses. |
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022.emnlp-main)
Copied to clipboard
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer
| Challenge: | Large language models can in-context learn by conditioning on a few input-label pairs and making predictions for new inputs. |
| Approach: | They propose to use ground truth demonstrations to replace labels in demonstrations . they also show that other aspects of the demonstrations are key drivers of endtask performance . |
| Outcome: | The proposed model outperforms zeroshot inference on a wide range of tasks using ground truth demonstrations. |
SciFact-Open: Towards open-domain scientific claim verification (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Current scientific claim verification systems can achieve very strong performance on limited contexts, in some cases approaching human agreement. |
| Approach: | They propose to pool and annotate top predictions from four state-of-the-art scientific claim verification models to evaluate their performance against large corpora. |
| Outcome: | The proposed system performs well on a corpus of 500K scientific abstracts. |
Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statements (2023.emnlp-main)
Copied to clipboard
| Challenge: | Despite the advances of language models, they still produce text that contains trivial commonsense errors. |
| Approach: | They propose a general-purpose commonsense statement verification model that learns to estimate the plausibility of declarative statements based on commonsensical knowledge. |
| Outcome: | The proposed model outperforms existing models that can be repurposed for commonsense verification, even including GPT-3.5/ChatGPT/GPT-4. |
ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts (2022.emnlp-main)
Copied to clipboard
| Challenge: | a new multi-task, parameter-efficient language model tuning method learns to transfer knowledge across different tasks via a mixture of soft prompts. |
| Approach: | They propose a multi-task, parameter-efficient language model tuning method that uses soft prompts to learn to transfer knowledge across different tasks. |
| Outcome: | The proposed method outperforms prompt tuning and outperfies or matches fully fine-tuned tuning approaches that use 10 times more parameters. |
MultiVerS: Improving scientific claim verification with weak supervision and full-document context (2022.findings-naacl)
Copied to clipboard
| Challenge: | a new approach to scientific claim verification uses a document-level fact-checking label to label scientific documents . a multitask approach combines a shared encoding of the claim and document context . |
| Approach: | They propose a system which predicts a fact-checking label and identifies rationales in a multitask fashion based on a shared encoding of the claim and full document context. |
| Outcome: | The proposed approach outperforms baselines on three scientific claim verification datasets . it can learn from instances annotated with a document-level fact-checking label, but lacking sentence-level rationales based on the datasets. |
Multi-modal Information Extraction from Text, Semi-structured, and Tabular Data on the Web (2020.acl-tutorials)
Copied to clipboard
| Challenge: | a tutorial explores the commonalities in the challenges and solutions developed to address information extraction from the World Wide Web. |
| Approach: | This tutorial examines methods for extracting information from the World Wide Web . it explores the commonalities in the challenges and solutions developed to address these different forms of text . |
| Outcome: | This paper examines the commonalities in the challenges and solutions developed to address the World Wide Web. |
Pyramidal Recurrent Unit for Language Modeling (D18-1)
Copied to clipboard
| Challenge: | Long short term memory units are powerful tools for language modeling, but their performance can be limited by the number of parameters. |
| Approach: | They propose a pyramidal recurrent unit which enables learning representations in high dimensional space with more generalization power and fewer parameters. |
| Outcome: | The proposed model outperforms existing models with different gating mechanisms and transformations on word-level language modeling tasks. |
Joint Passage Ranking for Diverse Multi-Answer Retrieval (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to multi-answer retrieval cannot reason about the set of passages jointly. |
| Approach: | They propose a joint passage retrieval model focusing on reranking to solve multi-answer retrieval problem. |
| Outcome: | The proposed model outperforms baseline models on three multi-answer datasets. |
Logic-Guided Data Augmentation and Regularization for Consistent Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies have focused on improving the consistency of comparison questions . current models show inconsistent comparison predictions due to small datasets . |
| Approach: | They propose a method that integrates logic rules and neural models to improve the accuracy of comparison questions by enhancing labeled training data. |
| Outcome: | The proposed method improves state-of-the-art models on WIQA and QuaRel and reduces consistency violations by 58% on HotpotQA. |
Semi-Supervised Event Extraction with Paraphrase Clusters (N18-2)
Copied to clipboard
| Challenge: | Existing event extraction systems are limited in their accuracy due to the lack of available training data. |
| Approach: | They propose a method for self-training event extraction systems by bootstrapping additional training data. |
| Outcome: | The proposed method improves on ACE 2005 and TAC-KBP 2015 datasets. |
SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature (2025.emnlp-main)
Copied to clipboard
David Wadden, Kejian Shi, Jacob Morrison, Alan Li, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, Doug Downey, Hannaneh Hajishirzi, Arman Cohan
| Challenge: | ScIRIFF is the only entirely expert-written instruction-following dataset for scientific literature understanding . it features complex instructions with long input contexts, detailed task descriptions, and structured outputs. |
| Approach: | They present a dataset of 137K instruction-following instances for training and evaluation . they finetuned large language models using a mix of general domain and ScIRIFF instructions . |
| Outcome: | The proposed dataset shows that on nine out-of-distribution held-out tasks, the model performs better than baselines trained on general domain instructions. |